In the attack on Fort Wagner, the regiment was assigned the honor of leading the assault, and, when the division was drawn up on the beach at nightfall preparatory to the attack, the order to advance was delayed until the Fifty-fourth marched by and took its place at the head of the column. It charged under fire over a long distance of sandy plain, reaching the ditch, where many of the men climbed the parapet and entered the outer works; but the fort proved impregnable, and a bloody repulse ensued, the regiment losing 34 killed, 146 wounded, and 92 captured or missing.

The Fifty-fourth was organized in April, 1863, at Readville, Mass., and was one of the first colored regiments organized in the Northern States. The men came from the free colored citizens of New England and the Middle States, while many came from far Western States to embrace this, their first opportunity to enlist. Governor Andrews tendered the Colonelcy to Captain Robert G. Shaw, of the Second Massachusetts Infantry, who accepted. Shaw was killed at Fort Wagner; he was the first man on the parapet, where he fell, shot through the heart. At Olustee, the regiment lost 11 killed, 68 wounded and 8 missing; at Honey Hill, 3 killed, 38 wounded, and 4 missing; at Boykin's Mill, 2 killed, and 20 wounded. After the close of the war it remained in South Carolina, on garrison duty, until August 20, 1865, when it was mustered out and ordered to Boston, where the men received their final payment and discharge.

Shortly after Olustee, the Sergeant-Major (colored), was commissioned a lieutenant by Governor Andrews, for gallantry in that battle; but for a long time the United States Government refused to muster him in, on account of his color. Admittance to the Invalid Corps was also refused a private who was disabled at Fort Wagner. Full pay was also refused these men for sixteen months. On seven successive pay-days they were tendered $7 per month, but each time it was refused and a white soldier's pay demanded. On September 28, 1864, the men were paid in full from the date of enlistment, at $13 per month.