J. Rosamond Johnson began playing the piano at age four. He received formal musical training at the New England Conservatory of Music n Boston, specializing in piano, organ, composition, and voice. He studied piano with Charles Dennée and Mme. Dietrich Strong, organ with George Whiting, harmony with Carl Reissman and Davenport Kerrison, and voice with William and Clarence B. Ashenden. He also received an honorary M.A. from Atlanta University in 1917.

By 1901 he moved to New York where he traveled in the vaudeville show, Oriental America. Before moving to New York he had taught music in Jacksonville, Florida, and wrote the popular songs Lil' Gal and Since You Went Away with his brother, James Weldon Johnson. In 1900 he and James Weldon Johnson composed "Lift Every Voice and Sing," one of the most influential songs of the twentieth century. In 1901 he and his brother began their collaboration with Bob Cole. The trio produced numerous hits including The Maiden with the Dreamy Eyes, the Old Flag Never Touched the Ground, My Castle on the Nile, Under the Bamboo Tree, and Didn't He Ramble. The Johnsons and Bob Cole composed music for the musicals The Sleeping Beauty and Humpty Dumpty before they split in 1906. J. Rosamond wrote just one more show, Mr. Lode of Kole, in 1909.

J. Rosamond was the music director at Hammerstein's London Opera House from 1912 to 1913. He then returned to New York where he taught at the Music School Settlement for Colored People, collaborated with James Weldon Johnson on two collections of spirituals, The Book of American Negro Spirituals (1925) and The Second Book of Negro Spirituals (1926). J. Rosamond Johnson published two more song collections, Shout Songs in 1936 and Rolling Along in Song in 1937⁵¹. Following the publication of his final song collection, Johnson performed in several musicals including Mamba's Daughters from 1939-1941, Cabin in the Sky in 1940, Porgy and Bess from 1942-1943, and A Young American in 1946.