Ruffin was born Davis Eli Ruffin on January 18, 1941 in the rural unincorporated community of Whynot, Mississippi, the son of Eli, a Baptist minister, and Ophelia Ruffin (née Davis). His father was strict and at times violently abusive. Ruffin's mother died just months after his birth and his father married a schoolteacher, Earline, in 1942. As a young child, Ruffin, along with his other siblings (older brothers Quincy and Jimmy, and sister Rita Mae), traveled with their father and their stepmother as a family gospel group opening shows for Mahalia Jackson and The Five Blind Boys of Mississippi among others. He also had an older sister Rosetta who died as a child. Ruffin sang in the choir at Mount Salem Methodist Church, talent shows, and wherever else he could. In 1955, at the age of fourteen, he left home under the guardianship of a minister and went to Memphis, Tennessee with the purpose of pursuing the ministry.
At age 15, Ruffin went to Hot Springs, Arkansas with the jazz musician Phineas Newborn, Sr. There they played at the Fifty Grand Ballroom and Casino. Ruffin continued to sing at talent shows, worked with horses at a jockey club and eventually became a member of the Dixie Nightingales. He also sang with the Soul Stirrers briefly after the departure of Johnnie Taylor. It was in Ruffin's travels as a teenager that he met such later popular personalities as Elvis Presley, Little Richard, Bobby Womack, the Staple Singers, the Swan Silvertones and the Dixie Hummingbirds.
At age 15, Ruffin went to Hot Springs, Arkansas with the jazz musician Phineas Newborn, Sr. There they played at the Fifty Grand Ballroom and Casino. Ruffin continued to sing at talent shows, worked with horses at a jockey club and eventually became a member of the Dixie Nightingales. He also sang with the Soul Stirrers briefly after the departure of Johnnie Taylor. It was in Ruffin's travels as a teenager that he met such later popular personalities as Elvis Presley, Little Richard, Bobby Womack, the Staple Singers, the Swan Silvertones and the Dixie Hummingbirds.