He was first married to Alice Dickerson (married Aug. 23, 1894), who died in 1899. A few months after her death, he married Lily Bell White Padgett (married Nov. 9, 1899), widow of J. T. Padgett, to whom he was married for ten years. She died in 1909. In 1914, he married a third time, this time to Stella Clark Dorman (married Sept. 27, 1914), widow of Will Dorman. After she died, he never married again.
His three marriages produced seven children: With Alice, he had daughter Mary Ann Sheffield and son James Ewell "Buddy" Sheffield. With Lily, he had daughter Ruby Elza Sheffield (later Goff). With Stella, he had daughters Belva Maxine Sheffield (Handley), Frances Goodwin Sheffield (Beene), Laura B. Sheffield (Brown), and Zelda Finiece Sheffield (Thompson).
He has too many grandchildren, great-grandchildren, and great-great-grandchildren to count.
He was a farmer and a pillar of the community. Jasper served on the school board many times, and believed that a good education should be attainable for all, regardless of race.