Son of William Kelly Sr. and Mary Bryans Kelly, he was the fourth of their nine children. Born in 1820 at Ballyshiel, near Clare, County Armagh, Ireland, he was a stone mason and, in the U.S., a farmer. He married Jane Robinson on 26 May 1849 in Belvidere, Boone County, Illinois. They had several children there, and then moved west to eventually settle on their farm near Clare, Johnson County, Kansas (named after their homeplace in Ireland).
They had 14 children in all. The following children were born in Belvidere, Illinois:
Mary Elizabeth/Eliza (the oldest, 1850-1873), Agnes (1853-1929, married George P. Dawson), William James (1854-1936; married Hattie Sophia Millikan), Jane (1856-1942; married John Robinson Atwood), Margaret (1857-1917; married Edward M. Day), and Thomas Alexander Kelly (1859-1942; married Carrie Violet Runner);
and the rest were born in Kansas:
John (1861-1937; married Sarah Ann Millikan), Moses (1863-1945; married Laura Cady Hunt), Isabel "Birdie" (1865-1952; mar. John M. Shields), George Francis "Frank" (1867-1941; mar. Laura May Kelly), Sarah Ann "Sadie" (1868-?; married Will Ryan), Helen Winifred (1870-1955; married Allen R. Millikan), Lucinda "Cindy" (1872-1950; married George B. Dent), and the youngest child who died at birth.