Named for his Gifford grandfather, Joseph was the eldest child of Abraham Gifford and Lucy Sherman. He spent his young childhood in eastern NY near Easton and Troy and moved at the age of 13 with his family to what is now Collins in Erie County, NY.
At the age of 23, Joseph married Silvia or Sylvia Jenney, a daughter of John Jenney and Catharine Davis. After their marriage, the couple lived in Collins for several years where both Joseph and Sylvia were active in the Society of Friends. By the fall of 1830, the couple and their young son, Giles Jacob, moved permanently to Huron County, Ohio joining the elder Jenneys and Joseph's brother, Humphrey (who married Sylvia's sister, Jane.)
Settling in Ripley Twp., Joseph and Sylvia raised their family of 10 children: Giles Jacob, Lucy 1828-1829 (buried in NY), John Job b. 1831, Charles b. 1833, Obadiah 1835-1837, Sylvia b. 1837, Elizabeth b. 1838, Sarah Jane b. 1842, Joseph B. b. 1845, and Benjamin b. 1847. About 1853, the family moved west to Springfield Twp., Williams County, Ohio, where they lived for several years. They were probably planning a move further west to Kansas with their children when Joseph died on 12m 1d 1863.
Joseph's final resting place is in the private cemetery in Greenwich Twp. on what was once the farm of Sylvia's parents and later that of her brother, Abram D. Jenney. The gravesite includes both a headstone and footstone with transcription from the reading by Huron County OGS Chapter members, including Paula Porter Griffin and Henry Timman.