Peter Terrell Fite was the son of Leonard Fite and Winny West. His family moved from North Carolina to Overton County, Tennessee about 1825 (his father died on the way, in Knox County, TN). He married Catherine Elizabeth "Elizabeth" Matthews about 1830.

His paternal grandparents, Peter Fite, Sr., and Anna Barbery Summers Fite are buried at Goshen Presbyterian Church Cemetery, Belmont, Gaston County, NC.

Peter and Elizabeth's children were:
- Peter Terrell Fite, Jr. (b. 1831, Overton County, TN; d. 1843, Overton County, TN)
- Sarah Manerva Fite Green (b. May 01, 1833, Overton County, TN; d. Oct 18, 1900) (see Find A Grave Memorial# 44875468). Married Newton Green.
- Rufus Mitchell Fite (b. Aug 07, 1834, Overton County, TN; d. Oct 03, 1911, Kingston, Hunt County, TX, burial location unknown). Married 1) Elizabeth Frances Peak (bef. 1853) and 2) Rebecca J. Ray on 9/21/1877, Putnam County, TN. (Rebecca Ray Fite is buried at Texas State Cemetery, Austin, Travis County, TX). He fought in the Confederate army during the Civil War.
- Permilla Fite (b. 1836, Overton County, TN; d. Unknown)
- Winny Belle Caroline Fite Bilbrey (b. Dec 27, 1838, Overton County, TN; d. Oct 28, 1926, Pickett County, TN) (see Find A Grave Memorial# 44875680). Married James Jefferson "Jefferson" Bilbrey.
- Cynthia Adeline Fite Pennycuff (b. Jul 1842, Overton County, TN; d. abt. 1924, Fentress County, TN, buried at Franklin Cemetery, Poplar Cove, Fentress County, TN) (see Find A Grave Memorial# 44875844). Married John Pennycuff.
- Mary L. "Polly" Fite Moody (b. May 05, 1844, Overton County, TN; d. May 29, 1918, Forbus, Pickett County, TN, (see Find A Grave Memorial# 20938614). Married Green C. Moody.
- John Calvin Fite (b. John Calvin Fite, Overton County, TN; d. Dec 12, 1929, Toledo, Lucas County, OH, buried at Maplewood Cemetery, Harrison, Boone Co, AR) (see Find A Grave Memorial# 54648733). Married 1) Isabelle Jane Burford on Mar 28, 1867, Henderson, Rusk County, TX; 2) Lilly West (bef. 1884); 3) Esther Allinder on Sep 20, 1888 in Faulkner County, AR. He fought in the Confederate army during the Civil War.
- Nancy Jane Fite Evans (b. 1846, Overton County, TN; d. aft. 1880, burial location unknown). Married Euin Shelby "Shelby" Evans in 9/1869.
- Gemima Lou Ann "Mima" Fite Manis (name also shown as Jemima) (b. May 08, 1850, Overton County, TN; d. Nov 26, 1923, Pickett County, TN) (see Find A Grave Memorial# 44877203). Married James Madison "Jim" Manis.
- Frances Elizabeth "Lizzie" Fite Hinds (b. Jul 04, 1852, Fentress County, TN; d. Jan 09, 1919, Fentress County, TN) (see Find A Grave Memorial# 44877593). Married Joel E. "Jode" Hinds.