Son of Robert Carter and his wife, Mary Nelson Carter, the daughter of General Thomas Nelson.
First marriage to Juliet Muse Gaines, the daughter of Henry Gaines. She died at 28 years. He went on to marry Anne Willing Page, known as "Sweet Anne Page." They purchased "Annfield," where they lived together for many years.