According to family tradition and a number of later documents, Annie Lee Jones was born 30 October 1876 in Richmond, Virginia. Earlier evidence, however, including her birth record, shows that Anna (later nicknamed "Annie") Lee was born in Dover Township, Goochland County, to Richard T. and Mary Ann Blalock Jones. Her actual birthdate appears to have been 31 August 1876. By 1880 the family had settled in nearby Tuckahoe, Henrico, which later became part of Richmond. Annie Lee appears to have spent most of her childhood and young adulthood in Richmond City and the vicinity. She had one sister, Florence Estelle, who was ten years her junior.

Annie Lee married William Richard Cardwell in Petersburg, Virginia on 11 November 1904, according to family tradition. Their Petersburg marriage record gives 9 November 1904 as the date. The couple lived first in Petersburg and then Ettrick (Matoaca), Virginia, before moving to North Carolina in about 1913. The first six of their seven children were born in Virginia: Brunetta Frances, Mary Estelle, Philip Herbert, Simon Olando, William Richard, and Russell Edward. Their youngest child, Lula May, was born in Edgecombe County, North Carolina, in 1916. Philip and William appear to have died at an early age.

The Cardwell family moved frequently during the years when the children were young and William Richard was beginning to build his plumbing and pipe fitting business. By the later 1920s, however, Annie Lee fell ill and was compelled to return to Virginia, where she would be cared for by family in the home of her daughter Brunetta "Peggy" Cartos at 326 Bank Street in Norfolk City. According to family tradition, Annie Lee Cardwell died on 14 July 1928, the 12th birthday of her youngest child, Lula May. Her death record states, however, that Annie Lee died on July 15th, the day after Lula May's birthday, perhaps indicating a last act of love on her part.