Albert was one of 7 children of Franklin Johnson Brown and Margaret A. Lambeth (Margaret was Cherokee). He was born in Washington County, Texas, according to his death certificate. He married Sophronia "Fronie" Alcorn (about 1862 - 1880). He married Henrietta "Nettie" Johnson (1866 - 1902) who died young at about age 36. Albert later married Ella Diddle.