Albert was one of 7 children of Franklin Johnson Brown and Margaret A. Lambeth (Margaret was full Cherokee). He was probably born in North Carolina or Texas. He married Fronie Alcorn (about 1862 - 1880), who died in childbirth. He married Henrietta "Nettie" Johnson (1866 - 1902) who died young at about age 36. Albert later married Ella Diddle.