Son of Dr. Gustavus Richard Brown of Rose Hill, Port Tobacco, Maryland. Grandson of Dr. Gustavus Brown of Scotland and Maryland. Nephew of Margaret Stone. He rented part of Haberdeventure from Mildred (Stone) Daniel, daughter of Thomas and Margaret Stone. He purchased part of the plantation from her in 1831.