Married Matilda (Middleton) Riley on December 10, 1818, while acting as guardian of Matilda's younger brother. Isaac was 44 and Matilda was 18 at the time.
Isaac owned a plantation (currently located at 11420 Old Georgetown Road, Rockville, MD) on which a slave named Josiah Henson lived. Josiah wrote an autobiography about his life during slavery. His life story became the basis of the book written by Harriet Beecher Stowe called "Uncle Tom's Cabin." Josiah fled to Canada with the Underground Railroad and is buried in Dresden, Ontario, Canada.
From the AUTOBIOGRAPHY OF THE REV. JOSIAH HENSON: “My brothers and sisters were bid off first, and one by one, while my mother, paralyzed by grief, held me by the hand. Her turn came, and she was bought by Isaac Riley, of Montgomery county. Then I was offered to the assembled purchasers. My mother, half-distracted with the thought of parting forever from all her children, pushed through the crowd, while the bidding for me was going on, to the spot where Riley was standing. She fell at his feet, and clung to his knees, entreating him in tones that a mother only could command, to buy her baby as well as herself, and spare to her one, at least, of her little ones. Will it, can it be believed that this man, thus appealed to, was capable not merely of turning a deaf ear to her supplication, but of disengaging himself from her with such violent blows and kicks, as to reduce her to the necessity of creeping out of his reach, and mingling the groan of bodily suffering with the sob of a breaking heart? As she crawled away from the brutal man, I heard her sob out, “Oh, Lord Jesus, how long, how long shall I suffer this way?” I must have been then between five and six years old.”… “I faithfully served Riley for many years. He was coarse and vulgar in his habits, and unprincipled and cruel in his general deportment. His slaves had little opportunity for relaxation from wearying labour, were supplied with the scantiest means of sustaining their toil by necessary food, and had no security for personal rights. When such a master is a tyrant, the slaves often become cringing, treacherous, false, and thieving. Riley and his slaves were no exception to the general rule, but might be cited as apt illustrations of the nature of the relation.”