Philemon was born in Maryland to Thomas and Rachel Plummer. He married Sarah Holland Harrison, widow of William Harrison on June 10, 1795. They had 6 children. Sarah died in 1817 and Philemon married Mary Henrix on September 7, 1818. They had two children. Philemon died on August 10, 1821.