Ernest was the eldest child of Daniel Webster Frye and Josephine Lunt. He was born and raised on a farm in Maine. He married Hazel Gertrude Robbins on Dec 3, 1911 in Massachusetts. Together, they had 8 children. For most of his life, he was a farmer and raised chickens.