Son of "Old Billy" William and Polly Barker Crisp who settled in the Mountain Creek section of what is now Graham County. Old Billy was the first white man known to have lived in what is presently known as the Cheoah Valley. He left the Mountain Creek area after a while and settled in what is now Stecoah.
Joel was one of eleven children. He married Anna Elmina Sumpter/Sumter and they had four daughters and five sons.