Joe worked in the steel mills of Ohio until they started closing after the turn of the Century. When there was no work at home, he and his brothers would "hobo" on the rails riding west on freight trains to work in the wheat fields or south looking for whatever work could be found. Crew, VA was a significant rail center with Blackstone nearby to the east along the rail line. It was here that he and brother Sam arrived around 1912 looking for work. At about this same time their younger brother Leroy had died from complications of an operation, followed by the death of his mother three months later. Joe's father, Robert Keller, then moved from Ohio to Nottoway to live with his sons. Here Joe met and then married Edna Clay on the first day of December 1915 and settled down to start a family. A year later their first child was born and was named after the brother who had died. His middle name came from Edna's father's name. Over the next five years three more children were born. Late comers were Harry who was born in 1924 and finally Helen born in 1929. Two years later, in 1931, Joe's brother Sam died, followed by his father's death in 1933 at the age of 85.

While raising his family, Joe tried his hand at running a bowling alley in South Hill and a sawmill in Nottoway Co. When WWII began, the army opened Camp Pickett in nearby Blackstone, VA. This made it difficult to find help for the sawmill so Joe sold the farm in 1942 and moved to a 30 acre farm off of Hull Street Road in Chesterfield County. Also influencing his decision to move was to be closer to Richmond so his three sons who were in the military could more easily get home to visit. After the war he added on a room for Leroy and Maria, but they turned it down, not wanting to live as far out in the country as it was in those days. Initially he tried running a sawmill over in Curles Neck, but found labor a problem there as well. He then went to work for Richmond Engineering making large tanks. To do so he had to read blueprints, so when his eyesight began to fail he had to give that up. He and Edna then gardened and built chicken houses in order to raise chickens for Southern States Cooperative. Southern States would furnish the baby chicks and they would feed and care for them until they matured. Southern States would then pick them up and replace them with new chicks. There never seemed to be a shortage of fried chicken for Sunday dinners.

In 1957 they sold the big farm house and built a smaller one story house on the eastern part of the farm. They had earlier given Arthur about 8 acres along Hull Street on which he built a small home. In 1954 when Louise wanted to move closer to Richmond, he sold his brother Harry 4.48 acres of his property on which he and Shirley built the home in which they lived. Joe also sold Joe, Jr. land on which to build but he eventually moved closer to town as well. The neighborhood is now called Kellersley in his honor.

In his last years, Joe suffered a stroke, fell and broke his hip. He never really recovered and was hospitalized until his death.