James was born the son of Thomas Rae and Ruth Cecilia Maughan Williamson. His early education was received in Cache County schools. However, this schooling was cut short when, at the age of fifteen, he left home to go with his uncle, James Hyrum Hall, to Montana where he became an expert driver of a freighting team. When his father died, James had to come home to help provide for the family and operate the farm. Like his father before him, he was a successful farmer. He worked the land, but he never abused it. He always kept his farm equipment in good repair. When he was ready to use it, he didn't have to make last-minute repairs. He loved good horses and cattle and took excellent care of both.
James married Martha McKay Leishman on December 20, 1905 in the Logan Temple. They were the parents of five children. He was gentle, kind, and thoughtful, immaculate in word and deed. He was blessed with musical talents that provided his family and others with great enjoyment. He loved to play his accordion, mouth organ, and saw harp. His tenor voice was soft and mellow. He loved to sing his children to sleep with some of their favorite songs.
Their door was always open. It was a gathering place with plenty to eat and a comfortable bed. Their home was often referred to by the children as the "Williamson Hotel." James furnished a wagon and team of well-groomed horses to convey the dead from the church to the cemetery following a funeral. He and Martha worked on the Old Folks' Committee and built many floats for celebrations. At one time, he drove a stagecoach from Dillon, Montana, to West Yellowstone. They carried passengers and, once in a while, large sums of money and mail. On several occasions he had conflicts with Indians and outlaws. When ill health forced him to give up the farm, it was a sad day for the family. He lived a life of honor, hard work, and shared this with his fellowmen. He gave life his very best.