James Whitaker was born in Fairview, Buncombe County, North Carolina, the sixth child of William Whitaker, Sr. and Mary Cannaday. He married Melinda Fishel in 1824. They settled on his father's farm and they had the first of their six children one year later.

In 1834 he and his wife joined other Fairview families who had moved to Davies, County, Missouri. In 1836 his wife Melinda died, leaving him with six small children.

In 1838 he married 17 year old Nancy Woodland, who was the daugher of John and Celia Steepleford Woodland. Davies County, Missouri was the home to many Mormons, including the Woodlands, and James converted to the Mormon Church. They lived in Davies County for two years until they and other Mormons were driven out by a mob who attacked and killed Mormons at the "Massacre of Haun's Hill". in 1839 they and thousands of other Mormons moved to Hancock County, Illinois, where they founded the town of Nauvoo. Nauvoo became a prosperous city, but by 1843 some Illinois residents became jealous of their growing political power and they were once again attacked. James and Nancy were camped on the banks of the Mississippi River on the night of January 14th, 1844 when their son Leander was born. They crossed the river and camped at the Mormon's winter quarter at Council Bluffs, Iowa. The Mormon leader Brigham Young asked James, a wheelwright by trade, to stay in Council Bluffs to repair and build wagons for the Mormon exodus to Utah. In 1850 James and Nancy, along with their children and other family members, joined the William Snow Wagon Train Company, arriving in the Great Salt Lake Valley in September 1850.

James and Nancy settled in Willard, Box Elder County, Utah. He built the first molasses mill in Willard and became a prominent farmer and rancher in the area. In 1865 James and his son Leander went to Marsh Valley, Idaho and erected the first sawmill in the area. Leander remained in Idaho, but James sold his interest in the mill and returned to Willard where he lived until his death in 1892. His wife Nancy outlived her husband by another 28 years, living into her 100th year.