James was born to parents John Chadwick and Emily Tracy Tooth. He was the last of five children. His older sister, Emily Othello Tooth (1908-1909), died at the age of 8 months. His three other siblings were, Zelda (1907-1915), John Edward (1909-1915) and Glena (1911-1915).
Jame's parents secured a baby sitter to watch over him and his three siblings on the evening of Friday, August 20, 1915. His parents went to a nearby town to attend a dance. While at the dance, they received notice and quickly returned home to find their home had caught on fire. The babysitter, James and his brother and sisters all died in this tragic fire.
Having buried the remainder of their children, his parents had no more. The family home and fire took place in the town of Churchill, Idaho which is no longer in existence.