James Franklin Nelson was born in Tennessee about 1823 and was married to Hannah Sophie Bray about 1842. Hannah was the daughter of John Bray and Hetty Bower and was born February 19, 1846. The Bray's were in Hot Spring County, Arkansas for the 1850 census. The Nelson's emigrated to Arkansas sometime between 1853 and 1857. Hannah and James were the parents of six children, four of whom were born in Tennessee and the last two being born in Arkansas.
The children of James and Hannah were: Althea Jane, John Harrison, William Manley, James Franklin (Jr), Nancy Caroline, and Henry Carrol. On Sept 13, 1864, Hannah died leaving James with six children with five of those being minors. Henry Carrol was only four when his mother died.

On June 10, 1866, James Nelson married his second wife Mrs. Emeline Ewing. Emeline had previously married Reuben Ewing on July 7, 1858 but he had died before 1866. Emeline and Reuben had one child, Amanda Jane. Mary Emeline was born in Georgia in 1833 and was the daughter of Christopher Isaac and Matilda McGinnis. When Emeline married James Nelson she was 10 years his junior. James Nelson and Emeline McGinnis Nelson were the parents of four children: Eddy Sherman, Julia Ann, Mattie Etta, and Benjamin Jefferson. Unfortunately, Mary Emeline died in 1874, the same year her youngest son was born. All of the children of James and Emeline were under the age 9 of when she died.

James Nelson married a third time in Clark County, Arkansas on August 22, 1878, about four years after Emeline had passed away. His third wife was a widow Mrs. Martha E. Bratton. Martha was the widowed sister of Emeline. Martha was born about 1841 was 18 years younger than James Nelson. Martha was married previously to Thomas Green Bratton in about 1860. Martha and Green were the parents of four children before Green died sometime between 1871 and 1878. Martha and Green's children were: Eliza, James (Jim), Victoria, and Albert (Boz). James and Martha did not have any children before her death sometime in 1879. (An inventory of her estate was made in November of 1879. Her brother, James was the administrator of her estate.)

The fourth and final time that James Nelson married was to another widow. James married Delilah Mathis who on their marriage record of September 18, 1879 in Clark County, AR is listed as Delilah Mathews. Delilah was born about 1851 in Arkansas to Thomas Duncan and Sarah McTigret. She was previously married to William Mathis about 1871. William and Delilah had four children before he died about 1877. Delilah had four minor children when she married James Nelson all under the age of 8 and there was a 28 year age difference between her and James. James and Delilah had two children: Arabella (Bella) and Mary Garrett (Garritte). These two children were born in 1880 and 1882. It is probable that Delilah was only a few weeks if not maybe a month pregnant with Garrett when James Nelson died. T J Steed (a son-in-law of James) was appointed to be the administrator of James' estate on February 18, 1882 and Garrett was born in October of 1882. The date of 1882 that comes from tombstone records could be wrong also. In April 1892 a J P Langley had petitioned the court to adopt Belle (age 12) and Elizabeth G (age 11) stating that he had acted as guardian/father to these two children who had no father or mother for the last five years. Joseph Leach who had married Delilah Nelson after James' death contested the adoption and blocked Mr. Langley's petition stating he had married their mother but had not been diligent in his duties to make sure the girls had been taken care of since their mother's death. He had since interviewed numerous people and had ascertained that Mr. Langley had had the girls plowing his fields and working on his farm and doing work not fit for little girls. He petitioned the court to appoint their half-brother, J J Mathis, as their guardian. Delilah must have died about 1885-1887.

James Franklin Nelson was the father of twelve children.