James Garner was the son of James and Mary (Brice) Garner, who were living in Union County, South Carolina, at the time when he was born. By 1830, it appears that the Garner family were living in Gwinnett County, Georgia; and around 1844, son James married Margaret Lanier. They settled on a farm in the county's Berkshire District, and had a very large family.

During the Civil War, James Garner served as a 2nd Sergeant in Company B, 42nd Georgia Infantry. The Georgia 42nd Infantry Regiment was assembled at Camp McDonald, Georgia, in March, 1862, with men from Gwinnett, De Kalb, Newton, Walton, Fulton, and Calhoun counties. It fought at Chickasaw Bayou and Champion's Hill, but in July 1863 was captured at Vicksburg. Exchanged and brigaded, the regiment continued the fight in various battles from Missionary Ridge to Bentonville. By the end of the war, when the 42nd surrendered with the Army of Tennessee, the 42nd counted just 5 officers and 126 men.

James survived it all and returned to his farm in Gwinnett County, where he lived out his life. He appeared to be a prosperous man. On 26 June 1879, he wrote a long, thoughtful will in which he gave directions for "my property and estate of every description whether personal, real or mixed." He made sure his "well beloved wife Margaret Garner and our minor children or those remaining in the home" were provided for. His will named the following children who were alive at the time: "Eliza J. McDaniel, wife of A. W. McDaniel, R. Pinkney Garner, Mary A. Massey, wife of Winship S. Massey, Sarah L. Mathews, wife of Noah Mathews, James A. Garner, Georgia E. Pursell, wife of David N. Pursell, Darling L. Garner, Margaret E. Garner, Martha L. Garner and Eli S. Garner."

In his will, James also ordered that "my body be buried in a decent and Christian-like manner in my family burying ground by my children who have gone before, and my grave marked after the manner and style of theirs." The children who had "gone before" were an unnamed infant, John B. Garner, and William Thomas Garner. James also wrote, "It is my will and desire that my beloved wife be buried by my side and in the manner and style heretofore indicated for myself, unless she should otherwise direct."