James Arnold married Mary Brainerd in 1758 in East Haddam, Middlesex, CT, daughter of Elijah Brainerd (1707-1764) and Phebe Davis (1710-1791).

Captain James Arnold commanded a company of New Hampshire men originating out of Portsmouth, New Hampshire when he was appointed as captain of the "Congress" in 1776. James Arnold who commanded the "Congress" Galley was born in East Haddam, Connecticut on February 24, 1735/36. James Arnold from Haddam married Mary Brainerd in 1758 and they had five (nine) children.

In May of 1776, James Arnold from Haddam was appointed by the General Assembly of Connecticut as a captain of a company in a Connecticut regiment. By the middle of July, he was captain of a company in Colonel Wingate's New Hampshire Regiment that was assigned to the Northern Army in Ticonderoga. There is no clear indication of how he went from a town in Connecticut to a New Hampshire regiment. James Arnold must have recruited the majority of his company in the Portsmouth area and, in fact, may have been selected as a company commander based on his previous seafaring experience. New Hampshire contributed two captains in Arnold's fleet and many members of the crews that manned the vessels also came from New Hampshire.

One of the confusing aspects of the battle is that there were two officers on the "Congress" named Arnold. One was its captain, James Arnold, and the other was the commodore of the fleet, General Benedict Arnold, who started out on the "Royal Savage" but moved to the "Congress" at the last minute.

In 1777, James Arnold, as Master of the New Hampshire Schooner "Mary", obtained a permit to sail to the West Indies. In 1778, he was still Master of the "Mary". There is no further record of James Arnold being in the Revolutionary War either in the infantry or the navy.

After the war, James Arnold moved his family to Vermont in 1783. (Source: Notes on the Captains of the Vessels in the Battle of Valcour Island under Brigadier General Benedict Arnold, by Stephen Darley)

By 1800, it appears he was living in Richmond, VT, as he appears in the 1800 census there.