In 1917, a group of wealthy men including J.P. Morgan Jr., Robert Lovett, and Charles Coffin bought land next to the Locust Valley Reformed Church. Their hope was to create a special place for a burial, worthy of the wealth they had acquired in life. Acquisition of substantial acreage for this cemetery was hastened in 1922, with the need to find a suitable burial site for Henry Davison, an influential banker and leader of the international League of Red Cross Societies during World War I. The Locust Valley Cemetery Association, led by Edward Stettinius, Sr., negotiated for six acres of hilly wooded land north of the historic Reformed Church. James Dawson and Percival Gallagher from Olmsted Brothers were hired to design a road through the thick undergrowth and lay out potential burial sites, some as large garden rooms integrated into the topography, and others on a smaller grid. 

Echoing the landscape aesthetic established by the De Forest site at The Memorial Cemetery of St. John's Church, the Olmsted design for the half-acre Davison lot modulated variable grades with native stone walls and steps to create a ‘room' centered around an open lawn, with alcoves for flat burial tablets, canopied by trees and enclosed by banks of flowering shrubs planted in drifts. At Addition 2, Lot 120 a path through textured plantings led to steps descending to a catacomb, embellished with a Tiffany window. Other ‘rooms' in this group were later subdivided into smaller spaces.

The Olmsted firm generated twelve private design commissions for family rooms, Stettinius later consulted with Olmsted employee Carl Rust Parker to rearrange the landscape into smaller plots to meet public demand. By 1931 the firm had designed a second addition to the northeast, consisting of varied smaller lots, accessed from a curving loop road or wide paths. To continue the garden room ideal, belts of laurel and rhododendron enclosed the individual plots, most intended to hold four tombs with little open space. Planning for a third addition abutting the west side of the site began in 1936; the firm shaped this gently sloping narrow strip into more traditional single plots arranged in rows separated by plantings.

The Olmsted firm had a significant impact on overall planning as the cemetery expanded by the 1940s to its present 32 acres. Today, this non-sectarian cemetery is maintained by the Association, in which all plot holders are members.