The land for Oak Grove Cemetery was purchased from Dr. Alexander Read (6 acres) and Bethel Pennyman (2 acres) for $1300. Additional funds were allocated, which included $770 for the construction of a stonewall surrounding the cemetery and $330 was spent laying out the making of carriage way and paths. The original entrance was on Smith Street, which has formal granite pillars. The cemetery was opened on October 6, 1843.

In 1868 an additional 14 acres were purchased and laid out. In 1887 another large tract of land was purchased across Parker Street and developed into additional burial space. Today the cemetery is 100 acres in size. Oak Grove Cemetery is the first publicly-owned cemetery in New Bedford. It contains an exclusively African American section as well as narrow pathways, some meant for carriages while others just for foot traffic. Many of the lots were purchased for entire families, which are set off by granite curbing. A number of these curbings would originally have had a surrounding wrought fence with an entrance gate. This was meant to keep the spirits in their place at rest so they would not wander and disturb the living.

Several graves as well as their monuments were moved here as the death dates predate the establishment of the cemetery. Many of these early stones are very plain, or perhaps it was the Quaker background of the owners that influenced their final resting place. Most of the cemetery is laid out in the then-fashionable rural cemetery style of winding lanes, although the northernmost section has a more open layout, made partly due to complaints about the cluttered nature of the rural cemetery style.