The cemetery is located four miles east of Leesville and two and a half miles north of the north entrance to Fort Polk. The cemetery is adjacent to the Castor Baptist Church and just west of the intersection of LA-468 and LA 184, which goes to Fort Polk. Vehicle access is available only from a driveway directly to LA-468 and not through the church parking lot.

Castor Cemetery is one of the largest rural graveyards in Vernon Parish, and has been enlarged several times over the years. The oldest section is near the Church.