MongoDB Production Operations Checklist with Practical Examples

Intro

Running MongoDB in production is straightforward if you standardize a few decisions and script the routine work. This guide gives you a practical checklist with copy-paste examples for:

• Configuration and replica sets
• Security hardening
• Monitoring and troubleshooting
• Backups and reliable restores
• Maintenance routines
• Upgrades and rollouts
• Common pitfalls to avoid

Examples use mongosh, systemd, and Node.js/Express so you can adapt them quickly.

---

Workflow Overview

A clear, repeatable workflow keeps production stable:

1. Plan

• Define SLOs: p95 latency, availability, RPO/RTO.
• Capacity: working set vs RAM, IOPS, growth rate.

1. Configure

• Apply a baseline mongod.conf and systemd unit.
• Use a 3-node replica set minimum.

1. Secure

• Enable authorization, create RBAC users, require TLS.
• Restrict network access and rotate secrets.

1. Connect apps

• Safe driver settings: pooled connections, timeouts, majority writes.

1. Observe

• Dashboards and alerts: replication lag, cache, slow ops, disk.

1. Protect data

• Logical + snapshot backups. Practice restores.

1. Maintain

• Health checks, index reviews, log rotation, targeted validation.

1. Upgrade

• Rolling upgrades with backups and feature compatibility gating.

1. Iterate

• Start small, measure, expand.

---

Configuration Baseline

Use consistent, version-controlled config and OS settings.

Example /etc/mongod.conf:

storage:
  dbPath: /var/lib/mongo
  journal:
    enabled: true
  wiredTiger:
    engineConfig:
      cacheSizeGB: 4
net:
  port: 27017
  bindIp: 127.0.0.1,10.0.0.10
security:
  authorization: enabled
replication:
  replSetName: rs0
setParameter:
  diagnosticDataCollectionEnabled: true

Example systemd unit /etc/systemd/system/mongod.service:

[Unit]
Description=MongoDB Database Server
After=network.target

[Service]
User=mongodb
Group=mongodb
ExecStart=/usr/bin/mongod --config /etc/mongod.conf
LimitNOFILE=64000
Restart=on-failure
Type=notify

[Install]
WantedBy=multi-user.target

Replica set init (replace hosts with your actual IPs):

mongosh --host 10.0.0.10 --eval 'rs.initiate({_id:"rs0", members:[{_id:0, host:"10.0.0.10:27017"},{_id:1, host:"10.0.0.11:27017"},{_id:2, host:"10.0.0.12:27017"}]})'

OS and filesystem notes

• Ensure vm.swappiness=1 for database nodes (add to /etc/sysctl.d/99-mongodb.conf and run sysctl --system).
• Use XFS or ext4 with noatime mount option.
• Provision IOPS headroom; avoid noisy neighbors on shared storage.
• Set vm.max_map_count=262144 and file descriptors (ulimit -n 64000) high enough for workload.

---

Security Hardening

1. Enable auth and create users

mongosh --host primary-host <<'EOF'
use admin
db.createUser({
  user: "siteRootAdmin",
  pwd: "REDACTED-STRONG-PASSWORD",
  roles: [ { role: "root", db: "admin" } ]
})
use appdb
db.createUser({
  user: "appUser",
  pwd: "REDACTED-App-Pass",
  roles: [ { role: "readWrite", db: "appdb" } ]
})
EOF

1. Require TLS

In mongod.conf:

net:
  tls:
    mode: requireTLS
    certificateKeyFile: /etc/ssl/mongo/server.pem
    CAFile: /etc/ssl/mongo/ca.pem

1. Network controls

• Bind to explicit IPs only (not 0.0.0.0).
• Restrict ingress to app and admin subnets via security groups or firewall rules.
• Use separate security groups for client traffic (port 27017) and replication traffic if possible.

1. Secrets and keys

• Rotate DB passwords and TLS certs every 90 days.
• Avoid storing credentials in app repos; use environment variables or a secrets manager (e.g., HashiCorp Vault, AWS Secrets Manager).

---

Observability and Monitoring

Key signals to watch

• Replication: rs.status(), replication lag < your RPO (e.g., < 10 seconds).
• WiredTiger cache: cache used vs configured; evictions rising with high latency signals memory pressure.
• Connections: spikes can indicate leaks or storms.
• Operation mix: insert/update/delete/query rates.
• Queues and locks: active readers/writers, lock percentage during spikes.
• Disk: IOPS, latency, free space, checkpoint times.
• Slow queries: capture and fix top offenders.

Quick tools

# High-level counters
mongostat --rowcount 5
# Per-collection activity
mongotop 5

Server-side profiling (use sparingly)

// In mongosh, enable slow op threshold 200ms
db.setProfilingLevel(1, { slowms: 200 })
// Later, disable
db.setProfilingLevel(0)

Alert suggestions

• Replication lag > 10s (tune to your RPO).
• Cache dirty/used approaching 90% with rising latency.
• Disk free < 15% or filesystem almost full.
• Connections near pool limit.
• Oplog window < restore window requirement.

Logging

• Keep log files on separate volume from data if possible.
• Rotate with logRotate command and system logrotate (daily, keep 14 days, compress).

---

Backups and Restore

Combine logical and snapshot strategies.

1. Logical dumps for portability

# Replica set-aware logical dump with oplog for PITR window
mongodump \
  --uri "mongodb://appUser:REDACTED@rs0-1,rs0-2,rs0-3/admin?replicaSet=rs0" \
  --out /backups/$(date +%F) \
  --oplog

Restore test:

mongorestore \
  --uri "mongodb://restoreUser:REDACTED@localhost:27017/admin" \
  --dir /backups/2026-07-10 \
  --oplogReplay \
  --drop

1. Filesystem snapshots for speed (on a secondary)

# Quiesce the secondary for a clean snapshot
mongosh --host rs0/secondary-host --eval 'db.fsyncLock()'
# Create and mount LVM snapshot (example; adapt to your storage)
lvcreate -L 20G -s -n mongo-data-snap /dev/vg0/mongo-data
mount -o ro /dev/vg0/mongo-data-snap /mnt/mongo-snap
# Copy snapshot off-host
rsync -aH --numeric-ids /mnt/mongo-snap/ /backup-target/mongo/
# Cleanup
umount /mnt/mongo-snap
lvremove -f /dev/vg0/mongo-data-snap
mongosh --host rs0/secondary-host --eval 'db.fsyncUnlock()'

Backup policy checklist

• Define RPO and RTO targets.
• Store backups offsite and encrypt at rest.
• Keep at least one recent full backup verified by restore.
• Automate restore tests (e.g., nightly into a disposable environment) and validate counts and checksums.

---

Maintenance and Routine Ops

Weekly

• Review slow queries; add or adjust indexes.
• Check replication lag trends and oplog window.
• Verify backup job success and last tested restore timestamp.

Daily

• Inspect dashboards for error spikes and storage growth.
• Rotate logs if needed.

Monthly

• Validate a subset of collections:

// Sample validation; avoid heavy full-db validate during peak
db.getCollectionNames().slice(0,5).forEach(c => printjson(db.getCollection(c).validate({ full: false })))

Index care

• Prefer compound indexes that match query patterns.
• Drop unused indexes after verifying via $indexStats or db.collection.aggregate([{$indexStats:{}}]).

Operational safety

• Avoid running compact or full validation during peak usage.
• Schedule heavy tasks with rate limits (e.g., rs.syncFrom() for initial sync throttling).

---

Upgrades and Rollouts

Prepare

• Read release notes for breaking changes.
• Verify drivers are compatible with target version.
• Back up and test a restore.

Rolling upgrade (replica set)

1. Upgrade a secondary

systemctl stop mongod
# ... upgrade binaries via your package manager ...
systemctl start mongod
# Wait for SECONDARY state and full replication catch-up

1. Repeat for all secondaries

1. Step down primary and upgrade it

mongosh --host primary --eval 'rs.stepDown(60)'
# Upgrade and restart primary node

1. Feature compatibility version (after all nodes upgraded)

// Set FCV to the new version after stability checks
db.adminCommand({ setFeatureCompatibilityVersion: "7.0" })

Observability during upgrade

• Watch replication lag, elections, and client errors.
• Pause 5-10 minutes between nodes to ensure stability.

Rollback plan

• If errors rise, stop and roll back upgraded nodes to previous version using your package manager.
• Keep FCV at previous version until rollback complete.

---

Local Pilot Plan

Start small, measure, then expand.

Scope

• A 3-node test replica set (containers or VMs).
• One Express API with 2-3 endpoints.
• Minimal dashboards for core metrics.
• One backup job and one automated restore test.

Steps

1. Provision

• Launch 3 nodes and apply the baseline mongod.conf and systemd.
• Initiate the replica set.

1. Secure

• Create admin and app users, enable TLS.

1. Connect an Express API

// app.js
const express = require('express');
const { MongoClient } = require('mongodb');

const uri = 'mongodb://appUser:[email protected],10.0.0.11,10.0.0.12/appdb?replicaSet=rs0&tls=true';
const client = new MongoClient(uri, {
  maxPoolSize: 50,
  minPoolSize: 5,
  waitQueueTimeoutMS: 5000,
  serverSelectionTimeoutMS: 5000,
  socketTimeoutMS: 20000,
  retryWrites: true,
  w: 'majority',
  wtimeoutMS: 5000
});

const app = express();
app.use(express.json());

app.get('/healthz', async (req, res) => {
  try {
    await client.db('admin').command({ ping: 1 });
    res.status(200).send('ok');
  } catch (e) {
    res.status(500).send('not-ok');
  }
});

app.post('/items', async (req, res) => {
  try {
    const r = await client.db('appdb').collection('items').insertOne(req.body);
    res.status(201).json({ id: r.insertedId });
  } catch (e) {
    res.status(500).json({ error: 'insert_failed' });
  }
});

(async () => {
  await client.connect();
  app.listen(3000, () => console.log('API on :3000'));
})();

1. Back up and restore

• Run mongodump with --oplog.
• Restore into a throwaway MongoDB and run a quick data validation.

1. Failure drills

• Kill the primary and confirm failover completes within target.

Pilot success criteria

• p95 API latency within target under nominal load.
• Automatic failover time within target (e.g., < 30 seconds).
• Successful restore completed within RTO and data validated.

---

Common Production Pitfalls

• No auth or TLS: enable authorization and require TLS.
• Single-node deployments: use at least 3 nodes for high availability.
• Unsafe writes: avoid w:1 without journaling; prefer w: "majority" and journaled writes.
• Missing indexes: slow queries and high CPU; add indexes for filter and sort fields.
• Huge documents: MongoDB has a 16 MB document size limit; redesign schema or use GridFS where needed.
• Hot primary: all reads on primary; consider appropriate read preferences for read-heavy workloads.
• Small oplog: replication falls behind and point-in-time recovery window shrinks; size oplog based on write rate (e.g., 10-20% of data volume).
• Untested backups: a backup you have not restored is not proven; schedule restore tests.
• Uncontrolled growth: TTL indexes for ephemeral data; archive old collections.
• Upgrades without FCV planning: set and verify feature compatibility at the right time.

Quick checks

# Auth on?
mongosh --eval 'db.runCommand({ connectionStatus: 1 })'
# Replication healthy?
mongosh --eval 'rs.status()'
# Slow ops?
mongosh --eval 'db.getSiblingDB("admin\)).system.profile.find().sort({ts:-1}).limit(5)'

---

Conclusion

You now have a practical, end-to-end checklist to run MongoDB in production: a hardened configuration, strong security, actionable monitoring, reliable backups with tested restores, safe maintenance routines, and controlled upgrades. Start with the local pilot, measure failover and restore times, fix gaps, and then scale the same patterns to staging and production.