MongoDB troubleshooting with practical examples: a step-by-step implementation guide

Intro

MongoDB is resilient, yet production systems still hit snags: services fail to start, connections time out, queries slow to a crawl, or replicas drift. This guide gives you a verified, step-by-step playbook: what to check first, safe commands to run, how to interpret outputs, when to escalate, and how to roll back without surprise side effects. The emphasis is on quick, repeatable diagnostics and reversible changes you can validate within minutes.

The approach:

• Inventory the environment so you do not chase the wrong problem.
• Collect logs and metrics before changing anything.
• Apply the smallest safe fix, verify, then proceed.
• Keep a rollback plan for every action.

Version and environment inventory

Small version or topology details often explain symptoms. Capture these once per environment and keep them handy during incidents.

• Server and storage basics
• Binary version (on host):

    mongod --version

• Server build info and version (via shell):

    mongosh --eval 'db.serverBuildInfo().version'

• Storage engine:

    mongosh --eval 'db.serverStatus().storageEngine'

• Feature compatibility version (FCV):

    mongosh admin --eval 'db.adminCommand({ getParameter: 1, featureCompatibilityVersion: 1 })'

• Topology
• Replica set status:

    mongosh --eval 'rs.status()'

• Sharded cluster overview:

    mongosh --eval 'sh.status()'

• Logs and service control
• Service status (systemd):

    sudo systemctl status mongod

• Recent logs (last hour):

    sudo journalctl -u mongod -S -1h

• File logs (typical paths):
• Linux: /var/log/mongodb/mongod.log
• Windows: C:\Program Files\MongoDB\Server\7.0\log\mongod.log
• Driver and app details (Node.js example)
• Driver version:

    npm ls mongodb

• Connection string example:

    mongodb+srv://appuser:[email protected]/appdb?retryWrites=true&maxPoolSize=50&serverSelectionTimeoutMS=5000

• Any pooling overrides (Mongoose or native driver) changed recently.
• Host quick checks
• Disk space:

    df -h

• CPU and memory:

    top

• File descriptors (Linux):

    ulimit -n

Safe configuration defaults during triage

Prefer reversible, low-risk actions while diagnosing:

• Write concern: use w: "majority" for critical writes during consistency checks (confirm app expectations first).
• Read preference for non-critical dashboards: primaryPreferred to ride out brief elections; keep critical reads on the primary when consistency matters.
• Avoid global server changes. Prefer session- or command-scoped options.
• Adjust driver timeouts (for example serverSelectionTimeoutMS=5000) rather than server timeouts when isolating DNS or network issues.
• For new indexes on a replica set, use a rolling approach: build on secondaries first, then primary, while monitoring lag.
• Do not run --repair unless you have validated backups and understand tradeoffs. Prefer restore or resync.

Verification and diagnostics (before changing anything)

• Is the server up and reachable?
• Service:

    sudo systemctl is-active --quiet mongod && echo RUNNING || echo STOPPED

• Port open:

    ss -ltnp | grep :27017
    # Windows: netstat -ano | findstr 27017

• From the app host, test ping:

    mongosh "mongodb://hostA:27017/admin?serverSelectionTimeoutMS=3000" --eval 'db.runCommand({ ping: 1 })'

Expected: output includes { ok: 1 }.

• What do the logs say?
• Last 200 lines:

    sudo tail -n 200 /var/log/mongodb/mongod.log

• Look for: authentication failures, WiredTiger (WT) errors, elections, connection storms, slow queries, disk-full messages.

• Connection pressure

  mongosh --eval 'db.serverStatus().connections'

Healthy under moderate load: current should be well below available. If current approaches available, timeouts are likely.

• Replication health (replica sets)

  mongosh --eval 'rs.status()'
  mongosh --eval 'rs.printSecondaryReplicationInfo()'

Expected: members show PRIMARY or SECONDARY, with small lag (seconds) in steady state.

• Slow operations and locks
• Current operations waiting on locks:

    mongosh --eval '
    db.aggregate([
      { $currentOp: { allUsers: true, idleConnections: false } },
      { $match: { op: { $in: ["query","insert","update","delete","getmore"] }, waitingForLock: true } },
      { $project: { op:1, ns:1, secs_running:1, waitingForLock:1, client:1, msg:1 } }
    ]).toArray()'

• Per-collection index access:

    mongosh appdb --eval '
    db.orders.aggregate([
      { $indexStats: {} },
      { $project: { name:1, accesses: "$accesses.ops", since: "$accesses.since" } }
    ]).toArray()'

• Disk and cache
• Disk space (keep 10–20% free):

    df -h

• WiredTiger cache:

    mongosh --eval 'db.serverStatus().wiredTiger.cache'

Watch for high usage near the configured maximum and frequent evictions correlated with slow queries.

Quick symptom map (what to check first)

• App shows ECONNREFUSED: mongod not running or port blocked. Check systemctl status mongod and ss -ltnp | grep 27017.
• Authentication failed: wrong database or authSource mismatch. Try mongosh --authenticationDatabase admin -u appuser -p ....
• Slow queries and high CPU: likely missing index. Run explain("executionStats") for the query.
• Replication lag growing: slow secondary or network. Check rs.printSecondaryReplicationInfo().
• Disk full warnings: logs expanding or temp files. Check df -h, rotate logs, and clear safe temp directories.

Failure modes and recovery

Run a verification step after each change. Stop if the system stabilizes.

1) Server fails to start

Common logs: port in use, permission errors, or WiredTiger metadata errors.

• Confirm service and logs

  sudo systemctl status mongod
  sudo journalctl -u mongod -n 200

• Port conflict: find and stop the conflicting process safely

  ss -ltnp | grep :27017
  sudo fuser -k 27017/tcp

• Data path permissions (Linux defaults)

  sudo chown -R mongodb:mongodb /var/lib/mongo
  sudo chmod 700 /var/lib/mongo

• WiredTiger metadata errors (for example, messages about WiredTiger.turtle)
• Safest recovery: restore from a known-good backup or re-seed from a healthy replica.
• Last resort on a standalone:

    sudo systemctl stop mongod
    sudo cp -a /var/lib/mongo /var/lib/mongo.backup
    sudo -u mongodb mongod --dbpath /var/lib/mongo --repair
    sudo systemctl start mongod

• Verify:

  mongosh --eval 'db.adminCommand({ ping: 1 })'

2) Authentication failures

Symptoms: repeated "Authentication failed"; users missing; wrong authSource.

• Confirm the database that owns the user (often admin):

  mongosh "mongodb://appuser:S3cretP4ss!@hostA/admin" --eval 'db.runCommand({ connectionStatus: 1 })'

• List users (admin scope):

  mongosh admin -u clusterAdmin -p 'AdminP@ssw0rd!' --eval 'db.getUsers()'

• If user is missing, create the least-privileged user:

  mongosh admin -u clusterAdmin -p 'AdminP@ssw0rd!' --eval '
  db.createUser({ user: "appuser", pwd: "S3cretP4ss!", roles: [{ role: "readWrite", db: "appdb" }] })'

• If authSource is wrong, fix the URI:

  mongodb://appuser:S3cretP4ss!@hostA,hostB/appdb?replicaSet=rs0&authSource=admin

• Verify with a read/write test on appdb. Roll back any temporary roles after triage.

3) Connection pool exhaustion (Node.js/Express)

Symptoms: MongoNetworkTimeout, timeouts under load, frequent "connection pool cleared" messages.

• Check pressure:

  mongosh --eval 'db.serverStatus().connections'

• If clients churn connections, right-size and reuse a single client per process:

  const { MongoClient } = require('mongodb');
  const client = new MongoClient('mongodb+srv://appuser:[email protected]/appdb', {
    maxPoolSize: 50,
    minPoolSize: 5,
    waitQueueTimeoutMS: 2000,
    serverSelectionTimeoutMS: 5000,
    retryWrites: true
  });
  module.exports = client; // reuse this instance across the app

• Ensure DNS is stable and SRV records are current. Enable HTTP keep-alive on your API tier to avoid creating new connections per request.
• Verify: timeouts drop, serverStatus().connections stabilizes. Roll back pool sizes if contention worsens.

4) Slow queries and missing indexes

Symptoms: high CPU, slow reads, frequent collection scans.

• Get an explain plan:

  mongosh appdb --eval 'db.orders.find({ customerId: 12345, status: "OPEN" }).explain("executionStats")'

Warning signs: COLLSCAN, high totalDocsExamined, large executionTimeMillis.

• Inspect indexes:

  mongosh appdb --eval 'db.orders.getIndexes()'

• Create a targeted index that matches the query shape:

  mongosh appdb --eval 'db.orders.createIndex({ customerId: 1, status: 1 })'

• Re-run explain; expect IXSCAN with far fewer documents examined. If write overhead increases too much, roll back:

  mongosh appdb --eval 'db.orders.dropIndex({ customerId: 1, status: 1 })'

• In replica sets, reduce impact by building on a secondary first, then primary, watching replication lag.

5) Replication lag and rollbacks

Symptoms: secondaries fall behind; reads from secondaries are stale; rollbacks after failover.

• Identify the lagging member and amount of lag:

  mongosh --eval 'rs.printSecondaryReplicationInfo()'
  mongosh --eval 'rs.status()'

• Check network latency and disk I/O on the lagging node. Temporarily reduce write pressure or throttle batch jobs.
• If a member is far behind, resync it:

  sudo systemctl stop mongod
  sudo cp -a /var/lib/mongo /var/lib/mongo.before-resync
  sudo rm -rf /var/lib/mongo/*
  sudo systemctl start mongod
  # The member performs an initial sync from the primary

• For rollbacks after sudden primary loss, inspect rollback files on the affected node and reconcile at the application level.
• Verify: lag decreases steadily; rs.status() shows healthy states. If resync is lengthy, keep the node non-voting temporarily to protect availability.

6) Disk full, journal, and logs

Symptoms: writes fail; logs show "No space left on device"; WiredTiger throttling.

• Check space (and inodes on Linux):

  df -h
  df -i
  sudo du -sh /var/log/mongodb

• Recovery order:

1. Rotate and compress logs safely:

     sudo logrotate -f /etc/logrotate.d/mongod

1. Clear old diagnostic or temp files in application directories (never remove files from the MongoDB dbPath).
2. Add disk capacity or migrate the dbPath to a larger volume during planned maintenance (clean shutdown, update config, clean start).

• Verify: restore at least 10–20% free space; confirm new writes succeed.

7) Unique index build blocked by duplicates

Symptoms: E11000 duplicate key error when creating a unique index.

• Find duplicates on { email: 1 }:

  mongosh appdb --eval '
  const dupes = db.users.aggregate([
    { $group: { _id: "$email", c: { $sum: 1 }, ids: { $push: "$_id" } } },
    { $match: { c: { $gt: 1 } } },
    { $limit: 20 }
  ]).toArray();
  printjson(dupes);
  '

• Resolve duplicates: merge, delete, or re-key according to application rules.
• Create the unique index once clean:

  mongosh appdb --eval 'db.users.createIndex({ email: 1 }, { unique: true })'

• Verify: index exists; new writes respect uniqueness. If legacy flows break, drop the index and plan a phased migration.

Operations checklist (fast, safe progress)

• Inventory versions, topology, and environment facts.
• Check service status, port, and a basic ping.
• Read the last 200 log lines.
• Check connections and slow operations.
• Verify replication and disk space.
• Apply the smallest safe fix.
• Re-verify and document what changed and why.

Practical commands you can run safely

• Quick health ping:

  mongosh --eval 'db.runCommand({ ping: 1 })'

• Summarize connections:

  mongosh --eval 'db.serverStatus().connections'

• Identify a slow query plan:

  mongosh appdb --eval 'db.orders.find({ status: "OPEN" }).sort({ createdAt: -1 }).limit(10).explain("executionStats")'

• Replication lag summary:

  mongosh --eval 'rs.printSecondaryReplicationInfo()'

• Show indexes for a hot collection:

  mongosh appdb --eval 'db.orders.getIndexes()'

Expected results and how to read them

• Ping returns { ok: 1 }: server reachable; if the app still fails, focus on DNS, TLS, or auth.
• Connections: if current is near available, tune client pools and reduce connection churn.
• Explain plan: prefer IXSCAN with low documents examined; COLLSCAN suggests a missing or mismatched index.
• Replication: lags in minutes or hours need attention. Check disk and network on lagging members.
• WiredTiger cache: sustained high usage with evictions plus slow queries indicates memory pressure or poor indexes.

Failure verification and rollback

Every fix above includes a verification step. If the system does not improve, roll back immediately and gather more evidence.

• Driver tuning: revert pool or timeout values to prior known-good settings.
• Index changes: drop newly added indexes if they degrade writes; consider more selective compound indexes.
• Resync or stepdown actions: restore previous member states if election behavior harms availability.
• Storage repair: if repairs introduce divergence, restore from backup or reseed from a healthy replica.

Conclusion

Effective MongoDB troubleshooting is structured, observable, and reversible. Start by confirming versions and topology, read the most recent logs, and run a handful of high-signal status commands. Apply the smallest safe fix, verify, and only then proceed. The workflows here help you solve the most common failures—startup issues, auth errors, slow queries, connection pool pressure, replication lag, and disk capacity—while minimizing risk and preserving a clear rollback path. Adopt this flow on a single service first, validate outcomes, then standardize it across environments for faster, safer incident response.