REST API monitoring and alerts with practical examples: practical implementation guide

Reliable REST APIs do three things well: they expose clear health signals, raise timely alerts, and support smooth incident response. This guide shows what to monitor, which metrics matter, how to write alert rules that avoid noise, and how to design logs and dashboards for fast diagnosis. You will also get copy-paste examples in Node.js Express with MongoDB plus external calls to Stripe and the OpenAI API, and a safe Local Pilot Plan to validate everything on your laptop.

Who this is for:

• Developers building or operating REST APIs
• DevOps consultants rolling out production standards
• Technical startup teams who want signal without over-engineering

Outcomes you can expect:

• A concrete checklist of REST API metrics and alerts
• Structured logging that speeds up triage
• Dashboards that show what matters
• A repeatable incident workflow
• A local pilot you can run today

Workflow Overview

A dependable REST API monitoring workflow has six steps:

1. Instrument

• Emit RED metrics per route: Rate, Errors, Duration
• Add resource metrics for CPU, memory, event loop lag
• Wrap dependencies: MongoDB, Stripe, OpenAI
• Use request_id for correlation across logs and metrics

1. Collect

• Expose /metrics for scraping
• Ship JSON logs to your collector (stdout in containers is fine)

1. Alert

• Write a small set of crisp, actionable alerts
• Prefer SLO-based conditions over noisy thresholds

1. Visualize

• Dashboards that answer: is it up, is it fast, who is broken

1. Respond

• Lightweight runbooks, clear owners, fast rollback paths

1. Learn

• After incidents, refine metrics, alerts, and runbooks

What to monitor and why

Core REST API signals

• Availability: 2xx rate vs 5xx, by route
• Latency: p50, p95, p99, by route and method
• Traffic: RPS by route and method
• Errors: 4xx patterns (auth, validation) and 5xx (bugs, timeouts)
• Saturation: CPU, memory, event loop lag, queue length
• Timeouts and retries: count and rate per dependency
• Auth and rate limiting: 401/403/429 trends

Dependencies

• MongoDB: operation latency (find, insert, update), connection pool usage, timeouts, error codes
• Caches/queues: hit rate, queue depth, consumer lag

External providers

• Stripe: success and latency by endpoint, 4xx vs 5xx, 429s, rate limit remaining, idempotency conflicts
• OpenAI API: success, latency, token usage per request, 429s, timeouts, content filter errors where applicable

Metrics and alert rules

Use RED and USE principles:

• RED per route: requests per second, error rate, duration histograms
• USE per resource: CPU utilization, memory saturation, error counts

Example metric names

• http_server_requests_total{method, route, status_code}
• http_server_request_duration_seconds_bucket
• process_cpu_seconds_total, process_resident_memory_bytes
• nodejs_eventloop_lag_seconds (if available)
• mongo_query_duration_seconds_bucket{operation, collection, outcome}
• third_party_request_total{provider, operation, outcome}
• third_party_request_duration_seconds_bucket{provider, operation}
• third_party_rate_limit_exhausted_total{provider}

Example PromQL alert ideas (adapt to your labels)

• High error rate

sum(rate(http_server_requests_total{status_code=~"5.."}[5m]))
  /
sum(rate(http_server_requests_total[5m]))
  > 0.05
for: 10m

• Latency regression (exclude /metrics)

histogram_quantile(
  0.95,
  sum by (le) (rate(http_server_request_duration_seconds_bucket{route!="/metrics"}[5m]))
) > 0.300
for: 10m

• Event loop lag (Node.js)

max_over_time(nodejs_eventloop_lag_seconds[5m]) > 0.050
for: 10m

• MongoDB timeouts or errors

sum(rate(mongo_query_duration_seconds_count{outcome="timeout"}[5m])) > 1
or
sum(rate(mongo_query_errors_total[5m])) > 1

• Provider failures and 429s

sum(rate(third_party_request_total{provider=~"stripe|openai",outcome="error"}[5m])) > 2
or
increase(third_party_rate_limit_exhausted_total{provider=~"stripe|openai"}[15m]) > 0

SLO-based thinking

• Define SLI: success_rate, request_latency
• Set SLO: e.g., 99.5% success, p95 < 300 ms, 24x7
• Alert on budget burn: fast and slow burn multi-window alerts

Logs and structured events

Design JSON logs to answer who, what, where, when, and why:

Include in every request log

• ts: ISO timestamp
• level: info|warn|error
• request_id: UUID set at ingress
• method, route, path, status
• duration_ms
• remote_ip (sanitized), user_agent (optional)
• user_id or tenant_id if available and allowed

Dependency call log fields

• dependency: mongo|stripe|openai
• operation: findOne|charges.create|chat.completions
• outcome: ok|error|timeout|rate_limited
• status or code
• duration_ms
• retry_after_ms or rate_limit_remaining where available

Privacy and safety

• Never log secrets, tokens, card data, or prompts with PII
• Hash or omit user identifiers if unsure
• Sample verbose payload logs at low rates in non-production

Dashboards that matter

Home dashboard

• Top row: success rate, p95 latency, RPS, 5xx by route
• Trend row: p50-p99 latency, error budget burn, deploy markers

Route drill-down

• Route latency percentiles, throughput, 4xx vs 5xx distribution
• Top error messages and stack signatures

Dependencies

• MongoDB: p50-p99 latency per operation, connection pool in-use, timeouts
• Providers: success rate, latency, 429s, rate limit remaining

Saturation

• CPU, memory, event loop lag, GC pauses, container restarts

On-call view

• Open alerts, top offenders, runbook links

Practical examples by stack

The snippets below show a minimal but production-friendly setup for an Express API with Prometheus-style metrics and structured logs. They include MongoDB instrumentation and wrappers for Stripe and OpenAI API calls.

Express setup with metrics and logs

Install

npm i express prom-client pino pino-http uuid

server.js

const express = require('express');
const client = require('prom-client');
const pino = require('pino');
const pinoHttp = require('pino-http');
const { v4: uuidv4 } = require('uuid');

const app = express();
const logger = pino({ level: process.env.LOG_LEVEL || 'info' });

// Prometheus registry and default process metrics
const register = new client.Registry();
client.collectDefaultMetrics({ register });

// RED metrics
const httpDuration = new client.Histogram({
  name: 'http_server_request_duration_seconds',
  help: 'HTTP request duration in seconds',
  labelNames: ['method', 'route', 'status_code']
});
const httpRequests = new client.Counter({
  name: 'http_server_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status_code']
});
register.registerMetric(httpDuration);
register.registerMetric(httpRequests);

// Request ID and logging
app.use((req, res, next) => {
  req.id = req.headers['x-request-id'] || uuidv4();
  res.setHeader('x-request-id', req.id);
  next();
});
app.use(pinoHttp({ logger, genReqId: req => req.id }));

// Timing middleware
app.use((req, res, next) => {
  const end = httpDuration.startTimer({ method: req.method });
  res.on('finish', () => {
    const route = req.route ? req.route.path : req.path;
    const labels = { method: req.method, route, status_code: String(res.statusCode) };
    httpRequests.inc(labels, 1);
    end({ route, status_code: String(res.statusCode) });
    req.log.info({
      msg: 'request_complete', request_id: req.id, method: req.method,
      route, status: res.statusCode, duration_ms: res.responseTime
    });
  });
  next();
});

// Sample routes
app.get('/healthz', (req, res) => res.status(200).json({ ok: true }));

app.get('/api/v1/items/:id', async (req, res) => {
  // Imagine a DB call here
  res.json({ id: req.params.id, name: 'example' });
});

// Metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

// Error handler
app.use((err, req, res, next) => {
  req.log.error({ msg: 'unhandled_error', request_id: req.id, err });
  res.status(500).json({ error: 'internal_error', request_id: req.id });
});

const port = process.env.PORT || 3000;
app.listen(port, () => logger.info({ msg: 'listening', port }));

Notes

• Correlation: x-request-id is echoed back so clients and logs align
• Metrics: route label uses req.route.path when available to avoid cardinality blowups

MongoDB instrumentation

Install

npm i mongodb

mongo.js

const { MongoClient } = require('mongodb');
const client = require('prom-client');

const mongoDuration = new client.Histogram({
  name: 'mongo_query_duration_seconds',
  help: 'MongoDB query duration',
  labelNames: ['operation', 'collection', 'outcome']
});

async function withTiming(op, collection, fn) {
  const end = mongoDuration.startTimer({ operation: op, collection });
  try {
    const result = await fn();
    end({ outcome: 'ok' });
    return result;
  } catch (e) {
    const outcome = /timeout/i.test(e.message) ? 'timeout' : 'error';
    end({ outcome });
    throw e;
  }
}

async function createClient(uri) {
  const cli = new MongoClient(uri, { maxPoolSize: 20 });
  await cli.connect();
  return cli;
}

module.exports = { withTiming, createClient, mongoDuration };

Usage in a route

// in server.js
const { createClient, withTiming } = require('./mongo');

let db;
(async () => {
  const cli = await createClient(process.env.MONGO_URI || 'mongodb://localhost:27017');
  db = cli.db('app');
})();

app.get('/api/v1/items/:id', async (req, res, next) => {
  try {
    const id = req.params.id;
    const doc = await withTiming('findOne', 'items', () => db.collection('items').findOne({ _id: id }));
    if (!doc) return res.status(404).json({ error: 'not_found' });
    res.json(doc);
  } catch (e) { next(e); }
});

Stripe wrapper

Install

npm i stripe

stripe.js

const Stripe = require('stripe');
const client = require('prom-client');

const stripe = new Stripe(process.env.STRIPE_API_KEY || 'sk_test_xxx');

const tpReqs = new client.Counter({
  name: 'third_party_request_total',
  help: 'Third-party requests',
  labelNames: ['provider', 'operation', 'outcome']
});
const tpDur = new client.Histogram({
  name: 'third_party_request_duration_seconds',
  help: 'Third-party request duration',
  labelNames: ['provider', 'operation']
});
const tpRateExhaust = new client.Counter({
  name: 'third_party_rate_limit_exhausted_total',
  help: 'Third-party rate limit exhausted',
  labelNames: ['provider']
});

async function createCharge(params) {
  const op = 'charges.create';
  const end = tpDur.startTimer({ provider: 'stripe', operation: op });
  try {
    const res = await stripe.charges.create(params);
    tpReqs.inc({ provider: 'stripe', operation: op, outcome: 'ok' });
    if (res.headers && Number(res.headers['stripe-rate-limit-remaining']) === 0) {
      tpRateExhaust.inc({ provider: 'stripe' });
    }
    end();
    return res;
  } catch (e) {
    const outcome = e.statusCode === 429 ? 'rate_limited' : 'error';
    tpReqs.inc({ provider: 'stripe', operation: op, outcome });
    if (e.statusCode === 429) tpRateExhaust.inc({ provider: 'stripe' });
    end();
    throw e;
  }
}

module.exports = { createCharge };

OpenAI API wrapper

Install

npm i openai

openai.js

const { OpenAI } = require('openai');
const client = require('prom-client');

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY || 'sk-xxx' });

const tpReqs = new client.Counter({
  name: 'third_party_request_total',
  help: 'Third-party requests',
  labelNames: ['provider', 'operation', 'outcome']
});
const tpDur = new client.Histogram({
  name: 'third_party_request_duration_seconds',
  help: 'Third-party request duration',
  labelNames: ['provider', 'operation']
});
const tpRateExhaust = new client.Counter({
  name: 'third_party_rate_limit_exhausted_total',
  help: 'Third-party rate limit exhausted',
  labelNames: ['provider']
});

async function createChatCompletion(msgs) {
  const op = 'chat.completions';
  const end = tpDur.startTimer({ provider: 'openai', operation: op });
  try {
    const res = await openai.chat.completions.create({ model: 'gpt-4o-mini', messages: msgs });
    tpReqs.inc({ provider: 'openai', operation: op, outcome: 'ok' });
    // Simulate detection of 429 based on response headers if available
    end();
    return res;
  } catch (e) {
    const code = e.status || e.code || 0;
    const outcome = code === 429 ? 'rate_limited' : 'error';
    tpReqs.inc({ provider: 'openai', operation: op, outcome });
    if (code === 429) tpRateExhaust.inc({ provider: 'openai' });
    end();
    throw e;
  }
}

module.exports = { createChatCompletion };

Add a route that uses the wrappers

// in server.js
const { createCharge } = require('./stripe');
const { createChatCompletion } = require('./openai');

app.post('/api/v1/payments', express.json(), async (req, res, next) => {
  try {
    const charge = await createCharge({ amount: 100, currency: 'usd', source: 'tok_visa' });
    res.json({ id: charge.id, status: charge.status });
  } catch (e) { next(e); }
});

app.post('/api/v1/assist', express.json(), async (req, res, next) => {
  try {
    const { prompt } = req.body;
    const chat = await createChatCompletion([{ role: 'user', content: prompt }]);
    res.json({ reply: chat.choices?.[0]?.message?.content || '' });
  } catch (e) { next(e); }
});

Incident response workflows

Make alerts actionable

• Title: what broke and where (route, provider)
• Symptoms: error rate or latency details
• Links: route dashboard, recent deploys, runbook

On-call checklist

• Acknowledge the alert, assign a single owner
• Check the home dashboard and the affected route
• Confirm user impact. If severe, rollback recent change
• If external provider is failing, degrade gracefully or queue work
• Add request_id from a failing log line to trace downstream calls
• After fix, verify metrics, then write a short summary and next steps

Local Pilot Plan

Goal

• Validate metrics, logs, and a few alert formulas locally with minimal scope

Scope

• Two routes: a fast GET and a POST that touches MongoDB
• One Stripe wrapper call and one OpenAI wrapper call (can be stubbed)
• Collect default and RED metrics, expose /metrics

Steps

1. Instrument

• Add the Express middleware, MongoDB timing, and third-party wrappers

1. Simulate traffic

# Fast route
hey -z 30s -c 10 http://localhost:3000/healthz
# DB route
hey -z 30s -c 5 http://localhost:3000/api/v1/items/123

1. Inspect metrics

curl -s localhost:3000/metrics | grep http_server_request_duration_seconds

1. Trigger errors and 429s

• Point Stripe key to a test key and induce validation errors
• For OpenAI, simulate 429 by temporarily failing calls in code when a header flag is set

1. Validate log fields

• Ensure request_id, route, status, duration_ms, dependency fields appear

1. Draft alert expressions

• Paste example PromQL into your alerting tool and verify series exist

Success criteria

• You can see p95 latency and 5xx rate by route
• You can trace a failing request across app, MongoDB, and provider logs via request_id
• A sample alert would have fired under an induced failure

Next

• Roll out the same pattern to one more high-traffic route, then to other services

Conclusion

Monitor what users feel and what systems need. Start small with a local pilot, wire in correlation IDs, emit RED metrics, add a handful of strong alerts, and build dashboards that answer real questions. Practice incident response with lightweight runbooks and clear ownership. Once the pilot proves value, apply the same patterns across your REST API services.