Nginx Monitoring and Alerts with Practical Examples

Intro

Nginx is often the front door to your application. When it slows or breaks, everything looks down. This guide shows what to monitor, the metrics and log signals that matter, practical alert rules, dashboards that answer on-call questions fast, and an incident workflow you can run under pressure. It ends with a narrow, measurable local pilot you can run end-to-end before any rollout.

Workflow Overview

A clear, repeatable workflow keeps monitoring changes safe and focused:

Goals and scope

Define top questions to answer in minutes: Is Nginx up? Are clients failing (4xx/5xx)? Is upstream failure causing 502/504? Is latency or capacity the issue?
Choose a small set of signals per question: metrics for availability/capacity, logs for status codes and latency.

Instrumentation

Enable Nginx status for connection metrics (stub_status).
Export metrics to Prometheus via nginx-prometheus-exporter.
Ship and parse access logs for status codes and latency.

Visualization

Build an Overview dashboard for on-call: health, connections, request rate, error rate, latency, and top offenders.

Alerting

Page on symptoms that affect users (5xx rate, Nginx down).
Ticket on capacity trends (active connections rising, nearing limits).
Route to owners with context and a runbook link.

Incident response

Use a simple decision tree: client-side errors vs server errors vs upstream errors vs TLS problems.
Keep rollback and safe reload steps documented and tested.

What to Monitor in Nginx

Core health and capacity

Availability: process up, exporter up.
Connections: active, reading, writing, waiting.
Throughput: requests per second (approximated via accepted/handled deltas).
Configuration reload success and uptime continuity.

Traffic quality

Error rates: 5xx and 4xx from access logs.
Upstream errors: 502, 503, 504.
Client disconnects: 499.
Latency: request_time, upstream_response_time from logs.

Security and TLS hygiene

TLS certificate expiry for public endpoints (probe via Blackbox exporter).

Platform context

Host saturation: CPU, memory, file descriptors, network.

Metrics to Collect (Plain Nginx)

From nginx-prometheus-exporter (scraping stub_status):

nginx_connections_active
nginx_connections_reading
nginx_connections_writing
nginx_connections_waiting
nginx_connections_accepted
nginx_connections_handled
nginx_up

Host metrics (node exporter or equivalent):

node_filesystem_avail_bytes, node_network_receive_bytes_total, node_load1, node_cpu_seconds_total, file descriptor usage.

Kubernetes Ingress note If you use Nginx Ingress Controller, prefer its native metrics such as: nginx_ingress_controller_requests, nginx_ingress_controller_nginx_process_connections, and histogram buckets for latency where available.

Logs to Collect and Parse

Use a log format that emits status, request_time, and upstream_response_time. Example:

log_format main 'remote=$remote_addr host=$host method=$request_method uri=$request_uri '
  'status=$status bytes=$body_bytes_sent referer=$http_referer '
  'ua=$http_user_agent rt=$request_time urt=$upstream_response_time';
access_log /var/log/nginx/access.log main;

Parse into labels and numeric fields for:

status (e.g., 200, 499, 502, 504)
request_time (seconds)
upstream_response_time (seconds; may be - if no upstream)
host, uri prefix, method (for top-N views)

Example Alerts That Matter

Prometheus examples (plain Nginx exporter):

# Nginx process or exporter down
- alert: NginxDown
  expr: up{job="nginx"} == 0
  for: 1m
  labels:
    severity: page
  annotations:
    summary: Nginx down on {{ $labels.instance }}

- alert: NginxExporterDown
  expr: up{job="nginx-exporter"} == 0
  for: 2m
  labels:
    severity: ticket
  annotations:
    summary: Nginx exporter down on {{ $labels.instance }}

# Active connections above baseline for sustained period
# Baseline this threshold per host: watch nginx_connections_active for a week, set alert at ~80% of observed peak
- alert: NginxHighActiveConnections
  expr: nginx_connections_active > 500
  for: 5m
  labels:
    severity: ticket
  annotations:
    summary: High active connections on {{ $labels.instance }}

# Accepted minus handled indicates dropped connections
- alert: NginxConnectionDrops
  expr: rate(nginx_connections_accepted[5m]) - rate(nginx_connections_handled[5m]) > 0
  for: 2m
  labels:
    severity: page
  annotations:
    summary: Connection drops on {{ $labels.instance }} increasing

If you use Nginx Ingress Controller, you can also alert on HTTP codes directly from metrics:

- alert: IngressHigh5xxRate
  expr: sum by (ingress) (rate(nginx_ingress_controller_requests{status=~"5.."}[5m])) > 5
  for: 5m
  labels:
    severity: page
  annotations:
    summary: High 5xx rate on ingress {{ $labels.ingress }}

Loki Ruler example (log-based 5xx for plain Nginx):

# Assuming promtail extracts a 'status' label from access logs
- alert: NginxHigh5xxFromLogs
  expr: sum(rate({job="nginx_access", status=~"5.."}[5m])) > 5
  for: 5m
  labels:
    severity: page
  annotations:
    summary: High 5xx rate in Nginx logs

TLS certificate expiry with Blackbox exporter:

- alert: TLSSoonToExpire
  expr: (probe_ssl_earliest_cert_expiry - time()) < 14 * 24 * 3600
  for: 0m
  labels:
    severity: ticket
  annotations:
    summary: TLS certificate expires soon on {{ $labels.instance }}

Dashboards That Answer On-Call Questions

Overview (first screen)

Nginx up, exporter up
Active connections with states: reading, writing, waiting
Requests per second (derived from accepted/handled deltas)
Error rate: 5xx, 4xx from logs
Latency p50, p95 from logs (request_time)
Top URIs or hosts by 5xx and latency (example Loki query: topk(10, sum by (uri) (rate({job="nginx_access", status=~"5.."}[5m]))))

Capacity and health

Active vs waiting connections trend
Node CPU, memory, network saturation
Connection drops vs time

Traffic quality

499, 502, 504 over time
Upstream latency vs total latency (upstream_response_time vs request_time)

Each panel should have a description with: unit, query, and why it matters. Keep time range shortcuts at 5m, 1h, 24h for triage.

Incident Response Workflow

Fast triage

Is Nginx up? If down, check service manager, recent config changes, and reload safely.
Did 5xx spike? If yes, check 502/504 to decide if upstream is failing or Nginx is saturated.
Is latency high with low error rate? Investigate upstream slowness or network saturation.
Is 499 high? Clients are closing early; check timeouts, CDN behavior, and long-tail endpoints.

Fix patterns

Nginx down: validate config, then reload.

  nginx -t
  systemctl reload nginx
  # or: nginx -s reload

High active connections: raise worker_connections (e.g., from 1024 to 4096), review keepalive_timeout (default 75s, consider 30s for high-traffic), and verify upstream capacity.
502/504: check upstream health, DNS resolution, proxy_connect_timeout / proxy_read_timeout / proxy_send_timeout, and keepalive settings between Nginx and upstreams (proxy_http_version 1.1; proxy_set_header Connection "\);).
High 5xx from app: route error to app owners with top endpoints from logs.
TLS expiry: renew, deploy, and verify with blackbox probe.

Aftercare

Document the trigger, strongest signal, and single change that fixed it.
Add or tune one alert or dashboard to prevent repeat.

Local Pilot Plan

Goal: prove you can see health, errors, and capacity locally. Keep it narrow and measurable.

What you will run

Nginx with stub_status and an access log emitting status and latency
nginx-prometheus-exporter for metrics
Prometheus to scrape exporter (and optional blackbox)
Grafana for dashboards
Optional: Loki + Promtail for log-based 5xx and latency

Nginx Config (Local)

events { worker_connections 1024; }
http {
  log_format main 'remote=$remote_addr host=$host method=$request_method uri=$request_uri '
    'status=$status bytes=$body_bytes_sent rt=$request_time urt=$upstream_response_time';
  access_log /var/log/nginx/access.log main;

  server {
    listen 80;
    location / {
      return 200 'ok';
    }
    location /nginx_status {
      stub_status;
      allow 127.0.0.1;
      deny all;
    }
  }
}

Docker Compose (Local Lab)

services:
  nginx:
    image: nginx:1.25
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./logs:/var/log/nginx
    ports:
      - '8080:80'

  exporter:
    image: nginxinc/nginx-prometheus-exporter:0.11.0
    command: ['-nginx.scrape-uri', 'http://nginx/nginx_status']
    ports:
      - '9113:9113'
    depends_on: [nginx]

  prometheus:
    image: prom/prometheus:v2.54.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./alerts.yml:/etc/prometheus/alerts.yml:ro
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.enable-lifecycle'
    ports:
      - '9090:9090'
    depends_on: [exporter]

  grafana:
    image: grafana/grafana:11.0.0
    ports:
      - '3000:3000'
    depends_on: [prometheus]

  loki:
    image: grafana/loki:2.9.0
    command: ['-config.file=/etc/loki/local-config.yaml']
    volumes:
      - ./loki.yaml:/etc/loki/local-config.yaml:ro
    ports:
      - '3100:3100'

  promtail:
    image: grafana/promtail:2.9.0
    volumes:
      - ./promtail.yaml:/etc/promtail/config.yml:ro
      - ./logs:/var/log/nginx:ro
    command: ['--config.file=/etc/promtail/config.yml']
    depends_on: [loki]

prometheus.yml

scrape_configs:
  - job_name: nginx
    static_configs:
      - targets: ['exporter:9113']

rule_files:
  - /etc/prometheus/alerts.yml

alerts.yml

groups:
  - name: nginx-basic
    rules:
      - alert: NginxDown
        expr: up{job="nginx"} == 0
        for: 1m
        labels: {severity: "page"}
        annotations: {summary: "Nginx down"}

      - alert: NginxHighActiveConnections
        expr: nginx_connections_active > 50
        for: 2m
        labels: {severity: "ticket"}
        annotations: {summary: "High active connections"}

      - alert: NginxConnectionDrops
        expr: rate(nginx_connections_accepted[5m]) - rate(nginx_connections_handled[5m]) > 0
        for: 2m
        labels: {severity: "page"}
        annotations: {summary: "Connection drops increasing"}

promtail.yaml (extract status and latency to labels/fields)

server:
  http_listen_port: 9080
positions:
  filename: /tmp/positions.yaml
clients:
  - url: http://loki:3100/loki/api/v1/push
scrape_configs:
  - job_name: nginx_access
    static_configs:
      - targets: ['localhost']
        labels:
          job: nginx_access
          __path__: /var/log/nginx/access.log
    pipeline_stages:
      - regex:
          expression: '.*status=(?P<status>\d{3}).*rt=(?P<rt>[0-9.]+).*urt=(?P<urt>[0-9.\-]+)'
      - labels:
          status:

Smoke Test

Open Grafana at http://localhost:3000 and add Prometheus at http://prometheus:9090 as a data source.
Create a quick panel with nginx_connections_active.
Generate load: for i in {1..100}; do curl -s http://localhost:8080/; done
Confirm alerts fire in Prometheus for high active connections if you raise the load.
If using Loki, build a panel with query: {job="nginx_access"} and add a stat panel showing 5xx rate: sum(rate({job="nginx_access", status=~"5.."}[5m])).

Exit Criteria for the Pilot

You can see Nginx up, connections, and a clear requests trend.
A 5xx spike from logs is visible in minutes.
Three alerts fire under controlled conditions and stop when the condition clears.

Conclusion

Start with a local pilot that proves you can see availability, capacity, and error signals. Keep thresholds close to your baseline and promote only what you have tested. Expand to upstream health, latency percentiles, and TLS checks as your next steps. Document how to triage Nginx vs upstream issues, and keep reload and rollback commands ready.