## Intro

Nginx is often the front door to your application. When it slows or breaks, everything looks down. This guide shows what to monitor, the metrics and log signals that matter, practical alert rules, dashboards that answer on-call questions fast, and an incident workflow you can run under pressure. It ends with a narrow, measurable local pilot you can run end-to-end before any rollout.

## Workflow Overview

A clear, repeatable workflow keeps monitoring changes safe and focused:

- Goals and scope

- Define top questions to answer in minutes: Is Nginx up? Are clients failing (4xx/5xx)? Is upstream failure causing 502/504? Is latency or capacity the issue?

- Choose a small set of signals per question: metrics for availability/capacity, logs for status codes and latency.

- Instrumentation

- Enable Nginx status for connection metrics ( stub_status ).

- Export metrics to Prometheus via nginx-prometheus-exporter .

- Ship and parse access logs for status codes and latency.

- Visualization

- Build an Overview dashboard for on-call: health, connections, request rate, error rate, latency, and top offenders.

- Alerting

- Page on symptoms that affect users (5xx rate, Nginx down).

- Ticket on capacity trends (active connections rising, nearing limits).

- Route to owners with context and a runbook link.

- Incident response

- Use a simple decision tree: client-side errors vs server errors vs upstream errors vs TLS problems.

- Keep rollback and safe reload steps documented and tested.

### What to Monitor in Nginx

Core health and capacity

- Availability: process up, exporter up.

- Connections: active, reading, writing, waiting.

- Throughput: requests per second (approximated via accepted/handled deltas).

- Configuration reload success and uptime continuity.

Traffic quality

- Error rates: 5xx and 4xx from access logs.

- Upstream errors: 502, 503, 504.

- Client disconnects: 499.

- Latency: request_time , upstream_response_time from logs.

Security and TLS hygiene

- TLS certificate expiry for public endpoints (probe via Blackbox exporter).

Platform context

- Host saturation: CPU, memory, file descriptors, network.

### Metrics to Collect (Plain Nginx)

From nginx-prometheus-exporter (scraping stub_status ):

- nginx_connections_active

- nginx_connections_reading

- nginx_connections_writing

- nginx_connections_waiting

- nginx_connections_accepted

- nginx_connections_handled

- nginx_up

Host metrics (node exporter or equivalent):

- node_filesystem_avail_bytes , node_network_receive_bytes_total , node_load1 , node_cpu_seconds_total , file descriptor usage.

Kubernetes Ingress note If you use Nginx Ingress Controller, prefer its native metrics such as: nginx_ingress_controller_requests , nginx_ingress_controller_nginx_process_connections , and histogram buckets for latency where available.

### Logs to Collect and Parse

Use a log format that emits status, request_time , and upstream_response_time . Example:

log_format main 'remote=$remote_addr host=$host method=$request_method uri=$request_uri '
 'status=$status bytes=$body_bytes_sent referer=$http_referer '
 'ua=$http_user_agent rt=$request_time urt=$upstream_response_time';
access_log /var/log/nginx/access.log main; 
 Parse into labels and numeric fields for:

- status (e.g., 200, 499, 502, 504)

- request_time (seconds)

- upstream_response_time (seconds; may be - if no upstream)

- host, uri prefix, method (for top-N views)

### Example Alerts That Matter

Prometheus examples (plain Nginx exporter):

# Nginx process or exporter down
- alert: NginxDown
 expr: up{job="nginx"} == 0
 for: 1m
 labels:
 severity: page
 annotations:
 summary: Nginx down on {{ $labels.instance }}

- alert: NginxExporterDown
 expr: up{job="nginx-exporter"} == 0
 for: 2m
 labels:
 severity: ticket
 annotations:
 summary: Nginx exporter down on {{ $labels.instance }}

# Active connections above baseline for sustained period
# Baseline this threshold per host: watch nginx_connections_active for a week, set alert at ~80% of observed peak
- alert: NginxHighActiveConnections
 expr: nginx_connections_active > 500
 for: 5m
 labels:
 severity: ticket
 annotations:
 summary: High active connections on {{ $labels.instance }}

# Accepted minus handled indicates dropped connections
- alert: NginxConnectionDrops
 expr: rate(nginx_connections_accepted[5m]) - rate(nginx_connections_handled[5m]) > 0
 for: 2m
 labels:
 severity: page
 annotations:
 summary: Connection drops on {{ $labels.instance }} increasing 
 If you use Nginx Ingress Controller, you can also alert on HTTP codes directly from metrics:

- alert: IngressHigh5xxRate
 expr: sum by (ingress) (rate(nginx_ingress_controller_requests{status=~"5.."}[5m])) > 5
 for: 5m
 labels:
 severity: page
 annotations:
 summary: High 5xx rate on ingress {{ $labels.ingress }} 
 Loki Ruler example (log-based 5xx for plain Nginx):

# Assuming promtail extracts a 'status' label from access logs
- alert: NginxHigh5xxFromLogs
 expr: sum(rate({job="nginx_access", status=~"5.."}[5m])) > 5
 for: 5m
 labels:
 severity: page
 annotations:
 summary: High 5xx rate in Nginx logs 
 TLS certificate expiry with Blackbox exporter:

- alert: TLSSoonToExpire
 expr: (probe_ssl_earliest_cert_expiry - time()) < 14 * 24 * 3600
 for: 0m
 labels:
 severity: ticket
 annotations:
 summary: TLS certificate expires soon on {{ $labels.instance }} 

### Dashboards That Answer On-Call Questions

 Overview (first screen)

- Nginx up, exporter up

- Active connections with states: reading, writing, waiting

- Requests per second (derived from accepted/handled deltas)

- Error rate: 5xx, 4xx from logs

- Latency p50, p95 from logs ( request_time )

- Top URIs or hosts by 5xx and latency (example Loki query: topk(10, sum by (uri) (rate({job="nginx_access", status=~"5.."}[5m]))) )

Capacity and health

- Active vs waiting connections trend

- Node CPU, memory, network saturation

- Connection drops vs time

Traffic quality

- 499, 502, 504 over time

- Upstream latency vs total latency ( upstream_response_time vs request_time )

Each panel should have a description with: unit, query, and why it matters. Keep time range shortcuts at 5m, 1h, 24h for triage.

### Incident Response Workflow

Fast triage

- Is Nginx up? If down, check service manager, recent config changes, and reload safely.

- Did 5xx spike? If yes, check 502/504 to decide if upstream is failing or Nginx is saturated.

- Is latency high with low error rate? Investigate upstream slowness or network saturation.

- Is 499 high? Clients are closing early; check timeouts, CDN behavior, and long-tail endpoints.

Fix patterns

- Nginx down : validate config, then reload.

nginx -t
 systemctl reload nginx
 # or: nginx -s reload 

- High active connections : raise worker_connections (e.g., from 1024 to 4096), review keepalive_timeout (default 75s, consider 30s for high-traffic), and verify upstream capacity.

- 502/504 : check upstream health, DNS resolution, proxy_connect_timeout / proxy_read_timeout / proxy_send_timeout , and keepalive settings between Nginx and upstreams ( proxy_http_version 1.1; proxy_set_header Connection "\); ).

- High 5xx from app : route error to app owners with top endpoints from logs.

- TLS expiry : renew, deploy, and verify with blackbox probe.

 Aftercare

- Document the trigger, strongest signal, and single change that fixed it.

- Add or tune one alert or dashboard to prevent repeat.

## Local Pilot Plan

Goal: prove you can see health, errors, and capacity locally. Keep it narrow and measurable.

What you will run

- Nginx with stub_status and an access log emitting status and latency

- nginx-prometheus-exporter for metrics

- Prometheus to scrape exporter (and optional blackbox)

- Grafana for dashboards

- Optional: Loki + Promtail for log-based 5xx and latency

### Nginx Config (Local)

events { worker_connections 1024; }
http {
 log_format main 'remote=$remote_addr host=$host method=$request_method uri=$request_uri '
 'status=$status bytes=$body_bytes_sent rt=$request_time urt=$upstream_response_time';
 access_log /var/log/nginx/access.log main;

 server {
 listen 80;
 location / {
 return 200 'ok';
 }
 location /nginx_status {
 stub_status;
 allow 127.0.0.1;
 deny all;
 }
 }
} 

### Docker Compose (Local Lab)

 services:
 nginx:
 image: nginx:1.25
 volumes:
 - ./nginx.conf:/etc/nginx/nginx.conf:ro
 - ./logs:/var/log/nginx
 ports:
 - '8080:80'

 exporter:
 image: nginxinc/nginx-prometheus-exporter:0.11.0
 command: ['-nginx.scrape-uri', 'http://nginx/nginx_status']
 ports:
 - '9113:9113'
 depends_on: [nginx]

 prometheus:
 image: prom/prometheus:v2.54.0
 volumes:
 - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
 - ./alerts.yml:/etc/prometheus/alerts.yml:ro
 command:
 - '--config.file=/etc/prometheus/prometheus.yml'
 - '--web.enable-lifecycle'
 ports:
 - '9090:9090'
 depends_on: [exporter]

 grafana:
 image: grafana/grafana:11.0.0
 ports:
 - '3000:3000'
 depends_on: [prometheus]

 loki:
 image: grafana/loki:2.9.0
 command: ['-config.file=/etc/loki/local-config.yaml']
 volumes:
 - ./loki.yaml:/etc/loki/local-config.yaml:ro
 ports:
 - '3100:3100'

 promtail:
 image: grafana/promtail:2.9.0
 volumes:
 - ./promtail.yaml:/etc/promtail/config.yml:ro
 - ./logs:/var/log/nginx:ro
 command: ['--config.file=/etc/promtail/config.yml']
 depends_on: [loki] 

### prometheus.yml

 scrape_configs:
 - job_name: nginx
 static_configs:
 - targets: ['exporter:9113']

rule_files:
 - /etc/prometheus/alerts.yml 

### alerts.yml

 groups:
 - name: nginx-basic
 rules:
 - alert: NginxDown
 expr: up{job="nginx"} == 0
 for: 1m
 labels: {severity: "page"}
 annotations: {summary: "Nginx down"}

 - alert: NginxHighActiveConnections
 expr: nginx_connections_active > 50
 for: 2m
 labels: {severity: "ticket"}
 annotations: {summary: "High active connections"}

 - alert: NginxConnectionDrops
 expr: rate(nginx_connections_accepted[5m]) - rate(nginx_connections_handled[5m]) > 0
 for: 2m
 labels: {severity: "page"}
 annotations: {summary: "Connection drops increasing"} 

### promtail.yaml (extract status and latency to labels/fields)

 server:
 http_listen_port: 9080
positions:
 filename: /tmp/positions.yaml
clients:
 - url: http://loki:3100/loki/api/v1/push
scrape_configs:
 - job_name: nginx_access
 static_configs:
 - targets: ['localhost']
 labels:
 job: nginx_access
 __path__: /var/log/nginx/access.log
 pipeline_stages:
 - regex:
 expression: '.*status=(?P<status>\d{3}).*rt=(?P<rt>[0-9.]+).*urt=(?P<urt>[0-9.\-]+)'
 - labels:
 status: 

### Smoke Test

- Open Grafana at http://localhost:3000 and add Prometheus at http://prometheus:9090 as a data source.

- Create a quick panel with nginx_connections_active .

- Generate load: for i in {1..100}; do curl -s http://localhost:8080/; done

- Confirm alerts fire in Prometheus for high active connections if you raise the load.

- If using Loki, build a panel with query: {job="nginx_access"} and add a stat panel showing 5xx rate: sum(rate({job="nginx_access", status=~"5.."}[5m])) .

### Exit Criteria for the Pilot

- You can see Nginx up, connections, and a clear requests trend.

- A 5xx spike from logs is visible in minutes.

- Three alerts fire under controlled conditions and stop when the condition clears.

## Conclusion

 Start with a local pilot that proves you can see availability, capacity, and error signals. Keep thresholds close to your baseline and promote only what you have tested. Expand to upstream health, latency percentiles, and TLS checks as your next steps. Document how to triage Nginx vs upstream issues, and keep reload and rollback commands ready.