MinIO monitoring and alerts with practical examples: a production operations guide

Intro

MinIO powers S3-compatible object storage across Docker and Kubernetes. To keep it reliable and fast in production, you need clear visibility into availability, capacity, performance, and data safety, plus a small set of well-tuned alerts and concise runbooks.

This guide focuses on the signals that matter, shows concrete Prometheus queries and starter alert rules, and adds a signal-to-action table, alert tuning tips, a compact incident runbook, and a post-change verification checklist. It ends with a local pilot you can validate before rolling out to production.

What you will get:

• A prioritized monitoring scope and signal-to-action table
• Prometheus queries and copy-ready alert rules with tuning guidance
• Practical log and trace patterns for early warning
• A compact incident runbook and a post-change checklist
• A safe, measurable local pilot plan

What to monitor in MinIO

Focus on user impact first, then root causes and operator actions.

• Availability and API health:
• up{job="minio"}, S3 request rate and error ratio, request duration percentiles (p50, p95, p99)
• Capacity and growth:
• Cluster total/used bytes, free headroom, burn rate, hot buckets/prefixes (use mc admin bucket top to identify hot buckets by request count)
• Cluster and disk health:
• Offline nodes and drives, healing activity/backlog, disk I/O errors
• Performance:
• Read/write throughput, concurrency by operation, cache hit rate (if enabled)
• Security and access:
• AccessDenied spikes, admin operations, TLS handshake errors
• Data workflows:
• Replication lag and failures, batch/ETL throughput changes
• Host saturation (via node exporter or platform metrics):
• CPU, memory pressure, disk latency, network saturation

Signal-to-action table

Use this table to turn top-line signals into first checks and actions. Treat thresholds as starting points; baseline your own environment before enforcing.

Workflow overview

1. Expose metrics from MinIO

• MinIO exposes Prometheus metrics at /minio/v2/metrics/cluster.
• Quick check: curl http://MINIO_HOST:9000/minio/v2/metrics/cluster

1. Scrape with Prometheus

• Add a scrape job targeting your MinIO endpoints.

# prometheus.yml
scrape_configs:
 - job_name: minio
   metrics_path: /minio/v2/metrics/cluster
   static_configs:
   - targets: ["minio:9000\]
     labels:
       cluster: demo

1. Visualize with Grafana

• Build top-row panels for availability, latency, error ratio, and capacity. Use repeating panels by node for distributed clusters.

1. Alert on risks

• Start with capacity, nodes/targets down, high error ratio, and high latency. Expand to replication, cert expiry, and host saturation.

1. Add logs and traces

• Forward JSON logs to your log platform. Use mc admin trace for live S3 call sampling during incidents.

1. Respond with runbooks

• For each alert, document checks, likely causes, and safe mitigations.

Prometheus queries you can reuse

Metric names vary by MinIO version and build. In Prometheus, type minio_ to browse metrics and adjust filters (for example, job="minio").

• Cluster capacity percent used

100 * minio_cluster_capacity_used_bytes / minio_cluster_capacity_total_bytes

• S3 request rate by method (5m)

sum by (method) (rate(minio_s3_requests_total[5m]))

• 5xx error ratio (5m)

sum(rate(minio_s3_requests_errors_total{code=~"5.."}[5m]))
/ ignoring(code)
sum(rate(minio_s3_requests_total[5m]))

• P95 request latency (seconds)

histogram_quantile(
 0.95,
 sum by (le) (rate(minio_s3_request_duration_seconds_bucket[5m]))
)

• Offline nodes (if available)

minio_cluster_nodes_offline_total

• Offline disks (if available)

minio_disks_offline_total

• Replication backlog (example; verify names in your cluster)

sum(rate(minio_bucket_replication_pending_count[5m]))

*Note: Metric names vary by MinIO version; if minio_bucket_replication_pending_count is absent, try minio_bucket_replication_pending_total.

• Certificate expiry (via blackbox exporter scraping MinIO endpoint)

(probe_ssl_earliest_cert_expiry - time()) / 86400

• Host saturation (node exporter)

# CPU busy ratio
1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

# Disk busy time
avg by (instance, device) (rate(node_disk_io_time_seconds_total[5m]))

Starter alert rules

Begin with a minimal, actionable set. Tune thresholds to your SLOs and baseline, not universal values.

# minio.alerts.yml
groups:
- name: minio.alerts
  rules:
  - alert: MinIOClusterCapacityHigh
    expr: (minio_cluster_capacity_used_bytes / minio_cluster_capacity_total_bytes) > 0.85
    for: 15m
    labels:
      severity: warning
    annotations:
      summary: MinIO capacity is high
      description: Capacity exceeds target headroom. Plan cleanup or expansion.

  - alert: MinIOTargetsDown
    expr: count(up{job="minio"} == 0) > 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: One or more MinIO scrape targets are down
      description: Investigate node/pod health and restore redundancy.

  - alert: MinIOHigh5xxErrorRate
    expr: (sum(rate(minio_s3_requests_errors_total{code=~"5.."}[5m])) / sum(rate(minio_s3_requests_total[5m]))) > 0.02
    for: 10m
    labels:
      severity: critical
    annotations:
      summary: Elevated 5xx error ratio
      description: Sustained server errors indicate user impact.

  - alert: MinIOHighLatencyP95
    expr: histogram_quantile(0.95, sum by (le) (rate(minio_s3_request_duration_seconds_bucket[5m]))) > 0.5
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: High p95 S3 latency
      description: Requests may be slow due to I/O or network contention.

  - alert: MinIOReplicationLag
    expr: sum(rate(minio_bucket_replication_pending_count[5m])) > 0
    for: 30m
    labels:
      severity: warning
    annotations:
      summary: Replication backlog is growing
      description: Check network path, credentials, target site health.

  - alert: MinIOTLSCertExpiringSoon
    expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14
    for: 0m
    labels:
      severity: warning
    annotations:
      summary: MinIO TLS certificate expires soon
      description: Renew and deploy certificates before expiry.

  - alert: HostSaturationCPU
    expr: (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) > 0.9
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: Host CPU saturation
      description: High CPU can degrade MinIO latency and throughput.

Tip: If minio_cluster_nodes_offline_total is not available, use the up metric and known target counts to detect down targets.

Alert tuning and validation

Avoid one-size-fits-all thresholds. Tune with real traffic:

• Baseline first: capture p50/p95 latencies, headroom, and normal error ratios for at least a week. Set warning at the edge of normal, critical beyond sustained abnormal.
• Choose stable windows: use 5–15 minute rates for request/error metrics and a meaningful for: duration to ignore brief blips.
• Add hysteresis: warn at a lower threshold and clear at a slightly better level (for example, warn at 80% capacity, clear at 75%).
• Validate in Prometheus: run each alert expression in the graph UI and verify it returns series when your synthetic tests trigger it.
• Group alerts: group by cluster or instance to reduce duplicate pages. Inhibit downstream symptoms (for example, suppress latency alerts if targets are down).
• Review monthly: compare alerts against incident notes and adjust thresholds and durations.

Log signals that catch issues early

MinIO emits structured JSON logs. Capture them centrally and watch for:

• level:error or level:panic
• err fields containing disk I/O errors or network timeouts
• Messages containing healing, disk not found, permission denied
• Spikes of AccessDenied, SignatureDoesNotMatch, InvalidAccessKeyId

Local examples:

# Follow errors in container logs
docker logs -f minio | jq -r 'select(.level=="error\)) | .time + " " + .msg'

# Count error types by .err
docker logs minio 2>&1 | jq -r 'select(.level=="error\)) | .err' | sort | uniq -c | sort -nr | head

# Live trace of S3 calls (slow/failing)
mc alias set local http://localhost:9000 MINIO_ROOT_USER MINIO_ROOT_PASSWORD
mc admin trace --verbose local

Consider adding a log-based alert for frequent disk write failures or repeated healing restarts within a short window.

A practical dashboard layout

Organize panels by user impact first, then root causes.

• User impact
• Requests per second: sum(rate(minio_s3_requests_total[5m]))
• 4xx and 5xx ratio over total requests
• p50, p95, p99 latency using histogram_quantile
• Capacity and growth
• Percent used: 100 * used_bytes / total_bytes
• 7-day growth: increase(used_bytes[7d])
• Top buckets by size (if available)
• Health
• Offline nodes and disks, healing operations in progress
• Scrape freshness and up by target
• Performance
• Read and write throughput, concurrency by operation

Compact incident runbook

Keep runbooks short and actionable. Distinguish symptoms (what you see) from causes (why it happens).

• Capacity high
• Symptoms: uploads fail, slow PUTs, frequent healing
• Checks: percent used, 7-day growth, top buckets/prefixes
• Likely causes: backup/analytics spikes, lifecycle misconfig
• Actions: enforce lifecycle, clean failed multipart, add/expand storage
• Commands: mc admin info local, mc ls --recursive --summarize local/bucket, mc ilm rule ls local/bucket

• Nodes or targets down
• Symptoms: 5xx spikes, timeouts
• Checks: up{job="minio"}, platform/pod status, node reachability
• Likely causes: host reboot, disk detach, network partition
• Actions: restore connectivity, restart pods, verify and mount disks
• Commands: mc admin info local, kubectl get pods -l app=minio -A, systemctl status minio || docker ps

• High 5xx error ratio
• Symptoms: server errors visible to clients
• Checks: error ratio and latency trends, logs for timeouts/internal errors
• Likely causes: saturated disks/CPU, network issues, partial outage
• Actions: reduce load or scale out, fix network path, retry with backoff
• Commands: mc admin trace --errors local

• High p95 latency
• Symptoms: slow operations, client-side retries
• Checks: latency histogram by op and node, host I/O/CPU
• Likely causes: noisy neighbors, DNS/TLS overhead, slow disks
• Actions: isolate I/O, cache hot prefixes, tune concurrency and timeouts

• Replication lag
• Symptoms: delayed objects at target site
• Checks: backlog size/age, replication worker status, network path
• Likely causes: target site down, credentials/policy issue, bandwidth limits
• Actions: restore target, fix access, temporarily unthrottle or reshard workers

• Certificate near expiry or invalid
• Symptoms: TLS handshake failures
• Checks: days to expiry, full chain, SANs match hostnames
• Actions: renew and deploy certs, reload MinIO, verify with openssl s_client

• Host saturation
• Symptoms: general slowdown, timeouts across ops
• Checks: CPU busy, disk busy time, packet drops
• Actions: add CPU/IOPS, rebalance traffic, fix noisy workloads on shared hosts

Local pilot plan

Make the first step narrow, measurable, and easy to inspect locally.

1. Bring up MinIO and observability stack

# docker-compose.yml
version: "3.8"
services:
 minio:
   image: minio/minio:latest
   command: server --console-address \"\:9001\" /data
   ports:
   - "9000:9000"
   - "9001:9001"
   environment:
   - MINIO_ROOT_USER=minioadmin
   - MINIO_ROOT_PASSWORD=minioadmin123
   volumes:
   - ./data:/data

 prometheus:
   image: prom/prometheus:latest
   command: [\"--config.file=/etc/prometheus/prometheus.yml\\)
   volumes:
   - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
   ports:
   - "9090:9090"

 grafana:
   image: grafana/grafana-oss:latest
   ports:
   - "3000:3000"

# prometheus.yml
global:
  scrape_interval: 15s
scrape_configs:
 - job_name: minio
   metrics_path: /minio/v2/metrics/cluster
   static_configs:
   - targets: ["host.docker.internal:9000\) # use "minio:9000" if same network

1. Seed data and verify metrics

mc alias set local http://localhost:9000 minioadmin minioadmin123
mc mb local/demo
mc cp /etc/hosts local/demo/hosts-copy
curl -s http://localhost:9000/minio/v2/metrics/cluster | head

1. Install the starter alerts

• Load the alert rules above. For local testing, use lower thresholds (for example, capacity at 5%, latency p95 at 200 ms, error ratio at 1%).

1. Induce safe test signals

# Capacity: upload a 50 MB file (repeat as needed)
base64 /dev/zero | head -c 52428800 > bigfile.bin
mc cp bigfile.bin local/demo/

# Target down: stop MinIO container for 2 minutes
docker stop minio && sleep 120 && docker start minio

# Error ratio: send bad auth to produce 4xx (temporarily alert on 4xx)
curl -s -o /dev/null -w "%{http_code}\\n" http://localhost:9000/demo/hosts-copy

1. Validate and record outcomes

• Ensure alerts fire with actionable descriptions and auto-clear after recovery.
• Note any metric name differences and update dashboards/alerts.

Exit criteria:

• Three core alerts tested end-to-end
• Dashboard shows capacity, latency, and error trends
• Runbooks drafted for each alert

Post-change verification checklist

Run this after upgrades, config changes, or scaling events:

• API health: p50/p95 latency and error ratio back to baseline
• Capacity: used%, free headroom, growth trend normal
• Targets healthy: up{job="minio"} == 1 for all; scrape latency stable
• Logs clean: no new error types or repeated healing loops
• Replication: backlog steady or decreasing
• Certificates: days-to-expiry above internal target, handshake OK
• Host: CPU and disk busy within expected range
• Alerts: no flapping; silences removed after tests

Conclusion

You now have a production-focused approach to MinIO monitoring: prioritize availability, capacity, latency, and error signals; back them with clear Prometheus queries, small and actionable alerts, and concise runbooks; and verify everything with a safe local pilot and a post-change checklist. Keep tuning thresholds to your real traffic and SLOs, expand dashboards as you scale, and continuously fold incident lessons back into your alerts and runbooks.