## Intro MinIO powers S3-compatible object storage across Docker and Kubernetes. To keep it reliable and fast in production, you need clear visibility into availability, capacity, performance, and data safety, plus a small set of well-tuned alerts and concise runbooks. This guide focuses on the signals that matter, shows concrete Prometheus queries and starter alert rules, and adds a signal-to-action table, alert tuning tips, a compact incident runbook, and a post-change verification checklist. It ends with a local pilot you can validate before rolling out to production. What you will get: - A prioritized monitoring scope and signal-to-action table - Prometheus queries and copy-ready alert rules with tuning guidance - Practical log and trace patterns for early warning - A compact incident runbook and a post-change checklist - A safe, measurable local pilot plan ## What to monitor in MinIO Focus on user impact first, then root causes and operator actions. - Availability and API health: - up{job="minio"} , S3 request rate and error ratio, request duration percentiles (p50, p95, p99) - Capacity and growth: - Cluster total/used bytes, free headroom, burn rate, hot buckets/prefixes (use mc admin bucket top to identify hot buckets by request count) - Cluster and disk health: - Offline nodes and drives, healing activity/backlog, disk I/O errors - Performance: - Read/write throughput, concurrency by operation, cache hit rate (if enabled) - Security and access: - AccessDenied spikes, admin operations, TLS handshake errors - Data workflows: - Replication lag and failures, batch/ETL throughput changes - Host saturation (via node exporter or platform metrics): - CPU, memory pressure, disk latency, network saturation ## Signal-to-action table Use this table to turn top-line signals into first checks and actions. Treat thresholds as starting points; baseline your own environment before enforcing.

Area	Primary symptoms (what users see)	Useful metrics/logs	First checks	Immediate actions
Availability	Intermittent 5xx, timeouts	`up{job="minio"}`, `minio_s3_requests_total`, `minio_s3_requests_errors_total`, logs: connection timeout	Are any targets down? Recent deploy? Network/pod restarts?	Restore down pods/nodes, roll back unsafe changes, validate DNS/service endpoints
Capacity	Uploads fail, slow writes, healing storms after restarts	`minio_cluster_capacity_used_bytes`, `minio_cluster_capacity_total_bytes`, growth of used bytes	Trend of usage vs. headroom, large bucket/prefix growth, lifecycle rules	Enforce lifecycle, clean failed multipart, add/expand disks or nodes
Latency	Slow GET/PUT, client retries	`minio_s3_request_duration_seconds_bucket` (p95/p99), throughput per node	Is latency localized to ops or nodes? Any host I/O spikes?	Reduce concurrency, isolate I/O, cache hot prefixes, scale out
Errors	Spikes in 4xx/5xx, auth failures	`minio_s3_requests_errors_total` by code, logs: AccessDenied, SignatureDoesNotMatch	Recent key/policy changes? Client-side changes?	Restore credentials/policies, rate-limit abusive clients, confirm clock sync
Replication	Objects lagging across sites	e.g., `minio_bucket_replication_pending_count` or `minio_bucket_replication_lag_seconds` (verify exact names in your version), logs: replication failures	Backlog size and age, network path, target site health	Unthrottle replication, fix credentials/permissions, pause noisy jobs
Certificates	TLS handshake failures, clients cannot connect	Logs: TLS handshake errors, `blackbox_exporter probe_ssl_earliest_cert_expiry`	Expiry date, full chain installed, clocks in sync	Renew/deploy certs, reload MinIO, verify full chain and SANs
Host saturation	All ops slower, occasional timeouts	`node_cpu_seconds_total`, `node_disk_io_time_seconds_total`, `node_load1`, `node_network*`	CPU steal/waits, disk busy time, packet drops	Move noisy neighbors, add IOPS/CPU, tune queues, rebalance traffic

## Workflow overview - Expose metrics from MinIO - MinIO exposes Prometheus metrics at /minio/v2/metrics/cluster . - Quick check: curl http://MINIO_HOST:9000/minio/v2/metrics/cluster - Scrape with Prometheus - Add a scrape job targeting your MinIO endpoints. # prometheus.yml scrape_configs: - job_name: minio metrics_path: /minio/v2/metrics/cluster static_configs: - targets: ["minio:9000\] labels: cluster: demo - Visualize with Grafana - Build top-row panels for availability, latency, error ratio, and capacity. Use repeating panels by node for distributed clusters. - Alert on risks - Start with capacity, nodes/targets down, high error ratio, and high latency. Expand to replication, cert expiry, and host saturation. - Add logs and traces - Forward JSON logs to your log platform. Use mc admin trace for live S3 call sampling during incidents. - Respond with runbooks - For each alert, document checks, likely causes, and safe mitigations. ## Prometheus queries you can reuse Metric names vary by MinIO version and build. In Prometheus, type minio_ to browse metrics and adjust filters (for example, job="minio" ). - Cluster capacity percent used 100 * minio_cluster_capacity_used_bytes / minio_cluster_capacity_total_bytes - S3 request rate by method (5m) sum by (method) (rate(minio_s3_requests_total[5m])) - 5xx error ratio (5m) sum(rate(minio_s3_requests_errors_total{code=~"5.."}[5m])) / ignoring(code) sum(rate(minio_s3_requests_total[5m])) - P95 request latency (seconds) histogram_quantile( 0.95, sum by (le) (rate(minio_s3_request_duration_seconds_bucket[5m])) ) - Offline nodes (if available) minio_cluster_nodes_offline_total - Offline disks (if available) minio_disks_offline_total - Replication backlog (example; verify names in your cluster) sum(rate(minio_bucket_replication_pending_count[5m])) *Note: Metric names vary by MinIO version; if minio_bucket_replication_pending_count is absent, try minio_bucket_replication_pending_total . - Certificate expiry (via blackbox exporter scraping MinIO endpoint) (probe_ssl_earliest_cert_expiry - time()) / 86400 - Host saturation (node exporter) # CPU busy ratio 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) # Disk busy time avg by (instance, device) (rate(node_disk_io_time_seconds_total[5m])) ## Starter alert rules Begin with a minimal, actionable set. Tune thresholds to your SLOs and baseline, not universal values. # minio.alerts.yml groups: - name: minio.alerts rules: - alert: MinIOClusterCapacityHigh expr: (minio_cluster_capacity_used_bytes / minio_cluster_capacity_total_bytes) > 0.85 for: 15m labels: severity: warning annotations: summary: MinIO capacity is high description: Capacity exceeds target headroom. Plan cleanup or expansion. - alert: MinIOTargetsDown expr: count(up{job="minio"} == 0) > 0 for: 5m labels: severity: critical annotations: summary: One or more MinIO scrape targets are down description: Investigate node/pod health and restore redundancy. - alert: MinIOHigh5xxErrorRate expr: (sum(rate(minio_s3_requests_errors_total{code=~"5.."}[5m])) / sum(rate(minio_s3_requests_total[5m]))) > 0.02 for: 10m labels: severity: critical annotations: summary: Elevated 5xx error ratio description: Sustained server errors indicate user impact. - alert: MinIOHighLatencyP95 expr: histogram_quantile(0.95, sum by (le) (rate(minio_s3_request_duration_seconds_bucket[5m]))) > 0.5 for: 10m labels: severity: warning annotations: summary: High p95 S3 latency description: Requests may be slow due to I/O or network contention. - alert: MinIOReplicationLag expr: sum(rate(minio_bucket_replication_pending_count[5m])) > 0 for: 30m labels: severity: warning annotations: summary: Replication backlog is growing description: Check network path, credentials, target site health. - alert: MinIOTLSCertExpiringSoon expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14 for: 0m labels: severity: warning annotations: summary: MinIO TLS certificate expires soon description: Renew and deploy certificates before expiry. - alert: HostSaturationCPU expr: (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) > 0.9 for: 10m labels: severity: warning annotations: summary: Host CPU saturation description: High CPU can degrade MinIO latency and throughput. Tip: If minio_cluster_nodes_offline_total is not available, use the up metric and known target counts to detect down targets. ## Alert tuning and validation Avoid one-size-fits-all thresholds. Tune with real traffic: - Baseline first: capture p50/p95 latencies, headroom, and normal error ratios for at least a week. Set warning at the edge of normal, critical beyond sustained abnormal. - Choose stable windows: use 5–15 minute rates for request/error metrics and a meaningful for: duration to ignore brief blips. - Add hysteresis: warn at a lower threshold and clear at a slightly better level (for example, warn at 80% capacity, clear at 75%). - Validate in Prometheus: run each alert expression in the graph UI and verify it returns series when your synthetic tests trigger it. - Group alerts: group by cluster or instance to reduce duplicate pages. Inhibit downstream symptoms (for example, suppress latency alerts if targets are down). - Review monthly: compare alerts against incident notes and adjust thresholds and durations. ## Log signals that catch issues early MinIO emits structured JSON logs. Capture them centrally and watch for: - level:error or level:panic - err fields containing disk I/O errors or network timeouts - Messages containing healing , disk not found , permission denied - Spikes of AccessDenied , SignatureDoesNotMatch , InvalidAccessKeyId Local examples: # Follow errors in container logs docker logs -f minio | jq -r 'select(.level=="error\)) | .time + " " + .msg' # Count error types by .err docker logs minio 2>&1 | jq -r 'select(.level=="error\)) | .err' | sort | uniq -c | sort -nr | head # Live trace of S3 calls (slow/failing) mc alias set local http://localhost:9000 MINIO_ROOT_USER MINIO_ROOT_PASSWORD mc admin trace --verbose local Consider adding a log-based alert for frequent disk write failures or repeated healing restarts within a short window. ## A practical dashboard layout Organize panels by user impact first, then root causes. - User impact - Requests per second: sum(rate(minio_s3_requests_total[5m])) - 4xx and 5xx ratio over total requests - p50, p95, p99 latency using histogram_quantile - Capacity and growth - Percent used: 100 * used_bytes / total_bytes - 7-day growth: increase(used_bytes[7d]) - Top buckets by size (if available) - Health - Offline nodes and disks, healing operations in progress - Scrape freshness and up by target - Performance - Read and write throughput, concurrency by operation ## Compact incident runbook Keep runbooks short and actionable. Distinguish symptoms (what you see) from causes (why it happens). - Capacity high - Symptoms: uploads fail, slow PUTs, frequent healing - Checks: percent used, 7-day growth, top buckets/prefixes - Likely causes: backup/analytics spikes, lifecycle misconfig - Actions: enforce lifecycle, clean failed multipart, add/expand storage - Commands: mc admin info local , mc ls --recursive --summarize local/bucket , mc ilm rule ls local/bucket - Nodes or targets down - Symptoms: 5xx spikes, timeouts - Checks: up{job="minio"} , platform/pod status, node reachability - Likely causes: host reboot, disk detach, network partition - Actions: restore connectivity, restart pods, verify and mount disks - Commands: mc admin info local , kubectl get pods -l app=minio -A , systemctl status minio || docker ps - High 5xx error ratio - Symptoms: server errors visible to clients - Checks: error ratio and latency trends, logs for timeouts/internal errors - Likely causes: saturated disks/CPU, network issues, partial outage - Actions: reduce load or scale out, fix network path, retry with backoff - Commands: mc admin trace --errors local - High p95 latency - Symptoms: slow operations, client-side retries - Checks: latency histogram by op and node, host I/O/CPU - Likely causes: noisy neighbors, DNS/TLS overhead, slow disks - Actions: isolate I/O, cache hot prefixes, tune concurrency and timeouts - Replication lag - Symptoms: delayed objects at target site - Checks: backlog size/age, replication worker status, network path - Likely causes: target site down, credentials/policy issue, bandwidth limits - Actions: restore target, fix access, temporarily unthrottle or reshard workers - Certificate near expiry or invalid - Symptoms: TLS handshake failures - Checks: days to expiry, full chain, SANs match hostnames - Actions: renew and deploy certs, reload MinIO, verify with openssl s_client - Host saturation - Symptoms: general slowdown, timeouts across ops - Checks: CPU busy, disk busy time, packet drops - Actions: add CPU/IOPS, rebalance traffic, fix noisy workloads on shared hosts ## Local pilot plan Make the first step narrow, measurable, and easy to inspect locally. - Bring up MinIO and observability stack # docker-compose.yml version: "3.8" services: minio: image: minio/minio:latest command: server --console-address \"\:9001\" /data ports: - "9000:9000" - "9001:9001" environment: - MINIO_ROOT_USER=minioadmin - MINIO_ROOT_PASSWORD=minioadmin123 volumes: - ./data:/data prometheus: image: prom/prometheus:latest command: [\"--config.file=/etc/prometheus/prometheus.yml\\) volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro ports: - "9090:9090" grafana: image: grafana/grafana-oss:latest ports: - "3000:3000" # prometheus.yml global: scrape_interval: 15s scrape_configs: - job_name: minio metrics_path: /minio/v2/metrics/cluster static_configs: - targets: ["host.docker.internal:9000\) # use "minio:9000" if same network - Seed data and verify metrics mc alias set local http://localhost:9000 minioadmin minioadmin123 mc mb local/demo mc cp /etc/hosts local/demo/hosts-copy curl -s http://localhost:9000/minio/v2/metrics/cluster | head - Install the starter alerts - Load the alert rules above. For local testing, use lower thresholds (for example, capacity at 5%, latency p95 at 200 ms, error ratio at 1%). - Induce safe test signals # Capacity: upload a 50 MB file (repeat as needed) base64 /dev/zero | head -c 52428800 > bigfile.bin mc cp bigfile.bin local/demo/ # Target down: stop MinIO container for 2 minutes docker stop minio && sleep 120 && docker start minio # Error ratio: send bad auth to produce 4xx (temporarily alert on 4xx) curl -s -o /dev/null -w "%{http_code}\\n" http://localhost:9000/demo/hosts-copy - Validate and record outcomes - Ensure alerts fire with actionable descriptions and auto-clear after recovery. - Note any metric name differences and update dashboards/alerts. Exit criteria: - Three core alerts tested end-to-end - Dashboard shows capacity, latency, and error trends - Runbooks drafted for each alert ## Post-change verification checklist Run this after upgrades, config changes, or scaling events: - API health: p50/p95 latency and error ratio back to baseline - Capacity: used%, free headroom, growth trend normal - Targets healthy: up{job="minio"} == 1 for all; scrape latency stable - Logs clean: no new error types or repeated healing loops - Replication: backlog steady or decreasing - Certificates: days-to-expiry above internal target, handshake OK - Host: CPU and disk busy within expected range - Alerts: no flapping; silences removed after tests ## Conclusion You now have a production-focused approach to MinIO monitoring: prioritize availability, capacity, latency, and error signals; back them with clear Prometheus queries, small and actionable alerts, and concise runbooks; and verify everything with a safe local pilot and a post-change checklist. Keep tuning thresholds to your real traffic and SLOs, expand dashboards as you scale, and continuously fold incident lessons back into your alerts and runbooks.