MinIO monitoring and alerts with practical examples: practical implementation guide

Intro

MinIO powers S3-compatible object storage across Docker and Kubernetes. To keep it reliable and fast, you need clear visibility into capacity, performance, and cluster health.

This guide shows what to monitor, the most useful metrics, alert rules, log signals, dashboard panels, and incident response workflows. It ends with a local pilot you can validate before a production rollout.

What you will get:

• A checklist of MinIO metrics that matter
• Practical Prometheus alert rules you can copy and adapt
• Dashboard panel formulas for fast triage
• Log patterns that often precede incidents
• Runbooks for common failure modes
• A safe, measurable local pilot plan

What to monitor in MinIO

Focus on signals that represent user impact, data safety, and operator action.

1. Availability and API health

• up{job="minio"} across all targets
• S3 request rate and error ratio
• Request duration percentiles (p50, p95, p99)

1. Capacity and growth

• Cluster total and used bytes
• Bucket and prefix level usage if you track it
• Free space headroom and burn rate

1. Cluster and disk health

• Offline nodes or drives
• Healing activity and backlog
• Disk IO errors reported by MinIO

1. Performance

• Throughput in bytes per second
• Concurrent requests
• Cache hit rate if applicable

1. Security and access

• Access denied spikes
• Root or admin credential usage

1. Data workflows

• Replication or batch job lags if you use them

Workflow Overview

A lightweight, reliable workflow keeps the signal-to-noise ratio high and reduces rework.

1. Expose metrics from MinIO

• MinIO exposes Prometheus metrics on the server endpoint. The cluster-level endpoint is typically /minio/v2/metrics/cluster.
• Confirm locally: curl http://MINIO_HOST:9000/minio/v2/metrics/cluster

1. Scrape with Prometheus

• Add a job that targets your MinIO endpoints.

# prometheus.yml
scrape_configs:
  - job_name: minio
    metrics_path: /minio/v2/metrics/cluster
    static_configs:
      - targets: ["minio:9000"]
        labels:
          cluster: demo

1. Visualize with Grafana

• Build a dashboard around capacity, latency, errors, and node health.

1. Alert on risks

• Start with a small set of action-oriented alerts: high capacity, nodes down, and high error ratio.

1. Add logs and traces

• Forward JSON logs to your log system, or read locally with jq.
• Use mc admin trace to sample live S3 calls during incidents.

1. Respond with runbooks

• For each alert, write a short runbook: checks, likely causes, and safe mitigations.

Practical PromQL you can reuse

The following names are representative of common MinIO metrics. Verify metric names in your environment with the Prometheus UI search box by typing "minio_" and browsing matches. Adjust label filters like job="minio" as needed.

• Cluster capacity percent used

100 * minio_cluster_capacity_used_bytes / minio_cluster_capacity_total_bytes

• S3 request rate, 5 minute window

sum by (method) (rate(minio_s3_requests_total[5m]))

• 5xx error ratio over all S3 requests

sum(rate(minio_s3_requests_errors_total{code=~"5.."}[5m]))
/ ignoring(code)
sum(rate(minio_s3_requests_total[5m]))

• P95 request latency (seconds)

histogram_quantile(
  0.95,
  sum by (le) (rate(minio_s3_request_duration_seconds_bucket[5m]))
)

• Offline nodes (if exposed)

minio_cluster_nodes_offline_total

• Disk offline count (if exposed)

minio_disks_offline_total

Alert rules to start with

Begin with a minimal, actionable set. Tune thresholds for your SLOs and capacity plans.

# minio.alerts.yml
groups:
- name: minio.alerts
  rules:
  - alert: MinIOClusterCapacityHigh
    expr: minio_cluster_capacity_used_bytes / minio_cluster_capacity_total_bytes > 0.85
    for: 15m
    labels:
      severity: warning
    annotations:
      summary: "MinIO capacity above 85%"
      description: "Capacity is {{ $value | printf "%.2f" }} of total. Plan cleanup or expansion."

  - alert: MinIONodesDown
    expr: minio_cluster_nodes_offline_total > 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "One or more MinIO nodes are offline"
      description: "Investigate node health and restore redundancy."

  - alert: MinIOHigh5xxErrorRate
    expr: (sum(rate(minio_s3_requests_errors_total{code=~"5.."}[5m])) / sum(rate(minio_s3_requests_total[5m]))) > 0.02
    for: 10m
    labels:
      severity: critical
    annotations:
      summary: "MinIO 5xx error ratio above 2%"
      description: "Sustained server errors indicate user impact."

  - alert: MinIOHighLatencyP95
    expr: histogram_quantile(0.95, sum by (le) (rate(minio_s3_request_duration_seconds_bucket[5m]))) > 0.5
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "MinIO p95 S3 latency above 500 ms"
      description: "Requests may be slow due to IO or network contention."

Tip: if a nodes-offline metric is not available, use up{job="minio"} and known target counts to detect down targets.

Log signals that catch issues early

MinIO emits structured JSON logs. Capture them centrally and watch for:

• level: error or level: panic
• err fields that include disk errors or network timeouts
• Messages containing "healing", "disk not found", "permission denied"
• Sudden spikes of AccessDenied, SignatureDoesNotMatch, or InvalidAccessKeyId

Local examples:

• Filter errors from Docker logs

docker logs -f minio | jq -r 'select(.level=="error") | .time + " " + .msg'

• Count error types with jq

docker logs minio 2>&1 | jq -r 'select(.level=="error") | .err' | sort | uniq -c | sort -nr | head

• Live trace of S3 calls showing slow or failing operations

mc alias set local http://localhost:9000 MINIO_ROOT_USER MINIO_ROOT_PASSWORD
mc admin trace --verbose local

Consider adding a log-based alert for frequent disk write failures or repeated healing restarts within a short window.

A practical dashboard layout

Organize panels by user impact first, then by cause.

1. User impact

• Requests per second: sum(rate(minio_s3_requests_total[5m]))
• 4xx and 5xx ratio: errors over total requests
• p50, p95, p99 latency using histogram_quantile

1. Capacity and growth

• Percent used: 100 * used_bytes / total_bytes
• 7 day growth: derivative or increase of used_bytes
• Top buckets by size (if you ingest bucket metrics)

1. Health

• Offline nodes and disks
• Healing operations in progress
• Node status table with up and scrape freshness

1. Performance

• Read and write throughput
• Concurrent requests by operation

Use repeating panels by node if you run a distributed cluster.

Incident response workflows

Write short, action-oriented runbooks for the top risks.

1. Capacity high

• Checks:
• Dashboard: percent used and recent growth
• Buckets with sudden growth
• Likely causes: backups or analytics jobs, retention misconfig
• Actions:
• Enforce or adjust lifecycle rules
• Clean temporary objects or failed multipart uploads
• Add capacity or expand the cluster
• Commands:

mc admin info local
mc ls --recursive --summarize local/bucket
mc ilm rule ls local/bucket

1. Nodes offline

• Checks:
• Prometheus up and nodes_offline metric
• Container or pod status, node status
• Likely causes: host reboot, disk detach, network partition
• Actions:
• Restore connectivity and restart the node
• Verify disks are detected and mounted
• Commands:

mc admin info local
kubectl get pods -l app=minio -A
systemctl status minio || docker ps

1. High 5xx error ratio

• Checks:
• Error ratio and latency trends
• Logs for timeouts or internal errors
• Likely causes: overwhelmed disks, network path issues, resource limits
• Actions:
• Reduce load or add capacity
• Investigate slow disks and network
• Commands:

mc admin trace --errors local

1. High latency p95

• Checks: latency histogram and throughput per node
• Likely causes: noisy neighbor workloads, DNS or TLS overhead, slow disks
• Actions: isolate IO, cache hot prefixes, tune concurrency

1. Healing storms

• Checks: healing counters and logs mentioning healing
• Likely causes: repeated restarts, disks flapping
• Actions: stabilize hardware, allow healing to complete before more restarts
• Commands:

mc admin heal -r --dry-run local

Local Pilot Plan

Make the first step narrow, measurable, and easy to inspect locally.

1. Bring up MinIO and observability stack

• docker-compose.yml

version: "3.8"
services:
  minio:
    image: minio/minio: latest
    command: server --console-address ":9001" /data
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      - MINIO_ROOT_USER=minioadmin
      - MINIO_ROOT_PASSWORD=minioadmin123
    volumes:
      - ./data:/data

  prometheus:
    image: prom/prometheus: latest
    command: ["--config.file=/etc/prometheus/prometheus.yml"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml: ro
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana-oss: latest
    ports:
      - "3000:3000"

• prometheus.yml

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: minio
    metrics_path: /minio/v2/metrics/cluster
    static_configs:
      - targets: ["host.docker.internal:9000"]  # or "minio:9000" if same network

1. Seed data and verify metrics

mc alias set local http://localhost:9000 minioadmin minioadmin123
mc mb local/demo
mc cp /etc/hosts local/demo/hosts-copy
curl -s http://localhost:9000/minio/v2/metrics/cluster | head

1. Install the starter alerts

• Use the alert rules above. For local testing, lower thresholds:
• Capacity high at 5%
• 5xx ratio at 1%
• p95 latency at 200 ms

1. Induce safe test signals

• Capacity: upload a few large files until you cross 5%

base64 /dev/zero | head -c 52428800 > bigfile.bin
mc cp bigfile.bin local/demo/

• Node down: stop the MinIO container for 2 minutes

docker stop minio && sleep 120 && docker start minio

• Error ratio: send bad auth to create 4xx spikes (adjust the alert to track 4xx for this test)

curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9000/demo/hosts-copy

1. Validate and record outcomes

• Confirm alerts fire, carry actionable descriptions, and clear when recovered
• Capture panel screenshots and note any metric name differences you had to adjust

1. Exit criteria

• All three alerts tested end-to-end
• Dashboard shows capacity, latency, error trends
• Runbooks drafted for each alert

Conclusion

You now have a lean, proven approach for MinIO monitoring: focus on capacity, errors, latency, and node health; back it with concise alerts, logs, and runbooks; and validate everything with a local pilot before production. Next steps:

• Expand from single node to distributed MinIO and add node-level panels
• Track bucket growth and set lifecycle policies to manage capacity
• Integrate alerts with your on-call tooling and annotate runbooks with recent incident notes
• Continuously tune thresholds based on real traffic and SLOs