Redis is more than a key-value cache. In production, it serves as a primary data store, message broker, rate limiter, session backend, and distributed lock coordinator. Understanding its advanced internals — memory management, replication internals, persistence trade-offs, Lua scripting atomicity, and module extensibility — separates teams that operate Redis safely from those that encounter silent data loss, latency spikes, or cascade failures during incidents. This article walks through each concept with version-specific commands, observable signals, and verified recovery steps for Redis 6.2 through 7.4.
Memory Architecture and Eviction Internals
Redis stores all data in a single-threaded, in-memory dictionary. The allocator (jemalloc by default, optionally jemalloc, glibc, or tcmalloc) manages arena fragmentation. Every key carries overhead: a redisObject header (16 bytes on 64-bit), SDS string structure, dict entry, and expiration metadata if a TTL is set. A 100-byte string value typically consumes 72–104 bytes of actual RSS depending on encoding.
Observe memory breakdown:
redis-cli -h <host> -p <port> INFO memory
# Focus on: used_memory, used_memory_rss, mem_fragmentation_ratio, used_memory_lua
redis-cli -h <host> -p <port> MEMORY STATS
redis-cli -h <host> -p <port> MEMORY DOCTORA mem_fragmentation_ratio above 1.5 signals allocator fragmentation; above 2.0 often correlates with latency spikes during fork() for RDB/AOF rewrite. Mitigation: restart the instance during a maintenance window, or switch to activedefrag yes (Redis 6.0+) with active-defrag-threshold-lower 10 and active-defrag-threshold-upper 100.
Eviction policy selection is a capacity contract, not a default. Configure explicitly in redis.conf or at runtime:
redis-cli CONFIG SET maxmemory-policy allkeys-lru
# Alternatives: volatile-lru, allkeys-lfu, volatile-lfu, allkeys-random, noevictionVerify with CONFIG GET maxmemory-policy. For workloads with power-law access (e.g., session stores), allkeys-lfu (Redis 4.0+) outperforms LRU by tracking access frequency with a logarithmic counter. For write-heavy caches with uniform access, allkeys-lru remains predictable.
Key encoding transitions affect memory silently. A SET with integer values uses intset encoding for small sets; exceeding 512 entries or a single element > 64 bytes promotes to hashtable. A HASH with ≤ 512 fields and values ≤ 64 bytes uses ziplist (Redis < 7) or listpack (Redis 7+); exceeding either threshold promotes to hashtable, multiplying per-field overhead. Monitor with:
redis-cli DEBUG OBJECT <key>
# Output includes: encoding, serializedlength, refcountPlan capacity using serialized length, not logical key count.
Replication Internals and Failover Mechanics
Redis replication is asynchronous by default. The primary streams write commands to replicas via a replication backlog (default 1 MB, configurable via repl-backlog-size). A replica that disconnects and reconnects within the backlog window performs a partial resync (PSYNC); otherwise, it triggers a full resync: the primary forks, generates an RDB snapshot, transfers it, then streams the backlog.
Critical observability commands:
redis-cli -h <replica> INFO replication
# Fields: master_link_status, master_last_io_seconds_ago, master_sync_in_progress, replica_repl_offset
redis-cli -h <primary> INFO replication
# Fields: connected_slaves, repl_backlog_active, repl_backlog_size, repl_backlog_first_offset, repl_backlog_histlenA master_link_status:down with master_last_io_seconds_ago > repl-timeout (default 60s) means the replica has been disconnected longer than the timeout — it will request full resync on reconnect. If repl_backlog_histlen approaches repl-backlog-size, increase the backlog (e.g., to 64–256 MB for high-write workloads) to absorb network partitions.
WAIT command for durability guarantees:
redis-cli -h <primary> SET key value
redis-cli -h <primary> WAIT 2 1000
# Returns number of replicas that acknowledged the write within 1000msWAIT blocks the client until the specified replica count acknowledges the write offset. It does not guarantee persistence to disk (see AOF/fsync below), only replication acknowledgment. Use for critical paths like payment idempotency keys.
Sentinel and Redis Cluster failover differ fundamentally. Sentinel (standalone) monitors primaries, promotes a replica after quorum (sentinel monitor mymaster <ip> <port> 2), and rewrites client configuration via sentinel client-reconfig-script. Redis Cluster (Redis 3.0+) shards keys across 16384 hash slots; failover is internal, slot-based, and requires majority of masters. For Cluster, observe:
redis-cli -h <node> -p <port> CLUSTER INFO
redis-cli -h <node> -p <port> CLUSTER NODES
# Look for: cluster_state:ok, slots_assigned:16384, no fail? or handshake statesA CLUSTER FAILOVER on a replica triggers manual failover; CLUSTER FAILOVER FORCE bypasses primary reachability checks — use only during confirmed primary loss.
Persistence: RDB, AOF, and Hybrid Strategies
RDB snapshots are point-in-time forks. save 900 1, save 300 10, save 60 10000 in redis.conf trigger background saves after N changes within M seconds. The fork latency scales with dataset size: a 50 GB dataset on a 16 vCPU VM typically forks in 200–800 ms; on overcommitted hosts, 2–5 seconds. During fork, the primary blocks all writes (copy-on-write page faults). Monitor with:
redis-cli INFO persistence
# Fields: rdb_bgsave_in_progress, rdb_last_bgsave_status, rdb_last_bgsave_time_sec, rdb_changes_since_last_saveA rdb_last_bgsave_status:err with errno=12 (ENOMEM) means the fork failed due to overcommit policy. Fix: echo 1 > /proc/sys/vm/overcommit_memory or reduce dataset size.
AOF (Append-Only File) logs every write command. Three fsync policies:
always: fsync after every write — durable, ~10–50k ops/sec on NVMe.everysec(default): fsync once per second — balances throughput and durability; worst-case 1 second data loss.no: delegate to OS — highest throughput, data loss up to 30 seconds on crash.
redis-cli CONFIG SET appendfsync everysec
redis-cli CONFIG GET appendfsyncAOF rewrite (BGREWRITEAOF) compacts the log by replaying current dataset into a minimal command set. Trigger automatically via auto-aof-rewrite-percentage 100 and auto-aof-rewrite-min-size 64mb. Monitor rewrite progress:
redis-cli INFO persistence
# aof_rewrite_in_progress, aof_rewrite_scheduled, aof_last_rewrite_time_secHybrid persistence (Redis 4.0+) combines RDB preamble with AOF tail: aof-use-rdb-preamble yes. On restart, Redis loads the RDB prefix (fast) then replays the AOF suffix (durable). This is the recommended production default for datasets > 10 GB. Verify with:
redis-cli CONFIG GET aof-use-rdb-preamble
# Should return "yes"Recovery test: stop Redis, rename appendonly.aof to appendonly.aof.bak, restart — confirm dataset loads from RDB preamble and recent writes are present.
Lua Scripting Atomicity and EVALSHA Patterns
Redis executes Lua scripts atomically: no other command interleaves. This enables read-modify-write patterns (rate limiting, distributed locks, conditional updates) without external coordination. Scripts are cached by SHA1; EVALSHA avoids retransmitting script text.
Rate limiter example (sliding window, Redis 6.2+):
-- KEYS[1] = rate limit key, ARGV[1] = window ms, ARGV[2] = max requests, ARGV[3] = now ms
local key = KEYS[1]
local window = tonumber(ARGV[1])
local limit = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local start = now - window
redis.call('ZREMRANGEBYSCORE', key, '-inf', start)
local count = redis.call('ZCARD', key)
if count >= limit then
return {0, count}
end
redis.call('ZADD', key, now, now .. '-' .. math.random())
redis.call('PEXPIRE', key, window)
return {1, count + 1}Load once:
SCRIPT LOAD "$(cat ratelimit.lua)"
# Returns SHA1, e.g., "a1b2c3d4..."Execute:
redis-cli EVALSHA a1b2c3d4... 1 ratelimit:user:123 60000 100 $(date +%s%3N)
# Returns: 1) "1" 2) "42" (allowed, current count)Critical constraints: Scripts must be pure (no math.random without seed, no os.time, no external I/O). Maximum execution time: lua-time-limit (default 5000 ms). A slow script blocks the entire event loop. Profile with redis-cli --eval script.lua key arg and monitor INFO commandstats for evalsha latency percentiles.
Script cache eviction: SCRIPT FLUSH clears all cached scripts (dangerous in production). Prefer SCRIPT EXISTS <sha1> to verify, then SCRIPT LOAD on startup via init script. For Cluster, scripts must only access keys in the same hash slot — use hash tags: {user:123}:ratelimit and {user:123}:session route to the same slot.
Module Extensibility: RedisJSON, RediSearch, and Bloom Filters
Redis 4.0+ modules run in-process, extending data types and commands. Three production-grade modules:
RedisJSON (v2.x) provides native JSON document storage with path-based access:
redis-cli JSON.SET user:1001 $ '{"name":"Alice","orders":[{"id":55,"total":120},{"id":56,"total":85}]}'
redis-cli JSON.GET user:1001 $.orders[0].total
# Returns: "120"
redis-cli JSON.NUMINCRBY user:1001 $.orders[1].total 15
# Atomically increments nested valueIndexing via RediSearch enables secondary queries on JSON paths.
RediSearch (v2.x) adds full-text and numeric indexing:
redis-cli FT.CREATE idx:user ON JSON PREFIX 1 user: SCHEMA $.name AS name TEXT $.orders[*].total AS order_total NUMERIC
redis-cli FT.SEARCH idx:user "@order_total:[100 200]"
# Returns matching user: keys with order totals in rangeIndex updates are synchronous on write; monitor FT.INFO idx:user for num_docs, indexing_failures, inverted_sz_mb.
RedisBloom (v2.x) provides probabilistic membership with configurable false-positive rate:
redis-cli BF.RESERVE seen:users 0.01 1000000
# error rate 1%, initial capacity 1M
redis-cli BF.ADD seen:users "user:1001"
redis-cli BF.EXISTS seen:users "user:1001"
# Returns 1 (maybe) or 0 (definitely not)Use for cache penetration protection, deduplication pipelines, or A/B test bucketing. Memory scales with capacity × log2(1/error_rate).
Module loading: Add to redis.conf:
loadmodule /usr/lib/redis/modules/rejson.so
loadmodule /usr/lib/redis/modules/search.so
loadmodule /usr/lib/redis/modules/redisbloom.soVerify with MODULE LIST. Modules must match Redis ABI version; upgrade modules when upgrading Redis.
Operational Verification Checklist
Before any configuration change or version upgrade, run this sequence:
- Baseline capture:
redis-cli INFO all > baseline-$(date +%F-%H%M).txt
redis-cli MEMORY STATS >> baseline-$(date +%F-%H%M).txt
redis-cli --latency-history -i 1 > latency-baseline-$(date +%F-%H%M).log &- Single-scope change: Apply one configuration (e.g.,
CONFIG SET maxmemory-policy allkeys-lfu), then verify:
redis-cli CONFIG GET maxmemory-policy
# Confirm: "maxmemory-policy" "allkeys-lfu"- Observe for 2–3 GC cycles or 10 minutes: Check
INFO memoryformem_fragmentation_ratio,used_memory_peak, andevicted_keysrate.
- Rollback trigger: If
evicted_keysspikes unexpectedly, latency p99 > 10 ms, orrejected_connections> 0, revert immediately:
redis-cli CONFIG SET maxmemory-policy allkeys-lru- Document outcome: Record timestamp, command, observed metric delta, and decision in runbook.
Conclusion
Operating Redis at scale demands treating each advanced feature as a contract with measurable invariants: memory fragmentation ratio under 1.5, replication backlog sized for maximum expected partition duration, AOF fsync policy aligned with RPO, Lua scripts bounded by 5 ms p99, and module versions locked to Redis ABI. The commands and verification steps above are not theoretical — they are the exact sequence used to diagnose a 40 GB dataset fork stall, recover a replica stuck in full resync, and validate a rate limiter surviving a thundering herd. Start with one verification: capture INFO memory, run MEMORY DOCTOR, and compare mem_fragmentation_ratio against your baseline. That single observation often reveals the next capacity decision before it becomes an incident.