Redis takes up all the Memory and Crashes

Question

A redis server v2.8.4 is running on a Ubuntu 14.04 VPS with 8 GB RAM and 16 GB swap space (on SSDs). However htop shows that redis alone is taking up 22.4 G of memory!

redis-server eventually crashed due to out of memeory. Mem and Swp both hits 100% then redis-server is killed along with other services.

From dmesg:

[165578.047682] Out of memory: Kill process 10155 (redis-server) score 834 or sacrifice child
[165578.047896] Killed process 10155 (redis-server) total-vm:31038376kB, anon-rss:5636092kB, file-rss:0kB

Restarting redis-server from etiher a OOM crash or a service redis-server force-reload causes memory usage to drop to <100MB.

Question: Why does redis-server occupy more and more memory till it crashes? How can we prevent this?

Is it true that setting maxmemory will not work because once redis hits the maxmemory limit, it will start removing data?

After restarting redis-server

Redis version: Redis server v=2.8.4 sha=00000000:0 malloc=jemalloc-3.4.1 bits=64 build=a44a05d76f06a5d9

---

Update

When htop reports the memory usage of redis-server to be 4.4G RAM and 22.6G Swap, the amount of space taken up by all the keys in redis is only 60.59636307 MB, as reported by rdbtools. This is also the amount of RAM taken up by redis-server right after it restarts.

INFO ALL when redis-server is taking up tons of memory

mem_fragmentation_ratio:0.19

127.0.0.1:6379> INFO all

# Server
redis_version:2.8.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:a44a05d76f06a5d9
redis_mode:standalone
os:Linux 3.13.0-24-generic x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.2
process_id:26858
run_id:4d4a507b325e567d5ada203a0c65891bcf4d02de
tcp_port:6379
uptime_in_seconds:100011
uptime_in_days:1
hz:10
lru_clock:165668
config_file:/etc/redis/redis.conf

# Clients
connected_clients:60
client_longest_output_list:768774
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:23973468008
used_memory_human:22.33G
used_memory_rss:4563857408
used_memory_peak:24083474760
used_memory_peak_human:22.43G
used_memory_lua:33792
mem_fragmentation_ratio:0.19
mem_allocator:jemalloc-3.4.1

# Persistence
loading:0
rdb_changes_since_last_save:127835154
rdb_bgsave_in_progress:0
rdb_last_save_time:1406716479
rdb_last_bgsave_status:err
rdb_last_bgsave_time_sec:1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok

# Stats
total_connections_received:110
total_commands_processed:386765263
instantaneous_ops_per_sec:3002
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:1385878
keyspace_misses:23655
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:82

# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:10547.48
used_cpu_user:8240.36
used_cpu_sys_children:201.83
used_cpu_user_children:914.86

# Commandstats
cmdstat_del:calls=136,usec=1407,usec_per_call=10.35
cmdstat_exists:calls=161428,usec=1391252,usec_per_call=8.62
cmdstat_zadd:calls=64149642,usec=936323882,usec_per_call=14.60
cmdstat_zrem:calls=137,usec=2131,usec_per_call=15.55
cmdstat_zremrangebyscore:calls=2293,usec=111905082,usec_per_call=48802.91
cmdstat_zrange:calls=7925,usec=285907448,usec_per_call=36076.65
cmdstat_zrangebyscore:calls=921434,usec=292731002,usec_per_call=317.69
cmdstat_zcount:calls=8,usec=172,usec_per_call=21.50
cmdstat_zrevrange:calls=191184,usec=965447,usec_per_call=5.05
cmdstat_zcard:calls=5180,usec=13502,usec_per_call=2.61
cmdstat_zscore:calls=29856,usec=576044,usec_per_call=19.29
cmdstat_hset:calls=64145124,usec=199407095,usec_per_call=3.11
cmdstat_hget:calls=248487,usec=501220,usec_per_call=2.02
cmdstat_hincrby:calls=128339355,usec=2071112929,usec_per_call=16.14
cmdstat_hgetall:calls=193747,usec=1608260,usec_per_call=8.30
cmdstat_select:calls=1,usec=5,usec_per_call=5.00
cmdstat_rename:calls=134,usec=1090,usec_per_call=8.13
cmdstat_keys:calls=4503,usec=4997628,usec_per_call=1109.84
cmdstat_bgsave:calls=2,usec=20012,usec_per_call=10006.00
cmdstat_type:calls=603,usec=2736,usec_per_call=4.54
cmdstat_multi:calls=64181979,usec=383633610,usec_per_call=5.98
cmdstat_exec:calls=64181979,usec=4403181204,usec_per_call=68.60
cmdstat_info:calls=126,usec=28675,usec_per_call=227.58

# Keyspace
db0:keys=2109,expires=0,avg_ttl=0

Answer 1

1. Use the maxmemory to set a limit to how much your Redis database can grow too. Failing to do so, Redis will grow until the OS will kill it once memory is exhausted (per your current experience).
2. The usage of maxmemory should be coupled with maxmemory-policy - you can choose from different eviction policies depending on your use case's requirements. For example, if you use the allkeys-lru eviction policy, Redis will indeed start evicting (the least recently used) data once maxmemory has been reached. Alternatively, you can instruct Redis to evict only expire-able data with the volatile-lru or volatile-random policies. Lastly, you can set the policy to noeviction but that would mean that once memory has been exhausted, Redis will deny further writes with an OOM message.

Edit:

First disable swap - Redis and swap don't mix easily and this can certainly cause slowness.

Also do free -m instead of top for the full picture of your RAM's state (http://www.linuxatemyram.com/).

Answer 2

This is almost certainly memory fragmentation, as redis is well-known and loved in production and you probably haven't found a memory leak.

The recommendations about setting the size of the pool won't help fragmentation. You'll have to specifically lower the Redis size - lower than your actual memory size - because Redis can't account for fragmentation - but, in terms of a short answer, you'll have to do that, and start plan on restarting your servers frequently.

My rule of thumb working with a variety of operating systems and in-memory databases is you need 2x your actual memory, and the memory size will stabilize in about 2 weeks.

However, that depends on your actual allocation patterns, and the memory allocator you're using.

Right now, the best memory allocator I've found for servers is JEMalloc. We use it at Aerospike now to reduce (nearly remove) long-term memory fragmentation. JEMalloc has a feature that allows you to create a memory "arena" (pool), and on any allocation, choose which pool, thus giving you like-size allocations and to manage similar memory lifetime allocations. It's been a big win for us in the kind of cases you're discussing.

The Zend PHP engine is sophisticated in this regard, because all allocations inside the engine are either in per-transaction memory, or global memory. Per transaction memory is freed at one-fell-swoop at the end of the transaction, and thus can be very efficient.

If you're in Linux, the kernel memory allocator (Clib) has taken a number of twists and turns, and which version you're on will dramatically determine the amount of fragmentation, as will the actual application pattern. For example, some allocators are much better when you are slightly growing objects, some are much worse. Regrettably, even discussing with other Redis users means talking about which OS, and which OS version, you're using.

The fact that you can restart the server (from persistence) and get your memory back could mean a leak, but more likely points to fragmentation.

1. Disallow swap (it's better to OOM than to swap, for redis)
2. Decrease redis' memory size
3. Restart on a schedule