8

While stress-testing LVM on one of our Debian servers, I came across this issue where memory would fill up a lot to the point where it would run the server out of memory, but no process would claim the memory. See https://i.sstatic.net/ismcH.png, and see https://serverfault.com/a/449102/125894 for an explanation on the colors used in htop.

Why is this happening?
And is there any way to see what process is using the memory?
Htop is configured not to hide any processes, so what is it that htop is missing?

In this particular case, I can fairly certainly say that it is caused, directly or indirectly, by lvmcreate, lvmremove or dmsetup, as I was stress-testing that. Do note that this question is not about solving the LVM problem, but about why the memory isn't claimed by any process. Stopping all LVM commands does bring the memory back down to <600MB.

Output of free -m:

             total       used       free     shared    buffers     cached
Mem:         32153      31958        194          0         52       3830
-/+ buffers/cache:      28075       4077
Swap:          975          0        975


Output of top, on pastebin due to the length of it: http://pastebin.com/WchrpF7W

Community
  • 1
SharkWipf
  • 361

3 Answers3

3

Since you seem to be running I/O related tests (you didn't explain with details what you did with the LVM volumes), is there any chance that the behavior you noticed is related to the natural OS page cache?

I'm thinking about the Linux standard behavior of using (sometimes almost all) available memory (that is, memory that is not reserved by applications) to cache disk regions that were recently accessed, in order to speed up IOs.

Of course, active applications have priority over this cache, so in case a process needs more ram, the cache is dynamically shrinked.

There is a lot of documentation about Linux page cache on the web, here is one.

mbarthelemy
  • 378
2

I have the same problem, but I checked the kernel parameters located under /proc/sys/vm/ : overcommit_memory = Default 0 overcommit_ratio = Default 50 max_map_count = Default 65530

And in my case the parameter, root of my problems nr_hugepages, i haved the custom value, but change it for default

nr_hugepages = 0

For more information see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-memory-captun

Sorry for my english :(

Jeant
  • 21
1

For me it seemed to be the page cache not clearing itself.

sync; echo 1 | sudo tee /proc/sys/vm/drop_caches

Gave me 30gB back after a few seconds.

Hashbrown
  • 366