Committed memory in Linux is less than expected

Question

One of my linux server's memory utilization is weird. As far as I know larger Committed memory (Committed_AS in /proc/meminfo or kbcommit in sar -r) than application used memory is normal. But my server's commited memory is significantly small.

# cat /proc/meminfo
MemTotal:       32877480 kB
MemFree:         3511812 kB
Buffers:           19364 kB
Cached:         12080112 kB
SwapCached:            0 kB
Active:         22658640 kB
Inactive:        5906936 kB
Active(anon):   16460116 kB
Inactive(anon):     6652 kB
Active(file):    6198524 kB
Inactive(file):  5900284 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               544 kB
Writeback:             4 kB
AnonPages:      16482208 kB
Mapped:            17228 kB
Shmem:               672 kB
Slab:             529344 kB
SReclaimable:     460220 kB
SUnreclaim:        69124 kB
KernelStack:       12304 kB
PageTables:        51412 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    16438740 kB
Committed_AS:    4484680 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       66424 kB
VmallocChunk:   34359651996 kB
HardwareCorrupted:     0 kB
AnonHugePages:  15376384 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        4096 kB
DirectMap2M:     2093056 kB
DirectMap1G:    31457280 kB

As you can see Committed_AS is about 4.4GB. But one of process is using more than 14GB. This process doesn't have any mmaped file or any shared memory.

# pidstat -r 1 1
Linux 2.6.32-696.16.1.el6.x86_64 (appintprdsearch01)    08/23/21    _x86_64_    (8 CPU)
16:56:34          PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
16:56:35          414  19476.24      0.00 17260248 14356476  43.67  isc_mc
pmap -x 414 | egrep '^Address|^total'
Address           Kbytes     RSS   Dirty Mode   Mapping
total kB        17268588 14363848 14360208
pmap -x 414 | grep anon | awk '{s+=$3} END {print s}'
14326736

I wonder why this process's used memory does not affect committed memory. Is there something I lost?

# cat /etc/system-release
Red Hat Enterprise Linux Server release 6.8 (Santiago)
# uname -r
2.6.32-696.16.1.el6.x86_64

Thanks in advance!!

EDIT: I tried to show full pmap -x output taken today. but it's 758 lines long and exceed the post limit. So here's the summary. And I found out its anon pages are getting bigger.

# pmap -x 414 | grep -vw anon
414:   /usr/local/search/sf-1v530/bin/isc_mc -license /usr/local/search/sf-1v530/license/license.xml -conf ../config/config_mc.xml -pid /usr/local/search/sf-1v530/pid/isc_mc.pid -log /usr/local/search/sf-1v530/log/isc_mc
Address           Kbytes     RSS   Dirty Mode   Mapping
0000000000400000    4616    1720       0 r-x--  isc_mc
0000000000a81000    1304     716      20 rw---  isc_mc
0000003c54200000     128     108       0 r-x--  ld-2.12.so
0000003c54420000       4       4       4 r----  ld-2.12.so
0000003c54421000       4       4       4 rw---  ld-2.12.so
0000003c54600000    1576     584       0 r-x--  libc-2.12.so
0000003c5478a000    2048       0       0 -----  libc-2.12.so
0000003c5498a000      16      16       8 r----  libc-2.12.so
0000003c5498e000       8       8       8 rw---  libc-2.12.so
0000003c54a00000      92      72       0 r-x--  libpthread-2.12.so
0000003c54a17000    2048       0       0 -----  libpthread-2.12.so
0000003c54c17000       4       4       4 r----  libpthread-2.12.so
0000003c54c18000       4       4       4 rw---  libpthread-2.12.so
0000003c55600000     524      80       0 r-x--  libm-2.12.so
0000003c55683000    2044       0       0 -----  libm-2.12.so
0000003c55882000       4       4       4 r----  libm-2.12.so
0000003c55883000       4       4       4 rw---  libm-2.12.so
0000003c55a00000      84      16       0 r-x--  libz.so.1.2.3
0000003c55a15000    2044       0       0 -----  libz.so.1.2.3
0000003c55c14000       4       4       4 r----  libz.so.1.2.3
0000003c55c15000       4       4       4 rw---  libz.so.1.2.3
0000003c56600000      88      16       0 r-x--  libgcc_s-4.4.7-20120601.so.1
0000003c56616000    2044       0       0 -----  libgcc_s-4.4.7-20120601.so.1
0000003c56815000       4       4       4 rw---  libgcc_s-4.4.7-20120601.so.1
0000003c57200000     928     488       0 r-x--  libstdc++.so.6.0.13
0000003c572e8000    2048       0       0 -----  libstdc++.so.6.0.13
0000003c574e8000      28      28      28 r----  libstdc++.so.6.0.13
0000003c574ef000       8       8       8 rw---  libstdc++.so.6.0.13
00007ffdcbf56000      84      48      48 rw---    [ stack ]
----------------  ------  ------  ------
total kB        18462068 15806748 15802956

This is sorted size of anon map and its count.

# pmap -x 414 | grep -w anon | awk '{s[$2]++} END {for (x in s) {print x,s[x]}}' | sort
-rnk 1
254728 1
225520 1
197220 1
196608 1
131480 2
131072 2
124008 1
65740 3
65536 142
65532 5
65528 3
65524 2
65520 1
65512 2
65508 2
65504 3
65500 3
65488 2
65484 5
65480 2
65476 1
65464 2
65456 1
65452 1
65448 2
65440 4
65436 3
65432 3
65428 2
65424 1
65420 28
65412 1
65404 1
65400 1
65372 1
65192 1
65188 1
64804 1
64636 1
63940 1
63912 1
63792 1
63584 1
63408 1
63400 2
63244 1
63008 2
63004 1
61996 1
61848 1
61792 1
61624 1
60976 1
59940 1
58276 1
57736 1
55992 1
52348 1
51212 1
50084 1
40840 1
24696 1
15452 1
14324 1
13188 1
10396 1
10240 1
9544 1
7800 1
7260 1
7064 1
5596 1
4560 1
3912 1
3744 1
3688 1
3540 1
3216 1
2532 1
2528 2
2292 1
2136 2
2128 1
1952 1
1744 1
1624 1
1596 1
1024 169
900 1
732 1
348 1
344 1
164 1
136 1
132 1
124 1
116 28
112 1
108 2
104 3
100 3
96 4
88 2
84 2
80 1
72 2
60 1
56 2
52 5
48 2
36 3
32 3
28 2
24 2
16 3
12 2
8 4
4 179

EDIT: I've requested customer support to Redhat. I'm afraid I can not receive any anwser as RHEL6 was EOSed. Anyway I'll post here the result.

score 3 · Answer 1 · answered Aug 23 '21 at 15:31

Half of this host's memory is in anon private pages. Most of that is the process you identified. Committed_AS relatively low at 14% of MemTotal implies not much is in use yet for actual data.

Operating systems are lazy. Linux will not touch all 14 GB this program asked for. Instead, as this x86 CPU supports 2 MB and 1 GB pages, Linux assigns several of those to this process and pretends they are entirely zeroed. Notice DirectMap showing many hardware pages of 1 GB. HugePages can be configured explicitly by the administrator, but that meminfo shows zero of those configured.

Just this allocation takes very little physical memory, so Committed_AS starts low. If and when the application fills these with real data, it will increase.

Regarding capacity planning, right now this host currently has plenty of memory. Anon and shared memory is limited to half, 36% cached for fast file access, and a few GB free and unused to satisfy immediate memory allocation demands. And of course Committed_AS far under MemTotal

Even though the application's memory allocations fit the size of the host, find out why it was sized that way. If this is an in memory database sized for future growth, that might make sense. Or maybe 32 GB is your medium sized VM / small sized physical server, over sizing may be easier operationally.

Committed memory in Linux is less than expected

pmap -x 414 | egrep '^Address|^total'

pmap -x 414 | grep anon | awk '{s+=$3} END {print s}'

1 Answers1