Excessive Memory Usage Observations on Linux glibc Versions >= 2.10

Linux glibc versions >= 2.10 since RHEL 6 have produced noticeably larger core files than would be expected from memory usage statistics. A core file is normally close to the size of the process’s working memory space. The C runtime under GLIBC versions  >= 2.10 now allocates a new heap for each thread on its first memory allocation call, and each heap is 64 MB. This behavior can be reproduced with a small test program. Create a thread and have it sleep without calling malloc(), and process memory use increases only by the thread’s stack size. However, if your thread calls malloc(), process memory use increases by about 64 MB.

This behavior is by design and introduced for scalability purposes. Allocating a separate heap for each thread reduces contention between the threads up to a limited number of threads. The creation of new heaps is limited to 8 times your CPU core count (64-bit systems) or 2 times your CPU core count (32-bit systems).

Malloc per-thread arenas in glibc
You can modify this behavior via an environment variable visible to your process. Set the system environment variable MALLOC_ARENA_MAX to 1 before starting FairCom DB, then only one heap is created for your entire process, and observed memory use appears as may be expected. That is, process size plus any memory allocations. 

It is also possible to call mallopt() to change MALLOC_ARENA_MAX directly in an application. This needs to be done before your first malloc() call.

Example

#include <malloc.h>
...
mallopt(M_ARENA_MAX, 1);
...

FairCom engineers are studying whether to include a c-treeACE configuration option and allow modifying this value such that it is tunable for specific applications.