MEMORY

how much heap?

how much RAM?

how much page cache?

how to avoid OOM?

Full system dissection

RAM = 8GB

kernel = 200MB

/proc/meminfo [Slab + KernelStack + PageTables]

monitoring (logstash) = 800MB

heap = 4G

java -Xmx4G cq-quickstart

page cache = 2GB

/proc/meminfo [Active(file)]

For identifying which processes have pages committed in the physical memory, I've extensively used standard linux tools, such as `ps`, `pmap` and the `/proc/meminfo` virtual file. The exact commands are gathered in a script at https://github.com/volteanu/sling-memory-deep-dive/blob/master/scripts/cqmeminfo.sh which you can download and adapt for inspecting your own system. Running the above script on my test system, which has 8GB of RAM, yielded the following consumers: - Linux kernel and assisting processes (e.g. monitoring apps) occupy about 1GB of the physical memory - AEM java process committed 4GB, which corresponds exactly to the max heap size parameter (-Xmx) passed to the java command - The page cache, managed by the linux kernel and intensively used by AEM for optimising file access, especially for caching segment tar files - Linux kernel always keeps some free RAM at hand in case new processes are started and need to allocate some memory Finding the right balance between heap and page cache is necessary to operate the application at the highest performance while keeping the cost under control. In this case, for the given load and content size, allocating 4GB to the heap and leaving 2GB for the page cache proved to be the right choice to have the application running without running out of (heap) memory and with a sufficiently big cache to ensure small response times.

Heap dissection

Xmx = 4GB

segment cache (256MB)

deduplication caches (200MB)

template cache (64MB)

link checker (72MB)

sling discovery (42MB)

datastore cache (36MB)

lucene (23MB)

sling servlet resolver (3MB)

replication (3MB)

felix service registry (2MB)

As mentioned before, the memory footprint for the heap was precisely equal to the max heap parameter. This is expected, since the JVM is always committing the maximum allocated amount of memory and then manages it by itself. So, even if you're not always using all the heap, the java process will keep all of it reserved in case it's needed at some point. The first thing to notice after taking the heap dump of live objects is that the generated file size is smaller (sometimes much smaller) than the total heap size. This is explained by the dynamic nature of java memory management, which means a part of the heap is not accessible and can be garbage-collected anytime. After grouping the accessible objects into "components" (based on the package), I've been able to identify the following, in descending order by size: - segment cache takes exactly 256MB (by design/configuration) - deduplication cache - used by Online Revision Cleanup - takes 200MB (also by design) - link checker has some sort of cache that keeps 72MB of objects in this usecase - template cache, also used by the Oak segment module, allocates 64MB - Sling discovery holds a Json Factory reference that takes 42MB - Datastore manages a cache of about 36MB - Lucene requires about 32MB for index caching - Other parts worth mentioning: replication, service registry and sling servlet resolver are also present, but with smaller amounts (around 2-3MB) One important characteristic of these modules is how the memory usage changes over time. There are two categories in which we can split them: 1. Fixed size: usually size-bound caches, such as the segment cache, deduplication caches 2. Variable size: basically all the components that allocate memory proportionally to the number of concurrent requests handled by the system. While some components are in my example quite small in heap footprint, given the relatively low system load, they can quickly grow and put a lot of stress on the heap if the concurrent traffic increases. The takeaway of this introspection should be: Sling and Oak require a fixed, minimum amount of heap, on top of which a dynamic part must to be allocated depending on the load. Also, custom bundles will add some pressure, so don't forget to take them into consideration when sizing your instance.

Page cache dissection

During normal operations

https://asciinema.org/a/200219

To help understand how this cache is exercised, I've focused on the segmentstore and tried to visualise which parts of the repository are loaded into memory in a dynamic recording. For this, I've used https://github.com/hoytech/vmtouch to list the resident pages of each tar file at regular time intervals and create a time-lapse animation. vmtouch outputs, for each file, the number of resident pages out of the total number of pages in the file. Besides it also illustrates the exact area of the file that resides in memory, giving a nice overview of which areas in the segmentstore are being cached at a certain timestamp. For better understanding the output, I've also colour-coded the tars, depending on the (OnRC) generation: files with the same colour belong to the same generation and there are usually two generations in the segmentstore (old and current). Inside the same generation, the files can be further split between tars generated by the Online Revision Cleanup process (coloured background) and tars generated by the application (coloured text). During normal operations (that is during the day, when no maintenance task is running and the system runs without problems), the following is expected: - the old generation is not loaded into memory; these files are not used anymore since they are superseded by the newer generation and will be deleted at the next revision cleanup - from the current generation, there are files that are frequently accessed so these are cached, but there will always be a part that is not accessed; that can be linked to two causes: 1. there are newer revisions of the nodes, so some segments holding the previous revisions are not accessible anymore, or 2. there's dormant content in the repository that is not accessed by the application (e.g. versions, audit logs, disabled indices) Over time, in normal conditions, the set of cached tars should be quite stable, so the disk is not accessed frequently. Also, the set of cached files is usually called the "working set" and is used for properly sizing the RAM you should let available for the operating system to be allocated for the page cache.

When "trashing"

https://asciinema.org/a/200570

During Online Revision Cleanup

https://asciinema.org/a/200223

The Online Revision Cleanup (OnRC) is a special phase in which this cache is going through an intensive refresh. As noted above, before OnRC, the situation looks like this: - generation n-1 (old) is not memory mapped since it's not accessed anymore - generation n (current) is partially loaded, defining the current "working set" When OnRC starts: - generation n is fully traversed; this is visible in the animation as a rolling loading/unloading of the tar files from this generation - generation n+1 is created as a compact replica of generation n; you will notice the fast creation of new files - generation n-1 is safely deleted at the end After OnRC: - parts of generation n (which is now old) are still memory mapped because they are accessed by long-running tasks in the application that hold old references; as time passes, the tasks complete and generation n is progressively evicted until no page from these tars is present in memory - generation n+1 becomes the current generation and is loaded according to the usage; at the end we're reaching the same state as before OnRC, completing the daily cycle

It depends!

Heap

Page cache

on your system load (traffic)

on your content size (repository)
on your working set

How much?

slides and goodies available at

https://github.com/volteanu/sling-memory-deep-dive/