How the Resource Manager Works

How the Resource Manager Works

The GemFire resource manager prevents the cache from consuming too much memory by evicting old data. If the garbage collector is unable to keep up, the resource manager refuses additions to the cache until the collector has freed an adequate amount of memory.

You can use the resource manager in any Pivotal GemFire member, but you may not want to use it everywhere. For some members it might be better to occasionally restart after a hang or OME crash than to evict data and/or refuse distributed caching activities. Also, members that do not risk running past their memory limits would not benefit from the overhead required to run the resource manager. Cache servers are often configured to use the manager because they generally host more data and have more data activity than other members, requiring greater responsiveness in data cleanup and collection.

The resource manager has two threshold settings, each expressed as a percentage of the total tenured heap. Both are disabled by default.
  1. Eviction Threshold. Above this, the manager orders evictions for all regions with eviction-attributes set to lru-heap-percentage. This prompts dedicated background evictions, independent of any application threads and it also tells all application threads adding data to the regions to evict at least as much data as they add. The JVM garbage collector removes the evicted data, reducing heap use. The evictions continue until the manager determines that heap use is again below the eviction threshold.
  2. Critical Threshold. Above this, all activity that might add data to the cache is refused. This threshold is set above the eviction threshold and is intended to allow the eviction and GC work to catch up. This JVM, all other JVMs in the distributed system, and all clients to the system receive LowMemoryException for operations that would add to this critical member's heap consumption. Activities that fetch or reduce data are allowed. For a list of refused operations, see the Javadocs for the ResourceManager method setCriticalHeapPercentage.

When heap use passes the eviction threshold, in either direction, the manager logs an info-level message. When it passes the critical threshold, the manager logs an error-level message.

Note: Avoid passing the critical threshold. It might be better than a hang or an OME, but the GemFire member that becomes a critical member is a read-only member that refuses cache updates for all of its regions, including incoming distributed updates.

For more information, see com.gemstone.gemfire.cache.control.ResourceManager in the online API documentation.

How Background Eviction Is Performed

When the manager kicks off evictions:
  1. From all regions in the local cache that are configured for heap LRU eviction, the background eviction manager creates a randomized list containing one entry for each partitioned region bucket (primary or secondary) and one entry for each non-partitioned region. So each partitioned region bucket is treated the same as a single, non-partitioned region.
  2. The background eviction manager starts four evictor threads for each processor on the local machine. The manager passes each thread its share of the bucket/region list. The manager divides the bucket/region list as evenly as possible by count, and not by memory consumption.
  3. Each thread iterates round-robin over its bucket/region list, evicting one LRU entry per bucket/region until the resource manager sends a signal to stop evicting.

See also Memory Requirements for Cached Data.