Tanzu GemFire 9.10 Release Notes

Pivotal GemFire is now VMware Tanzu GemFire.

Tanzu GemFire 9.10 is based on Apache Geode version 1.12, so it also includes features and bug fixes from Apache Geode version 1.11. Tanzu GemFire 9.10 and Apache Geode share a common set of documentation. Therefore, references to both Tanzu GemFire and Apache Geode appear throughout this documentation; consider them equivalents.

NOTE: Version 9.10.5 of Tanzu GemFire incorporated fixes for issues related to recovery behavior and peer-to-peer TLS communication. We recommend that all customers running versions prior to 9.10.5 update to the latest maintenance release.

What’s New in Tanzu GemFire 9.10.9

Tanzu GemFire 9.10.9 includes the following new features and improvements:

• Tanzu GemFire 9.10.9 provides a new option enabling automatic retries in response to PdxSerializationException. To apply this mode, the client application must set the system property gemfire.enableQueryRetryOnPdxSerializationException=true.
  
  For example:

  client.invoke(() -> {
      System.setProperty(
        GeodeGlossary.GEMFIRE_PREFIX + "enableQueryRetryOnPdxSerializationException", "true");
      QueryService remoteQueryService = (PoolManager.find(poolName)).getQueryService();
      Query query = remoteQueryService.newQuery("SELECT DISTINCT id FROM pdxTest");
      SelectResults<TestObjectThrowsPdxSerializationException> selectResults = (SelectResults)query.execute();
  });

What’s New in Tanzu GemFire 9.10.8

Tanzu GemFire 9.10.8 includes the following new features and improvements:

• Tanzu GemFire version 9.10.8 introduces a Tanzu GemFire metrics module that provides out-of-the-box integration with Tanzu Observability by Wavefront. Metrics can be forwarded to Wavefront for analysis and alerting. For more information, see Tanzu Observability by Wavefront.

• Documentation was added for Tanzu GemFire Micrometer, an interface for emitting user-focused metrics to a variety of different consuming Application Performance Monitoring (APM) tools. This feature was released in Apache Geode v1.11, and so was included in the Tanzu Gemfire 9.10 release, but it was not previously documented in the Tanzu GemFire User Guide.

• Addressed an issue with deserialization of configuration information from old locators during rolling upgrades (GEODE-9289, GEM-3253).

• A new default setting for a JMX property affecting users of the Java 11 JDK (GEODE-9064, GEM-3185).

See Issues Resolved in Tanzu GemFire 9.10.8 for details regarding these and other issues addressed in this release.

What’s New in Tanzu GemFire 9.10.7

Tanzu GemFire versions 9.10.1 through 9.10.7 are maintenance releases that resolve a variety of issues—see Resolved Issues for more information.

What’s New in Tanzu GemFire 9.10

Tanzu GemFire 9.10 includes the following improvements:

• JMX now exposes the free disk space percentage as a statistic.

• Asynchronous event queues may now be created in a paused state. This allows better control of event dispatch, as the dispatch may be delayed until the application is ready for event consumption.

• The new pluggable OQL security framework restrict which classes and methods can be called from queries. A system administrator can change the security posture at runtime. Provided implementations cover several use cases, as described in Method Invocation Authorizers.

• There are new endpoints in the REST API for Management, including create and delete index, list deployed jars, show PDX configuration, and show the supported REST API versions.

Installing Tanzu GemFire 9.10

You can download Tanzu GemFire 9.10 from the Tanzu GemFire product download page.

For details on how to install Tanzu GemFire 9.10, see Installing Tanzu GemFire.

Upgrading to Tanzu GemFire 9.10

To upgrade from an earlier version of Tanzu GemFire to the current version, see Upgrading Tanzu GemFire. Version-specific upgrade details follow.

Upgrading to Version 9.10.6

The fix for issues GEODE-8799 and GEM-3041: addressed a performance issue by increasing the defaults for maximum number of pooled message processor threads and maximum partitioned region message processor threads. NOTE: Users should be aware that if their system had been hitting the old, lower default maximums, then upgrading may cause increased use of system resources as the system is no longer constrained to the old values. If your system depends on constraining system resources to the old default values, you can set these properties explicitly using DistributionManager.MAX_THREADS and DistributionManager.MAX_PR_THREADS, respectively.

The following table shows the old and new maximum values.

Upgrading from a Version prior to 9.1.1

When servers are upgraded to this Tanzu GemFire version from a version prior to version 9.1.1, their authentication of client messages is disabled in order to facilitate rolling upgrades. If your clients are set up to send authenticated messages, the servers will honor those messages, but will not enforce authentication until you reinstate authentication on each of the upgraded servers.

To reinstate secure client/server messaging, restart each server with the geode.disallow-internal-messages-without-credentials system property set to true. For example:

gfsh>start server --name=server_name --dir=server_config_dir \
--J=-Dgeode.disallow-internal-messages-without-credentials=true

Resolved Issues

The Tanzu GemFire 9.10 release is based on the Apache Geode 1.12 release, which also includes bug fixes from the Apache Geode 1.11 release. This section describes issue resolutions that significantly affect Tanzu GemFire applications. For a list of all issues resolved in Apache Geode 1.12, see the Geode 1.12 Release Notes. For a list of all issues resolved in Apache Geode 1.11, see the Geode 1.11 Release Notes. Ticket numbers of the form GEODE‑NNNN can be inspected at the Apache Software Foundation site for Geode issues.

Issues Resolved in Tanzu GemFire 9.10.9

GEODE-9141, GEM-3219: Fixed a corrupted DestroyRegionMessage that could cause a cache server to hang during shutdown.

GEODE-9180: Improved diagnostic logging by adding a warning when a heartbeat generation thread oversleeps.

GEODE-9295, GEM-3310: Corrected an issue with entry idle expiration that could disable all expiration, resulting in symptoms such as a drastic increase in session state region entry counts following an upgrade.

GEODE-9346, GEM-2816: Introduced an option allowing a client application to enable automatic retries in response to PdxSerializationException by setting the system property gemfire.enableQueryRetryOnPdxSerializationException=true.

GEODE-9380, GEM-3301: Addressed a significant performance degradation in peer-to-peer TLS handshake times by replacing sleep() calls with thread yields.

Issues Resolved in Tanzu GemFire 9.10.8

GEODE-9064, GEM-3185: Serialization filtering for JMX/RMI is configured by default on Java 11.

GEODE-9146, GEM-3223: Entry idle expiration on regions with eviction policy set to “destroy” will now expire entries consistently across the cluster.

GEODE-9289, GEM-3253: Addressed an issue with deserialization of configuration information from old locators during rolling upgrades.

GEODE-9307, GEM-3293: Corrected increased heap consumption following auto-reconnection caused by region references that should have been destroyed following a forced disconnect.

GEODE-9331, GEM-3300: Improved the efficiency of creating peer-to-peer connections when conserve-sockets=false by eliminating a redundant list of weak references.

GEODE-9339, GEM-3302: Upgraded json-smart to address CVE-2021-27568.

GEODE-9363, GEM-3306: Upgraded spring-core to address CVE‑2021‑22118.

Issues Resolved in Tanzu GemFire 9.10.7

GEODE-8217, GEM-2909: Fixed a problem with session state serialization and deserialization.

GEODE-8221, GEM-2908: Fixed a failure to update session state.

GEODE-8513, GEM-3030: Cleaned up inefficiencies and spurious errors when storing and retrieving session state information.

GEODE-8558, GEM-3077: Fixed syntax issue in the syntax for commented Pulse queries.

GEODE-8623, GEM-3088: Corrected a timing issue between DNS and Geode startup that could result in permanent unknown host exceptions.

GEODE-8671, GEM-3095: Fixed a serialization data corruption issue that arose when two threads simultaneously accessed a single PdxInstance.

GEODE-8684, GEM-3254: Ensured that the maxInactiveInterval setting is honored when the commit valve is disabled.

GEODE-8781, GEM- 3252: Fixed a problem with upgrades when attempting to restart from API-configured regions.

GEODE-8815, GEM-3128: Fixed a case in which cache closure due to an uncaught exception during member startup was incorrectly treated as a graceful shutdown.

GEODE-8926, GEM-3184: Fixed a problem with missed Continuous Query events when they occurred at the same time a post-CQ function was executing.

GEODE-8958, GEM-3027: Improved tombstone expiration logic with regard to the handling of future timestamps.

GEODE-8974, GEM-3205: Fixed vulnerability CVE-2020-13956 in apache-httpclient.

GEODE-8989, GEM-3211: Fixed vulnerability CVE-2021-22112 in spring-security.

GEODE-8996, GEM-3158: Re-established backward compatibility for rebalance and restore redundancy commands.

GEODE-9010, GEM-3218: Fixed vulnerability CVE-2020-27223 in jetty.

GEODE-9016, GEM-3212: Fixed a null-pointer exception that could occur when a continuous query generated a local-destroy event.

GEODE-9030, GEM-3221: Fixed a spurious Region Destroyed exception that could occur when a query was initiated on a partitioned region immediately following a startup or rebalance operation.

GEODE-9040, GEM-3215: Improved the handling of missing colocated regions, so that the SingleThreadColocationLogger now stops, as it should.

GEODE-9051: Added a feature to measure tenured heap consumption and record the information in the logs after garbage collection.

GEODE-9126, GEM-3251: Fixed vulnerability CVE-2021-28165 and CVE-2021-28164 in jetty.

Issues Resolved in Tanzu GemFire 9.10.6

GEODE-2644, GEM-3138: Restored member name to log entries.

GEODE-5922, GEM-3155: Fixed a performance issue in the event queues of WAN-connected clusters.

GEODE-7884, GEM-1535: Fixed a condition in which a server would hang during a cache close operation with an IllegalStateException error, due to a timer being set on an already-completed operation.

GEODE-8261, GEM-2975, GEM-3130: Fixed a null-pointer error that could occur when a client attempts to register interest after server shutdown has been initiated.

GEODE-8419, GEM-3079: SSL/TLS protocol, cipher suite configurations and per-component TLS properties were being ignored in some cases; these settings are now respected.

GEODE-8447, GEM-3019: The output of localized dates in log timestamps and Pulse query results now always include seconds. A previous fix for GEODE-8447 revised Pulse to display dates using the local time zone.

GEODE-8536, GEM-2872, GEM-2961, GEM-2997: Fixed a stack overflow that could occur when Lucene IndexWriter was unable to be created.

GEODE-8685, GEM-3174: Fixed export feature to no longer deserialize region values and classes.

GEODE-8686, GEM-1681: Dispensed with a rarely-called tombstone removal optimization that could occasionally cause a deadlock that prevented the completion of region creation.

GEODE-8721, GEM-3110: Fixed a condition under which the losing side of a network partition failed to shut down.

GEODE-8734, GEM-3104: Implemented a more robust approach for generating Geode statistics that accommodates a variety of /proc/net/netstat formats produced by different versions of Linux.

GEODE-8764, GEM-3148: Fixed an authentication problem that erroneously blocked the execution of Lucene queries requiring region-level permissions.

GEODE-8779, GEM-3143: Eliminated a case in which session management messages were sent unnecessarily to clients whose local caches were not enabled.

GEODE-8782: Added the ability to retrieve the Principal from the FunctionContext when a SecurityManager is enabled.

GEODE-8795, GEM-3111: Lucene queries now utilize security post-processing, if enabled.

GEODE-8799, GEM-3041: Addressed a performance issue by increasing the defaults for maximum number of pooled message processor threads and maximum partitioned region message processor threads. If your system depends on constraining system resources to the old default values, you can set these properties explicitly using DistributionManager.MAX_THREADS and DistributionManager.MAX_PR_THREADS, respectively.

The following table shows the old and new maximum values.

GEODE-8895, GEM-3170: Handle an unexpected socket closure with more grace by implementing retries internally, such that the client no longer needs to deal with an InternalGemFireException.

GEODE-8930, GEM-2677: Fixed a race condition that caused a deadlock when executing a create operation within a transaction with conserve-sockets set to true. With this fix, the deadlock no longer occurs, but best practice is to set conserve-sockets to false when using transactions.

Issues Resolved in Tanzu GemFire 9.10.5

GEODE-8238, GEM-2901: Improved connection close behavior to ensure final delivery of cluster messages during shutdown and avoid potential hangs.

GEODE-8432, GEM-2778, GEM-3026, GEM-3029: Fixed a hang that occurred with asynchronous event queues during rebalance operations.

GEODE-8475, GEM-1589: Resolved a deadlock in ParallelGatewaySenderQueue.

GEODE-8520, GEM-3056: GarbageCollectionCount metric no longer shows negative values. GCStatsMonitor now sums up all the GC stats to get the total GC count and GC time.

GEODE-8557, GEM-3066, GEM-3067: Reclassified “java.lang.IllegalStateException: NioSslEngine has been closed” to IOException, which allows retries of server connection failures.

GEODE-8564, GEM-3081: Fixed an exception thrown by an attempt to remove a managed connection that has already been invalidated. Corrected functionality to avoid unnecessary intermediary collection creations.

GEODE-8584, GEM-3092: Corrected a peer-to-peer messaging failure with TLS when attempting to replicate data while a connection is being closed.

GEODE-8651, GEM-3109: Corrected a problem that caused TLS-encrypted messaging to hang when conserve-sockets is set to false.

GEODE-8652, GEM-3099: Corrected a synchronization issue that caused TLS-encrypted message transmission to hang during shutdown.

Issues Resolved in Tanzu GemFire 9.10.4

GEODE-8385, GEM-2936: Fixed a problem with cluster shutdown which caused a hang on restart, due to ambiguity as to which node had the most up-to-date disk store. This could occur if nodes (locators or servers) were simultaneously shut down.

GEODE-8463, GEODE-8506, GEM-3020: Refined the behavior of BufferPool to always return a buffer that has exactly the requested capacity. In the past, BufferPool could return a buffer larger than the size requested, which could lead to decryption errors and lost messages when using TLS(SSL) protocol TLSv1.3.

GEODE-8478, GEM-3069: Fixed an issue that caused a gateway sender to shut down if (1) its alert-threshold was configured, (2) an event was on the queue longer than that specified threshold, (3) the logger was attempting to record an alert for that message, and (4) a field in the event object threw an exception while undergoing toString conversion.

GEODE-8489, GEM-3043: Restored the behavior of Pulse queries for consistency with earlier versions. For example, the result of a select * query once again includes attributes without values.

Issues Resolved in Tanzu GemFire 9.10.3

GEODE-6564, GEM-3001: Fixed a memory leak that occurred when a replicated region, configured with entry expiration, was cleared.

GEODE-8331: Fixed an issue that prevented gfsh v1.12 (GemFire 9.10) from connecting to a Geode v1.10 (GemFire 9.9) server. Now, if a Geode v1.10 (GemFire 9.9) command is incompatible with a command issued by the newer version of gfsh, it will fail with a detailed error message.

GEODE-8394, GEM-2989: Fixed an issue in which putAll operations or put operations with large objects could result in data corruption if invoked without having a sufficiently large read-timeout. Subsequent get operations or queries could experience failures due to this corruption.

GEODE-8447, GEM-3019: Pulse displays dates using the local time zone.

GEODE-8483, GEM-3004: Introduced a correction to JCA transaction logic that restores the ability of the system to detect commit conflict exceptions in concurrent access situations.

Issues Resolved in Tanzu GemFire 9.10.2

GEODE-8174, GEM-2884: Fixed an issue that caused an incorrect ConcurrentModificationException to be thrown when using JTA transactions.

• GEODE-8029, GEM-2919: Unused disk store backups (.drf files) are now deleted to prevent the proliferation of unused records and files, which could cause members to fail during startup while recovering disk stores.

GEODE-8131, GEM-2938: Eliminated a point of contention in the alert logging system that could occasionally block cache operations.

GEODE-8195: Improved WAN retry logic.

GEODE-8240: After a rolling upgrade, even though all locators were upgraded, the membership view indicated some were running the old version. This would have been evident in the gfsh list members command and in logs. The correct version number is now reported.

GEODE-8259, GEM-2943: When a client performs a single-hop getAll() operation and encounters a serialization error, the operation is now re-tried.

GEODE-8287, GEM-2897, GEM-2950: Ironed out a recently-introduced problem that caused a degradation in performance for join queries or queries that use multiple indexes. Performance in such cases has been restored to v8.2 levels.

Issues Resolved in Tanzu GemFire 9.10.1

GEODE-7851: Pulse logging has been restored.

GEODE-7940: Fixed a problem in which a parallel gateway sender stopped sending events if another gateway sender attached to the same region was destroyed.

GEODE-8055, GEM-2890: Indexes can now be created on subregions.

GEODE-8071, GEM-2896: Fixed the gfsh rebalance command to prevent locators from hanging during shutdown.

GEODE-8091, GEM-2898: Locators failed to start after upgrade to v9.10 if a deprecated gfsh start locator option, --load-cluster-configuration-from-dir, was in use. Support for the deprecated option has been restored to allow startup to proceed. Beginning with GemFire 9.5 and Geode 1.6, the gfsh command import cluster-configuration supersedes start locator --load-cluster-configuration-from-dir.

Issues Resolved in Tanzu GemFire 9.10

GEODE-7465, GEM-2773: Fixed an issue that threw RegionDestroyedException when an asynchronous event queue was stopped and then restarted.

GEODE-7473, GEM-2774: Fixed a memory leak in the accumulation of entry event objects that could occur when a gateway receiver existed for the same region as a gateway sender.

GEODE-7593, GEM-2795: Fixed a memory leak that caused a higher than expected heap size when eviction was enabled. The issue was due to indexes that retained references to serialized values when the indexed key was a PDX string.

GEODE-7374, GEM-2748: Tanzu GemFire no longer throws a ClassCastException when using the deprecated MemberCommandService.

GEODE-7373, GEM-2746: Enforce JMX credentials to be either a string or an array of strings.

GEODE-7036, GEM-2723: Fixed a bug that could result in a NullPointerException when restarting members and a locator rejoins the cluster to form a quorum.

GEODE-7264, GEM-2706: Updated the Jackson dependency to version 2.10.0.

GEODE-7334, GEM-2705: Tanzu GemFire no longer throws a ClassCastException when the developer REST API is enabled and JodaModule is in the classpath.

GEODE-7208, GEM-2698: Fixed Lucene index creation when there are non-primitive fields in a child class, and the fields are defined in the parent class.

GEODE-7268, GEM-2692: A gfsh alter region no longer causes a soft hang when many gateway senders exist, by making the gfsh command implementation work more like the equivalent Java API.

GEODE-7535, GEM-2654: Fixed a race condition that could improperly result in an EntryDestroyedException during a getAll operation, if a client proxy region exists and there are no transactions.

GEODE-7085, GEM-2647: Eliminated an IndexOutOfBoundsException while recovering values from disk when the region version value overflowed.

GEODE-7015, GEM-2604: Fixed a bug that could result in a hung server if a move bucket operation failed during rebalance due to a forced disconnect with recreated persistent partitioned regions.

GEODE-7042: The server launcher now waits until all of the server’s startup tasks have completed before updating the server’s status to “online”. The ServerLauncher.start() method now blocks until redundancy recovery and recovery of values from disk have completed. This change has the effect of making the gfsh start server command wait for all server startup tasks to complete before returning.

GEODE-4993: GatewaySender connection statistics are now stored after being captured.

GEODE-6973: Improved creation time performance when there are a large number of PDX types.

GEODE-7628, GEM-2747: Block JMX MBean creation when no SecurityManager interface is implemented.

GEODE-7755, GEM-2614: Fixed a bug that resulted in locator processes continuing to run even when the logs show them shut down.

GEODE-7728, GEM-2819: Fixed an exception thrown when executing an equi-join query and both fields are indexed.

GEODE-7294: Moved to Spring 5 and updated other third-party libraries to their latest security patch versions.

GEODE-7310: Fixed an issue in which taking a backup was not properly aborted if a member of the distributed system was lost during the backup process.

GEODE-7750, GEODE-7760, GEODE-7796, GEM-2821, GEM-2823: Fixed an issue which could cause a locator to fail to restart properly, and also corrupted the cluster configuration.

GEODE-7763: Reduced the performance degradation caused when a client has multiple threads, and each client thread does operations on the same region key.

Support

General support includes security vulnerability resolutions and critical bug fixes in all supported minor versions, while other maintenance is applied only to the latest supported minor release.

Obtaining and Installing Security Updates

New versions of Tanzu GemFire often include important security fixes, so VMware recommends you keep up to date with the latest releases.

For details about any security fixes in a particular release, see the Application Security Team page.