How Communication Works
GemFire uses a combination of TCP and UDP unicast and multicast for communication between members. You can change the default behavior to optimize communication for your system.
Client/server communication and gateway sender to gateway receiver communication uses TCP/IP sockets. The server listens for client communication at a published address and the client establishes the connection, sending its location. Similarly, the gateway receiver listens for gateway sender communication and the connection is established between sites.
In peer systems, for general messaging and region operations distribution, GemFire uses either TCP or UDP unicast. The default is TCP. You can use TCP or UDP unicast for all communications or you can use it as the default but then can target specific regions to use UDP multicast for operations distribution. The best combination for your installation depends in large part on your data use and event messaging.
TCP (Transmission Control Protocol) provides reliable in-order delivery of the system messages. TCP is more appropriate than UDP if the data is partitioned, if the distributed system is small, or if network loads are unpredictable. TCP is preferable to UDP unicast in smaller distributed systems because it implements more reliable communications at the operating system level than UDP and its performance can be substantially faster than UDP. As the size of the distributed system increases, however, the relatively small overhead of UDP makes it the better choice. TCP adds new threads and sockets to every member, causing more overhead as the system grows.
Note: Even when GemFire is configured to use UDP for messaging, GemFire uses a TCP connection when attempting to detect failed members. See Failure Detection and Membership Views for more details. In addition, the TCP connection’s ping is not used for keep alive purposes; it is only used to detect failed members. See TCP/IP KeepAlive Configuration for TCP keep alive configuration.
UDP (User Datagram Protocol) is a connectionless protocol which uses far fewer resources than TCP. Adding another process to the distributed system incurs little overhead for UDP messaging. UDP on its own is not reliable however, and messages are restricted in size to 64k bytes or less, including overhead for message headers. Large messages must be fragmented and transmitted as multiple datagram messages. Consequently, UDP is slower than TCP in many cases and unusable in other cases if network traffic is unpredictable or heavily congested.
UDP is used in GemFire for both unicast and multicast messaging. GemFire implements retransmission protocols to ensure proper delivery of messages over UDP.
UDP unicast is the alternative to TCP for general messaging. UDP is more appropriate than TCP for unicast messaging when there are a large number of processes in the distributed system, the network is not congested, cached objects are small, and applications can give the cache enough processing time to read from the network. If you disable TCP, GemFire uses UDP for unicast messaging.
For each member, GemFire selects a unique port for UDP unicast communication. You can restrict the range used for the selection by setting
membership-port-range in the
gemfire.properties file. Example:
In addition to UDP port configuration, the
membership-port-range property defines the TCP port used for failure detection. See the Reference for a description of the GemFire property.
Your options for general messaging and for default region operations messaging is between TCP and UDP unicast. You can choose to replace the default with UDP multicast for operations distribution of some or all of your regions. For every region where you want to use multicast, you set an additional attribute on the region itself.
When multicast is enabled for a region, all processes in the distributed system receive all events for the region. Every member receives each message for the region and has to unpack it, schedule it for processing, and then process it, all before discovering whether it is interested in the message. Multicasting is suitable, therefore, for regions that are of general interest in the distributed system, where most or all members have the region defined and are interested in receiving most or all messages for the region. Multicasting should not be used for regions that are of little general interest in the distributed system.
Multicast is most appropriate when the majority of processes in a distributed system are using the same cache regions and need to get updates for them, such as when the processes define replicated regions or have their regions configured to receive all events.
Even if you use multicast for a region, GemFire will send unicast messages when appropriate. If data is partitioned, multicast is not a useful option. Even with multicast enabled, partitioned regions still use unicast for almost all purposes.