LATEST VERSION: 9.1.1 - CHANGELOG
Pivotal GemFire® v9.1

Custom-Partition Your Region Data

By default, GemFire partitions each data entry into a bucket using a hashing policy on the key. Additionally, the physical location of the key-value pair is abstracted away from the application. You can change these policies for a partitioned region. You can provide your own data partitioning resolver and you can additionally specify which members host which data buckets.

Note: If you are colocating data between regions and custom partitioning the data in the regions, all colocated regions must use the same custom partitioning mechanism. See Colocate Data from Different Partitioned Regions.

For the default implementation of string-based partitioning, use org.apache.geode.cache.util.StringPrefixPartitionResolver. For standard partitioning, use org.apache.geode.cache.PartitionResolver. To implement fixed partitioning, use org.apache.geode.cache.FixedPartitionResolver.

Prerequisites

Procedure

  1. If using org.apache.geode.cache.PartitionResolver (standard partitioning) or org.apache.geode.cache.FixedPartitionResolver (fixed partitioning), implement the standard partitioning resolver or the fixed partitioning resolver in one of the following locations, listed here in the search order used by GemFire:

    • Custom class. You provide this class as the partition resolver to the region creation.
    • Entry key. You use the implementing key object for every operation on the region entries.
    • Cache callback argument. This implementation restricts you to using methods that accept a cache callback argument to manage the region entries. For a full list of the methods that take a callback argument, see the Region Javadocs.

    If using the default implementation of the string-based partition resolver, org.apache.geode.cache.util.StringPrefixPartitionResolver, specify all keys with the required syntax. Keys are strings, and contain the ’|’ character as a delimiter. The substring that precedes the ’|’ delimiter will used in the hash function that partitions the entry.

  2. If you need the resolver’s getName method, program that.

  3. If not using the default implementation of the string-based partition resolver, program the resolver’s getRoutingObject method to return the routing object for each entry, based on how you want to group the entries. Give the same routing object to entries you want to group together. GemFire will place the entries in the same bucket.

    Note: Only fields on the key should be used when creating the routing object. Do not use the value or additional metadata for this purpose.

    For example, here is an implementation on a region key object that groups the entries by month and year:

    Public class TradeKey implements PartitionResolver 
    { 
        private String tradeID; 
        private Month month; 
        private Year year; 
        public TradingKey(){ } 
        public TradingKey(Month month, Year year)
        { 
            this.month = month; 
            this.year = year; 
        } 
        public Serializable getRoutingObject(EntryOperation opDetails)
        { 
            return this.month + this.year; 
        }
    }
    
  4. For fixed partitioning only, program and configure additional fixed partitioning pieces:

    1. Set the fixed partition attributes for each member.

      These attributes define the data stored for the region by the member and must be different for different members. See org.apache.geode.cache.FixedPartitionAttributes for definitions of the attributes. Define each partition-name in your data host members for the region. For each partition name, in the member you want to host the primary copy, define it with is-primary set to true. In every member you want to host the secondary copy, define it with is-primary set to false (the default). The number of secondaries must match the number of redundant copies you have defined for the region. See Configure High Availability for a Partitioned Region.

      Note: Buckets for a partition are hosted only by the members that have defined the partition name in their FixedPartitionAttributes.

      These examples set the partition attributes for a member to be the primary host for the “Q1” partition data and a secondary host for “Q3” partition data.

      • XML:

        <cache>
           <region name="Trades">
              <region-attributes>
                 <partition-attributes redundant-copies="1">
                   <partition-resolver name="QuarterFixedPartitionResolver">
                      <class-name>myPackage.QuarterFixedPartitionResolver</class-name>
                   </partition-resolver>
                   <fixed-partition-attributes partition-name="Q1" is-primary="true"/>
                   <fixed-partition-attributes partition-name="Q3" is-primary="false" num-buckets="6"/>
                 </partition-attributes> 
              </region-attributes>
           </region>
        </cache>
        
      • Java:

        FixedPartitionAttribute fpa1 = FixedPartitionAttributes.createFixedPartition("Q1", true);
        FixedPartitionAttribute fpa3 = FixedPartitionAttributes.createFixedPartition("Q3", false, 6);
        
        PartitionAttributesFactory paf = new PartitionAttributesFactory()
             .setPartitionResolver(new QuarterFixedPartitionResolver())
             .setTotalNumBuckets(12)
             .setRedundantCopies(2)
             .addFixedPartitionAttribute(fpa1)
             .addFixedPartitionAttribute(fpa3);
        
        Cache c = new CacheFactory().create();
        
        Region r = c.createRegionFactory()
            .setPartitionAttributes(paf.create())
            .create("Trades");
        
      • gfsh:

        You cannot specify a partition resolver using gfsh.

    2. Program the FixedPartitionResolver getPartitionName method to return the name of the partition for each entry, based on where you want the entries to reside. GemFire uses getPartitionName and getRoutingObject to determine where an entry is placed.

      Note: To group entries, assign every entry in the group the same routing object and the same partition name.

      This example places the data based on date, with a different partition name for each quarter-year and a different routing object for each month.

      /**
       * Returns one of four different partition names
       * (Q1, Q2, Q3, Q4) depending on the entry's date
       */
      class QuarterFixedPartitionResolver implements
          FixedPartitionResolver<String, String> {
      
        @Override
        public String getPartitionName(EntryOperation<String, String> opDetails,
            Set<String> targetPartitions) {
      
           Date date = (Date)opDetails.getKey();
           Calendar cal = Calendar.getInstance();
           cal.setTime(date);
           int month = cal.get(Calendar.MONTH);
           if (month >= 0 && month < 3) {
              if (targetPartitions.contains("Q1")) return "Q1";
           }
           else if (month >= 3 && month < 6) {
              if (targetPartitions.contains("Q2")) return "Q2";
           }
           else if (month >= 6 && month < 9) {
              if (targetPartitions.contains("Q3")) return "Q3";
           }
           else if (month >= 9 && month < 12) {
              if (targetPartitions.contains("Q4")) return "Q4";
           }
           return "Invalid Quarter";
        }
      
        @Override
        public String getName() {
           return "QuarterFixedPartitionResolver";
        }
      
        @Override
        public Serializable getRoutingObject(EntryOperation<String, String> opDetails) {
           Date date = (Date)opDetails.getKey();
           Calendar cal = Calendar.getInstance();
           cal.setTime(date);
           int month = cal.get(Calendar.MONTH);
           return month;
        }
      
        @Override
        public void close() {
        }
      }
      
  5. Configure or program the region so GemFire finds your resolver for every operation that you perform on the region’s entries. How you do this depends on where you chose to program your custom partitioning implementation (step 1).

    • Custom class. Define the class for the region at creation. The resolver will be used for every entry operation. Use one of these methods:

      XML:

      <region name="trades">
          <region-attributes>
              <partition-attributes>
                  <partition-resolver name="TradesPartitionResolver"> 
                      <class-name>myPackage.TradesPartitionResolver
                      </class-name>
                  </partition-resolver>
              <partition-attributes>
          </region-attributes>
      </region>
      

      Java API:

      PartitionResolver resolver = new TradesPartitionResolver();
      PartitionAttributes attrs = 
          new PartitionAttributesFactory()
          .setPartitionResolver(resolver).create();
      
      Cache c = new CacheFactory().create();
      
      Region r = c.createRegionFactory()
          .setPartitionAttributes(attrs)
          .create("trades");
      

      gfsh:

      Add the option --partition-resolver to the gfsh create region command, specifying the package and class name of the custom partition resolver.

    • Entry key. Use the key object with the resolver implementation for every entry operation.

    • Cache callback argument. Provide the argument to every call that accesses an entry. This restricts you to calls that take a callback argument.

    If using the default implementation of the string-based partition resolver, configure with one of:

    XML:

    <region name="customers">
        <region-attributes>
            <partition-attributes>
                <partition-resolver name="StringPrefixPartitionResolver"> 
                    <class-name>org.apache.geode.cache.util.StringPrefixPartitionResolver
                    </class-name>
                </partition-resolver>
            <partition-attributes>
        </region-attributes>
    </region>
    

    Java API:

    PartitionAttributes attrs = 
        new PartitionAttributesFactory()
        .setPartitionResolver("StringPrefixPartitionResolver").create();
    
    Cache c = new CacheFactory().create();
    
    Region r = c.createRegionFactory()
        .setPartitionAttributes(attrs)
        .create("customers");
    

    gfsh:

    Add the option --partition-resolver=org.apache.geode.cache.util.StringPrefixPartitionResolver to the gfsh create region command.

  6. If your colocated data is in a server system, add the PartitionResolver implementation class to the CLASSPATH of your Java clients. The resolver is used for single hop access to partitioned region data in the servers.