Contents: Basic operations Applied JavaSpaces Transactions in JavaSpaces Session sharing Fail-over configuration Load-balanced clustering Atomic session read-modify-write Testing JSCart Session state change notifications Resources About the author Rate this article Related content: Jini and PvC Make room for JavaSpaces Scaling Web services and applications with JavaGroups Subscriptions: dW newsletters dW Subscription (CDs and downloads)

Sing Li (mailto:westmakaha@yahoo.com?cc=&subject=Building adaptive, scalable solutions with JavaSpaces)
Author, Wrox Press
30 September 2003

Commodity PC-based servers and networking hardware can be combined with open source Java software to cost effectively scale-out Web services and application deployments. In this second installment of the High-impact Web tier clustering series, Sing Li dives into typical cluster system design scenarios and shows why a one-size-fits-all solution may not exist, while solutions based on JavaSpaces and Jini technology can be adaptively deployed to satisfy differing requirements.

At the Web tier in the J2EE architecture, where application servers reside, application state information is often kept in the form of server-side sessions. By externalizing these sessions and replicating them among a group of networked servers, you can create a scalable and highly available cluster for executing Java Web applications and Web services. In the first article of this series "Scaling Web services and applications with JavaGroups," we examined how this can be accomplished using the JavaGroups communications toolkit to implement in-memory session replication.

In this second article, we will take a slightly different approach to scaling Web applications and Web services across networked server clusters. The technology we will use is JavaSpaces, a software service delivered as part of the Jini technology family (see Resources). JavaSpaces enables you to take a higher-level approach to the design of distributed systems in general (and clustered systems in particular), reducing the complexity of the design and enhancing adaptability. By implementing a distributed shared memory model using JavaSpaces, we will focus our design on the sharing of session data across the cluster of servers (versus replication as we did in the first article).

Designing distributed systems with three basic operations
Conceptually, JavaSpaces are distributed bags in which we can place Java objects, called entries (see the sidebar "Jini entries"). A space can be concurrently shared by multiple users (clients). Users are typically other networked nodes or independent Java virtual machines (JVMs). After a Java object is placed into the space (with a write operation), any user of the space can read its content by removing it from the space (using a take operation) or by leaving it untouched (using a read operation).

These three simple operations (write, take, and read) elegantly encapsulate the complete premise of JavaSpaces, and can be used to enable distributed, massively parallel, or clustered computing network configurations. Figure 1 illustrates the operations performed with JavaSpaces:

Figure 1. Basic operations of JavaSpaces

In Figure 1, the template-matching mechanism of JavaSpaces (see the sidebar "Template matching in JavaSpaces") enables clients to selectively read or take objects out of the space.

Essence of JavaSpaces
The first article in this series dealt with a group of processes; tracking group membership; and coordinating, sending, or receiving messages (as in JavaGroups). JavaSpaces enables us to design a distributed system without being aware of messages, groups, or membership size. In fact, when using the read, write, and take operations exclusively, a JavaSpaces distributed application can be written completely single threaded -- with no concurrency awareness whatsoever! In other words, you can work on a very high level of abstraction when designing clustered systems using JavaSpaces. Two key advantages to working at such a high level are the ability to:

Jini entries A Jini entry is a Java object that implements the net.core.jini.Entry interface. This interface is a marker interface, a subclass of java.io.Serializable, with no required methods to implement. Every Jini entry can contain fields that are Java objects. These fields must be serializable as well. When using JavaSpaces, each field of an entry is serialized independently. The entry is not the root of the serialized object graph, which means that if two fields have references to the same object, two copies of the same referenced objects will be serialized separately.

• Keep the design very simple and easy to understand, facilitating long-term maintenance
• Quickly adapt to varying requirements without significant code changes

These advantages are desirable in almost all software systems, as well as traditionally complex distributed applications. Of course, the overall complexity of the distributed problem has not changed. What has changed is where that complexity is handled. With JavaSpaces, almost all of the more complex orthogonal concerns are pushed below the API line -- to the specific implementation of the JavaSpaces service -- leaving the designer to deal with the actual application problem. The rationale is that these complex problems now only have to be solved once, by capable expert engineers and theoreticians. To understand this architecture further, we must go beyond the conceptual and take a look the physical manifestation of JavaSpaces.

Applied JavaSpaces
Physically, a JavaSpace is a Java interface implemented by a Jini federated service (see the sidebar "All about Jini"). The interface, net.jini.space.JavaSpace, is the only access that a JavaSpaces client has to the functionality of the service. While the JavaSpaces service is a remote service, a local Jini proxy (similar in action to a downloaded driver) is supplied by the service to the application, making the calls to JavaSpaces interface all "local" within the application's VM. Table 1 shows a more detailed description of each operation.

Table 1. Basic JavaSpaces operations

Operation	Description
read	Searches for an entry in the space that matches the template supplied and returns a copy of the entry. This operation blocks until an entry is available or the timeout specified is reached.
take	Searches for an entry in the space that matches the template supplied, removes the entry from the space, and returns a copy of the entry. This operation blocks until an entry is available or the timeout specified is reached.
write	Places a copy of an entry into the space.

These basic operations operate on objects called Jini entries. See the sidebar "Jini entries" for information on how Java objects may be attached to these entries as fields.

In addition to the three basic operations, the JavaSpace interface also has the three frequently used methods shown in Table 2:

Template matching in JavaSpaces The read, take, readIfExists, takeIfExists, and notify methods all use JavaSpaces's template-matching mechanism to determine which entry in a JavaSpaces to work on. More specifically, a template is simply an entry that is partially or fully populated. The entry type must reflect the object type that you want to match, meaning that an entry of the specified Java class/interface or a subclass will match. Any fields that contain null will act as a wildcard during matching. Any field that is populated must match exactly on a serialized binary compare level. This sort of associative matching is conducive to massively parallel operation, and potentially hardware-assisted implementation. See the JSK documentation in the Resources section for more information on JavaSpaces template matching.

Transactions in JavaSpaces
JavaSpaces operations can be performed in the context of a transaction. This is accomplished by:

1. Locating a transaction manager service
2. Creating a transaction using the TransactionManager interface
3. Supplying the transaction when you call the JavaSpaces operation(s)

A set of operations performed in the context of a transaction can only have two outcomes -- successful (transaction committed) or unsuccessful (transaction aborted). The effect of the set of operations is only visible to other JavaSpaces clients upon transaction commit. If the transaction aborts, any effects of the JavaSpaces operations (that is, writes and takes that change the content of the space) are rolled back and no other client will know that the attempt ever happened.

A single transaction can combine operations that are performed over multiple JavaSpaces, involving geographically distributed implementations. Under the hood, the transaction manager coordinates a distributed two-phase commit protocol between the participating JavaSpaces service instances.

Transaction semantics greatly simplify the handling of partial failure modes by eliminating them conceptually. By combining a take of entry m from space A with a write to space B, we created an atomic transfer operation. If anything fails during the transfer, entry m is not removed from space A. If the transfer is successful, we know for sure that m is in space B and has been removed from space A.

Now that we have the JavaSpaces concepts firmly in mind, we are ready to apply them to our Web tier clustering problem. In the previous article, we observed that a scalable, highly available cluster can be constructed with the in-memory replication of application sessions across a group of distributed servlet containers. Using JavaSpaces, we can relax our focus on the details of how to handle the actual replication of the sessions. Instead, we'll focus on the higher-level semantics of session sharing.

Using distributed shared memory for session sharing
Using JavaSpaces, we can create a software analog to a shared memory system across a network. This concept enables us to share session information across all clients of a space. Figure 2 illustrates session sharing:

Figure 2. Using distributed shared memory to implement shared sessions

In Figure 2, the application sessions information is maintained in the distributed shared memory. Any change made to the shared sessions by one of the application servers is visible to all server instances. All servers read the shared session information from the same (networked shared) memory location.

Supporting a master/stand-by configuration for fail-over
We'll now attempt to create the distributed shared memory semantics using JavaSpaces. In this initial scenario, we have combined a take operation with a write operation inside a transaction to create an update operation. Listing 1 is the actual Java code that you can find in the source distribution:

public boolean updateSession(Serializable sess) {
        if ((spaceService == null) || (transactionService == null))
            return false;
        Transaction transact = null;
        try {
            Transaction.Created tc = 
              TransactionFactory.create(transactionService,
            TRANSACTION_LEASE);
            transact = tc.transaction;
        } catch (Exception ex) {
            ex.printStackTrace();
            return false;
        }
        
        try {
            SessionEntry tmpEntry = (SessionEntry) spaceService.take( 
               new SessionEntry(SESSION_KEY, null),
               transact, JavaSpace.NO_WAIT);
            if (tmpEntry == null)  {
                transact.abort();
                return false;
            }
            spaceService.write(new SessionEntry(SESSION_KEY, sess),
            transact,  LONG_LEASE);
            
            transact.commit();
        } catch (Exception ex) {
            
            ex.printStackTrace();
            try {
                transact.abort();  }
            catch (Exception ex2) {
                ex2.printStackTrace();
            }
            return false;
        }
        
        return true;
    }

In this code, the transaction makes the take and write combination atomic (see the sidebar "Why do we use a transaction?" for an explanation). Note that the write operation does not operate on the data taken from the space. Instead, it writes an image of a serializable object that is passed from the JSCart application (a session image maintained by the application).

This design is useful in systems that use fail-over capabilities through a master/stand-by configuration. In this configuration, a master server takes all of the incoming requests, and a stand-by server is switched-in only when the master server crashes or fails. Figure 3 illustrates this scenario:

Figure 3. Fail-over configuration

Why do we use a transaction? Because any entry retrieved through a take operation is no longer available to other users, it may initially appear that we do not need a transaction around the take and write operations of an update. The main purpose for the transaction, however, is not to guard against concurrent access of the entry, but rather to prevent a session from vanishing should a server crash after the take but before the write. Using a transaction ensures that the session entry gets restored to the pre-transaction value once the lease on the transaction expires.

Because it normally does not handle any incoming requests, the stand-by server in Figure 3 accesses the shared session information in only two ways:

1. Upon startup, to obtain the session state at that time
2. When the master server crashes, to obtain the most current session state

Between stages 1 and 2, however, the stand-by server has no need to perform any read from the shared memory. Indeed, it may actually carry stale session information between stages 1 and 2 because it is oblivious to session changes until it reads the shared memory at stage 2.

This loosely coupled configuration is sufficient for implementation of a session-sharing cluster that supports fail-over. In practice, every clustering application may have its own unique requirements. The above solution, while efficient, may not adequately address such requirements. Using JavaSpaces to implement clustered solutions enables us to adapt to varying design challenges with minimal design and code change. To observe this agility first hand, we can examine how we may create a more tightly coupled solution that supports load balancing in addition to fail-over.

Scaling out with load-balanced clustering
To maximize the use of available resources, some clustered applications may require the use of the stand-by server (in the previous scenario) for handling incoming requests. The main benefit of this configuration is that the request processing load is spread over several servers in the cluster (in the case where there is more than one stand-by server), thus enabling an application to scale to a larger concurrent user base (called scaling out). This configuration is commonly known as a load-balanced cluster configuration. Some implementations may direct an incoming request to the least loaded server, while other implementations may use round-robin request distribution schemes. Figure 4 illustrates this concept:

Figure 4. Load-balanced cluster

Fail-over is automatic in the load-balanced cluster. If one of the servers fails, all the new incoming requests are distributed to the remaining servers.

In load-balanced clusters, we cannot use the shared memory implementation in the first scenario. This is because the loosely coupled solution can cause session information corruption.

In this scenario, we can no longer assume that the master server will be the only one that modifies a specific shared session. Any locally kept session information can become stale and out of synchronization at any time. To solve this problem, we must ensure that the read-modify-write operation to shared session data is atomic.

Atomic session read-modify-write
To ensure that any server in the cluster can handle requests for a session at any time, we must place significant restrictions on the maintenance of session data. Namely, servers must not use any locally kept image of a shared session as a source of modification -- because the image can be stale and lead to corruption. Instead, servers must read the session data from the shared memory before modifying it.

We have updated the JavaGroups shopping cart application from the first article to use JavaSpaces network shared memory (see Resources for the download). This application illustrates the internals of a single shopping cart application session. Changes to the code are localized in only two classes, as shown in Table 3:

The division of labor between JSCart and CartSessionManager is clear. JSCart maintains the GUI and goes to CartSessionManager for all its session data needs. CartSessionManager implements its API using JavaSpaces network shared memory -- oblivious to its client. This implies that CartSessionManager has no knowledge of how data is contained within the session. Session details are opaque to the CartSessionManager. This makes the atomic read-modify-write cycle a little tricky to implement because:

• Only CartSessionManager knows how to read and write to shared memory
• Only JSCart knows how to meaningfully modify a session

In the code, read-modify-write is implemented with the help of an interface called com.ibm.devworks.javaspace.PrepReadModifyWrite. Listing 2 shows how this interface is defined:

public interface PrepReadModifyWrite {
    public void afterRead(Object readSession, Object token);
    
}

There is only a single method in this interface, called afterRead().

When JSCart needs to perform the read-modify-write operation, it calls the readModifyWrite() method of the CartSessionManager object, passing an object implementing the PrepReadModifyWrite interface (in our case, it is the JSCart class itself).

This call is made by JSCart when a user adds an item to the cart using the GUI. It is handled in the itemOrdered() method, as shown in Listing 3:

public void itemOrdered(OrderEvent ev) {
        System.out.println("JSCart received ordered event - " + ev);
        
        boolean success =  sessionMgr.readModifyWrite(this,
        (Object) new OrderEvent(ev.getDesc(), ev.getPrice())) ;
    }

This enables CartSessionManager within a transaction to atomically:

1. Perform the JavaSpaces take operation
   
   
2. Use the afterRead() method of the PrepReadModifyWrite interface to return the opaque session back to JSCart for modification
   
   
3. Perform the JavaSpaces write operation with the modified data

The need to seed a space With the current specification of JavaSpaces, there is no known generalized way for a group of n clients to reliably collaborate to "seed" a shared singleton entry into the space. This is the reason for the session seeding utility implemented by a SessionSeeder class. This utility writes the initial session entry into the space before any of the clients start up. Conceptually, we can see this as initializing the networked memory for sharing or mapping out the region to share.

Testing JSCart with distributed shared memory
We can try out this version of JSCart by following these steps:

1. Start the Jini and JavaSpaces environment in the code\jini2 directory. From the code\jini2 directory, type startup (be sure to follow all the README.TXT instructions in each directory to set up the system).
   
   
2. Use the seedspc.bat batch file in the scripts directory to run the session seeding utility by writing a session entry into the JavaSpace. From the code directory, type scripts\seedspc (see the sidebar "The need to seed a space" for more information on seeding the space).
   
   
3. In the scripts directory, run the runcart.bat batch file to execute two or more instances of JSCart. From the code directory, type scripts\runcart.

If you need to build the binaries from the source code, use the compile.bat batch file in the code\src directory.

JSCart works with only one shared session to keep things simple and visual. In a production scenario, you will typically have many session entries, and each session may be independently modified. You will also need a separate shared entry to hold all the session IDs.

Now try adding items into the first cart. This is equivalent to add-item requests coming in to the first server. Notice that the two carts appear to be out of synchronization, similar to Figure 5:

Figure 5. Visual JavaSpaces carts appear to be out of synchronization

However, if we now simulate requests for the same session on the second server/cart (by adding products into the second cart), we will see that the second cart immediately "catches up" with the items in the first cart. You can alternate the addition of items any number of times across any number of carts, simulating incoming requests for the same session being distributed to the different servers, and you will find that the shared session information continues to remain consistent.

While this session-sharing approach is adequate for load-balanced clustered servers with fail-over capabilities, it falls short of what we need. The visual shopping cart actually needs to be notified whenever a shared memory write occurs so that it can update its visual GUI. This requirement is likely more stringent than those found in most clustered system. The solution is also more bandwidth-expensive since every shared memory write will now cause a blast of notifications across the network. Regardless, JavaSpaces is once again up to the challenge, adapting to varying requirements of different distributed applications.

Using remote events for session state change notifications
JSCart needs to reflect the state of the shared sessions visually at any time. To satisfy this special requirement, JSCart instances must be notified whenever the session state changes. This means that all JSCart instances must be notified whenever a shared memory write occurs at any of the instances.

Our CartSessionManager class provides an addDistributedWriteListener() method specifically for this purpose. JSCart calls this method to register itself as the listener. JSCart implements the com.ibm.devworks.javaspace.DistributedWriteListener interface, detailed in Listing 4:

public interface DistributedWriteListener extends java.util.EventListener {
    public void sessionChanged(Object session);
}

CartSessionManager ensures that the sessionChanged() method is called whenever the shared session information has been modified. This enables JSCart to update its GUI with the latest session data.

Internally, CartSessionManager uses the JavaSpaces notify operation to implement the event notification. JavaSpaces uses Jini's remote event notification mechanism (see Resources) to make a remote method invocation (RMI) call back to the listening client when an entry matching a specific template is written to the JavaSpace. Before this can happen, the client must register a remote listener using the JavaSpaces notify() method. We can see this code in the registerRemoteEvent() method of the CartSessionManager class, shown in Listing 5:

private boolean registerRemoteEvent(Configuration config) {
        if ((spaceService == null) || (transactionService == null))
            return false;
        
        EventRegistration evtReg;
        leaseMgr = new LeaseRenewalManager();
        try {
            Exporter exp =   (Exporter) config.getEntry(
            CONFIG_COMP,
            "serverExporter", Exporter.class,
            new net.jini.jeri.BasicJeriExporter(
            net.jini.jeri.tcp.TcpServerEndpoint.getInstance(0),
            new net.jini.jeri.BasicILFactory(),
            false, true));
            
            evtReg = 
              spaceService.notify( new SessionEntry(SESSION_KEY, null),
            null,  (RemoteEventListener)
            exp.export(this),
            /* LeaseTime is 3 minutes */     3 * 60 * 1000 ,
            null);
            
            leaseMgr.renewFor(evtReg.getLease(),
            Lease.FOREVER, 3 * 60 * 1000, null);
        } catch (Exception ex) {
            ex.printStackTrace();
            return false;
        }
        return true;
        
    }

Leases and self-healing, long-lived networks Jini makes extensive use of limited duration leases to support the notion of a self-healing, long-lived network. A lease is granted by the resource holder whenever the networked resource (physical resources such as memory, or conceptual resources such as a transaction) is claimed or held on behalf of a client. For example, a transaction created by a transaction manager is leased. This ensures eventual release of the resource even if the Jini client that created the transaction crashes and never rejoins the network. Jini clients that want to hold on to resources beyond the granted lease duration must renew the lease before it expires. See the JSK documentation in the Resources section for more detailed information on the use of Lease in Jini.

Note that in Listing 5 we use a JavaSpaces LeaseRenewalManager helper class to keep the lease on the event renewed. Remote event registrations are leased in Jini because each registration can hold up certain networked resources. By delegating lease renewal to the LeaseRenewalManager helper class, we can be sure that the lease will not expire as long as this JSCart instance is running. Another worthwhile note is the use of the export() method on the object exporter to make our object accessible with RMI. The Jini 2.0 enhancement on RMI now requires explicit export of remote objects -- the automatic stub insertion will no longer occur if we subclass from an exporter object.

Figure 6 shows the events registration and remote notification sequence, showing how the shared session state change notifications are implemented:

Figure 6. Event registration and remote notification sequence

To try out JSCart with distributed notifications enabled, you will need to go through the source code of the JSCart and CartSessionManager classes and carefully uncomment the explicitly marked sections of code. Recompile using the rcompile.bat file in the code\src directory. This will also create the required RMI stubs and generate the jscart-dl.jar file. See the README.TXT files in the source code for more details.

Conclusions
The use of JavaSpaces and Jini technology enables us to design clustering systems on a high level. Using the three basic operations of read, take, and write, we have created a distributed shared memory model to share application server sessions within a cluster. JavaSpaces' support for transaction and leasing allows us to design around partial failures and focus on the session sharing mechanism.

Every distributed application has its own unique requirements, and it would be difficult -- if not impossible -- to create an API substrate that satisfies all applications generically. We have observed the different requirements inherent in a master/stand-by fail-over session-sharing cluster, versus those of a fully load-balanced session-sharing cluster configuration. Working at a high level adaptively, JavaSpaces allows us to create solutions that satisfy these applications with minimal code changes. Our visual shopping cart actually requires distributed session write notifications to work properly. JavaSpaces comes through once again through its support for space-write remote notification. The custom solution was implemented with no redesign and minimal code changes.

Figure 7 shows a continuous spectrum of design possibilities, from conceptual on the very top to the physical at the very bottom:

Figure 7. The conceptual to physical spectrum of available clustering technologies

At the top of the spectrum, solution design is significantly easier because we can work with objects and components that are closer to the problem at hand. For example, we may be able to work on a "session" object basis. At the bottom of the spectrum we have raw implementation details (such as "send this packet on the network"). While working at the lower levels may offer greater control and flexibility, the engineer will also need to do significantly more design, coding, and testing. Working at a higher level enables the engineer to create custom solutions rapidly and focus on the actual problem, but at the same time place more reliance and faith on the substrate's implementation.

Don't miss the rest of this series Part 1, "Scaling Web services and applications with JavaGroups" (July 2003)

Clustering solution designers can select supporting technology from anywhere in the spectrum. We can see that the JavaSpaces solution ranks quite high, while the JavaGroups solution that we looked at last time ranks slightly lower. In the next and final article in this series, we will take a closer look at an exciting emerging distributed systems fabrication technology that ranks even higher than JavaSpaces on the conceptual spectrum, and see how it can help us to do more with less code when designing high-impact Web tier clustering solutions.

Resources

• Download the JSCart source code and configuration files used in this article.
  
  
• You can find an active community of Jini and JavaSpaces users at the official Jini community Web site.
  
  
• The latest release of Jini and JavaSpaces reference implementation, along with detailed documentation, are found in the Jini Technology Starter Kit (JSK).
  
  
• See another innovative application of JavaSpaces -- parallel decoding of MP3 files -- in "Make room for JavaSpaces" by Sussane Hupfer (developerWorks, October 2000).
  
  
• Jini and mobile wireless devices stand poised to change the landscape of modern computing by facilitating pervasive computing. See "Jini and PvC" by Roman Vichr and Vivek Malhotra (developerWorks, July 2002) for an interesting discussion of this possibility.
  
  
• See in-depth technical discussions from actual JavaSpaces users and developers in the JavaSpaces users mailing list archive.
  
  
• See this site for a production-quality, commercial implementation of a JavaSpaces service.
  
  
• Learn about Beowulf clusters on Linux systems in this article by Andrew Blais (developerWorks, September 2001).
  
  
• Discover how to work with IBM WebSphere session manager in this article by Steve Eaton (developerWorks, August 2001).
  
  
• Take a tutorial on Linux clustering with MOSIX by Daniel Robbins (developerWorks, December 2001).
  
  
• Find hundreds of articles about every aspect of Java programming in the developerWorks Java technology zone.
  
  

About the author Sing Li is the author of Early Adopter JXTA and Professional Jini, as well as numerous other books with Wrox Press. He is a regular contributor to technical magazines and is an active evangelist of the P2P evolution. Sing is a consultant and freelance writer and can be reached at westmakaha@yahoo.com.

What do you think of this document? Killer! (5) Good stuff (4) So-so; not bad (3) Needs work (2) Lame! (1) Comments?

developerWorks > Java technology