Thursday, 17 September 2009 16:53

Database Synchronization

Rate this item
(0 votes)

13.1 Synchronization and Mobility

Isolated mobile databases are just discrete islands of data packets. Backend databases play an essential role to glue mobile databases together to form a complete IT network.

  • Most business processes require field data to be aggregated at the back end for centralized decision making and access control.

  • Mobile users need to get periodic data updates from the enterprise back end. For example, a sales representative needs to update his inventory database with new data from the warehouse server several times a day.

  • The backend database is a content provisioning hub for mobile databases. This reduces the complexity of distributed data management.

  • The backend database is also a backup repository for mobile databases. This reduces the chance of data loss from dead batteries or lost devices.

13.1.1 The Disconnected but Synchronized Architecture

The need for the backend database does not always justify always-on connectivity. For most applications, the changes on both ends are not time critical and several-times-a-day updates are more than enough. That allows us to design disconnected but synchronized mobile applications. Under that architecture, the backend databases do not need to synchronize every little change at real time. It is a scalable and reliable solution (see Figure 13.1).

Figure 13.1. The disconnected but synchronized mobile application architecture.


 

The technical challenge for the synchronized mobile database architecture is to develop efficient, secure, and reliable two-way communication channels between mobile databases and back ends. This is where mobile database vendors sell their value-added proprietary synchronization solutions.

13.1.2 Mobile Database Synchronization

Mobile database vendors offer synchronization engines that synchronize their mobile databases to either their own or third-party backend databases. A typical synchronization engine works as the following. The process is also illustrated in Figure 13.2.


Figure 13.2. The synchronization process.
 
  • The synchronization engine (a.k.a. server) is part of the mobile middleware. It connects to both backend databases and the mobile devices it serves.

  • We can configure the synchronization server and specify which backend tables are available (publication) to which users. Filters and custom conflict resolution logics can also be programmed into the synchronization server.

  • The synchronization server connects to backend databases using open protocols like JDBC or database-specific protocols. Proprietary protocols allow the synchronization engine to take advantage of database-specific features and optimization.

  • Mobile databases connect to the synchronization servers via proprietary protocols. Those protocols take advantage of vendor-specific optimization to reduce bandwidth usage, minimize workload on the device side, and enhance security.


Since database synchronization is highly vendor dependent, we discuss solutions from different vendors in the next several sections. Detailed discussions on vendor-specific tools and APIs are beyond the scope of this book. Interested readers should refer to vendor documentation and technical support. The source code examples listed in this chapter are only for illustration purposes.


13.2 PointBase UniSync

PointBase provides a pure Java-based, platform-independent synchronization engine called UniSync. UniSync (v4.5) can synchronize enterprise databases (Oracle, DB2, Sybase, and MS SQL Server) and workgroup databases (PointBase Embedded) with PointBase Micro mobile databases. The key concepts in UniSync are hubs and spokes. The synchronization process is as follows:

  1. Create corresponding databases and tables on both the backend server and mobile devices.

  2. Create a hub on the synchronization server. The hub contains publications that specify the backend tables (or partial tables) available for synchronization (publish).

  3. Use the hub object to create spokes. Spokes are objects on the synchronization server representing mobile devices. Each spoke has an ID. It can subscribe to the publications in the same hub through subscription objects. Using a spoke ID, the mobile device connects to the matching spoke and synchronizes to the subscribed backend tables.

  4. Start the synchronization server. This basically involves executing the main() method of class com.pointbase.me.sync.Server. The server class is available in PointBase distribution package. There are several ways to run the server in different environments. Please refer to PointBase documentation for more details and example scripts. By default, the server listens at port 8124.

  5. Initiate the synchronization process using a spoke ID and spoke stub classes residing on the mobile devices.

Figure 13.3 illustrates the architectural concepts of hubs and spokes.

Figure 13.3. Hub and spokes in PointBase UniSync.

 

13.2.1 Serverside Code Walk Through

Listing 13.1 uses example code from UniSync's basicMicro_to_Oracle sample application to illustrate the synchronization process between a PointBase Micro database and an Oracle backend database. On the synchronization server side, we first create a hub and a publication to expose the tables to be synchronized. Then, we create a spoke that subscribes to that publication. Finally, we run the server at a specific port.

Listing 13.1. PointBase UniSync server application
import com.pointbase.unisync.api.*;



// A PB Embedded DB is used by the sync

// engine to store metadata.

String PB_DRIVER = "com.pointbase.jdbc.jdbcUniversalDriver";

String PB_EMBEDDED_URL = "jdbc:pointbase:embedded:HUBDB,new";

String PB_USER = "PBPUBLIC";

String PB_PASSWORD = "PBPUBLIC";



// Parameters to access the Oracle back end

String DB_DRIVER = "oracle.jdbc.driver.OracleDriver";

String DB_URL = "jdbc:oracle:";

String DB_USER = "PBPUBLIC";

String DB_PASSWORD = "PBPUBLIC";



// Those point to the same Oracle database

String UNISYNC_URL = "jdbc:pointbase:oracle:";

String UNISYNC_DRIVER =

"com.pointbase.driver.jdbc.oracle.OracleDriver";



// Configuration constants

String DATA_SOURCE = "Data";

String HUB_NAME = "Hub1";

String PUB = "Pub1";

String SUB = "Sub1";

String SPOKE_NAME = "Spoke1";

String SPOKE_PASSWORD = "password";



// Tables to be synchronized

String[] tableNames = new String[] {DB_USER + ".NAMECARD"};



// The Sync manager uses a PB Embedded DB for metadata

SyncManager manager = SyncManager.getInstance(PB_EMBEDDED_URL,

PB_DRIVER, PB_USER, PB_PASSWORD);



// Get or create a hub

Hub hub=manager.getHub(HUB_NAME);

if (hub == null)

hub = manager.createHub(HUB_NAME);



// Get or create a backend data source

SyncDataSource dataSource = manager.getSyncDataSource(DATA_SOURCE);

if (dataSource == null)

dataSource = manager.createSyncDataSource(DATA_SOURCE,

UNISYNC_URL, UNISYNC_DRIVER, DB_USER, DB_PASSWORD);



// Get or create a publication of

// the specified tables

Publication pub = hub.getPublication(PUB);

if (pub == null) {

pub = hub.newPublication(PUB, DATA_SOURCE, tableNames);

// Make the Publication available to Spokes

hub.publish(pub);

}



// Get or create a spoke that

// subscribes to the publication

SpokeConfig spoke = hub.getSpokeConfig(SPOKE_NAME);

if (spoke == null) {

spoke = hub.createSpokeConfig(SPOKE_NAME);

spoke.savePassword(SPOKE_PASSWORD);

// Subscribe to the Publication

Subscription sub = spoke.newSubscription(SUB,

SyncDataSource.DEFAULT,PUB);

spoke.subscribe(sub);

}



// Start the server at the default port 8124

// or the port specified by command line parameter

// -Dtcp.listenerPort = hub.startServer();

13.2.2 Clientside Code Walk Through

On the client side, a mobile device contacts the server to obtain a spoke and synchronizes tables that are subscribed to that spoke (Listing 13.2). The client API works on both the CDC and CLDC.

Listing 13.2. PointBase UniSync client side application
import com.pointbase.me.jdbc.*;


// User database parameters

String PB_MICRO_URL = "jdbc:pointbase:micro:SPOKEDB";

String PB_DRIVER = "com.pointbase.me.jdbc.jdbcDriver";

String PB_USER = "PBPUBLIC";

String PB_PASSWORD = "PBPUBLIC";



// How to connect the sync server

HUB_URL = "http://sync.server:8124";



// Constants to define the Spoke name

// and the password

String SPOKE_NAME      = "Spoke1";

String SPOKE_PASSWORD  = "password";



// First establish a DB connection to PB_MICRO_URL

Class.forName(PB_DRIVER);

Connection conn = DriverManager.getConnection(PB_MICRO_URL,

                           PB_USER, PB_PASSWORD);

// SyncManager stores temp info in local DB

SyncManager manager=SyncManager.getInstance(conn);



// Get or create a spoke

Spoke spoke = manager.getSpoke(SPOKE_NAME);

if (spoke == null) {

  spoke=manager.createSpoke(SPOKE_NAME);



  // Save the URL of the Hub, so that the

  // Spoke can communicate with the Hub

  spoke.saveHubURL(HUB_URL);



  // Save the Spoke's password. This should be

  //  same as the one provided in the Hub side.

  spoke.savePassword(SPOKE_PASSWORD);



  // Load the configuration from the Hub

  // This will also create the NameCard table

  spoke.loadConfig();



  // This creates the NameCard table on the device

  spoke.getSnapshot();

}



// Call sync to synchronize the databases

spoke.sync();

13.3 IBM DB2 Everyplace

DB2e databases and IBM FastRecordStores (see Section 12.3) synchronize with backend DB2 Universal databases or any other JDBC data sources through the DB2e Sync server. In DB2e v8.1, multiple combinations of synchronization clients, transport mechanisms and backend data sources are supported through the adaptor architecture (Figure 13.4).

Figure 13.4. The adaptor architecture in DB2 synchronization engine.
 
  • The synchronization client on the mobile device can handle DB2e databases, native PIM lists and files.

  • The clientside synchronization engine communicates with the synchronization server via HTTP, WAP or Bluetooth. Both proprietary binary protocols and SyncML are supported.

  • The synchronization server supports the DB2 Universal Database, any JDBC database as well as Domino and Exchange servers.

We can start the Sync server from the Mobile Devices Administration Center in DB2e's server console. Using the administration center UI, we can create users and subscription sets, associate users with subscription sets, specify encryption levels, and define conflict resolution logic. On the client side, DB2e Sync runs natively on three mobile platforms: Palm OS, Windows CE, and Symbian OS. Using device-native GUIs, a mobile user can login to the Sync server, download the subscription sets, manage subscription sets, and synchronize mobile databases. For more information on how to configure and run the DB2e Sync server and client, please refer to DB2e documentation (see "Resources").

13.3.1 Access DB2e Sync Programmatically

The above synchronization process requires human interaction with a standalone client program. That is not always convenient. DB2e provides a set of J2ME APIs that allow users to access the DB2e Sync server programmatically. Those Java APIs are just JNI wrappers over DB2e's native Sync methods. Listing 13.3 illustrates the use of such APIs.

Listing 13.3. DB2e Sync client application
import com.ibm.mobileservices.isync.*;

import com.ibm.mobileservices.isync.db2e.jni.*;



// Get a synchronization provider

ISyncProvider provider = DB2eISyncProvider.getInstance();



// Connect to an IBM Sync server

// "host, port, userID, passwrd"

// specify parameters of



// the remote IBM Sync server.

ISyncService service = provider.createSyncService(host, port,

                             userID, passwrd);



// Specify a directory to sync into

ISyncConfigStore config = service.getConfigStore("path");



// Manage available subscription sets from

// the client

ISyncSubscriptionSet [] subsets = config.getSubscriptionSets();

for ( int i; i < subsets.length; i++ ) {

  ISyncSubscriptionSet subset = subsets[i];

  // Enable synchronization of all tables

  subset.enable();

}



ISyncDriver syncer = config.getSyncDriver();



// Synchronize. There could be three return values

//

// ISync.RTN_SUCCEEDED: synchronization succeeded

// ISync.RTN_CANCELED: synchronization canceled

// ISync.RTN_FAILED: synchronization failed

int rc = syncer.sync();



// Detailed results for each subscription set

//

// ssArr[i].getStatus() returns:

// ISync.STATUS_READY, ISync.STATUS_COMPLETED,

// STATUS_CANCELED or ISync.STATUS_FAILED

ISyncSubscriptionSet ssArr[] = config.getSubscriptionSets();

for (int i=0; i < ssArr.length; i++ ) {

  System.out.print ("Subscription Set: " + ssArr[i].getName() +

                    " Status: " + ssArr[i].getStatus());



// Close resources

syncer.close();

config.close();

service.close();

13.3.2 Sync with MIDP FastRecordStore

DB2e supports a mechanism to synchronize backend databases with its proprietary FastRecordStores on MIDP devices. The following code snippet (Listing 13.4) illustrates the synchronization process. If this is the first time we synchronize, a new FastRecordStore with the same table name will be created and populated with data rows from the table.

Listing 13.4. Using DB2e Sync to synchronize MIDP FastRecordStore
import com.ibm.mobileservices.isync.*;

import com.ibm.mobileservices.isync.midp.*;



ISyncProvider provider = MIDPISyncProvider.getInstance();

ISyncService service = provider.createSyncService(host, port,

                               user, password);

ISyncConfigStore config = service.getConfigStore(null);

ISyncDriver syncer = config.getSyncDriver();



int rc = syncer.sync();



syncer.close();

config.close();

service.close();

13.4 iAnywhere Solutions MobiLink

MobiLink is the synchronization engine in iAnywhere Solutions' SQL Anywhere Studio. It synchronizes iAnywhere Adaptive Server Anywhere and UltraLite mobile databases with enterprise databases, including Oracle, IBM DB2, Microsoft SQL, and Sybase Adaptive Server Enterprise. Important features of MobiLink are the following:

  • Flexible synchronization logic through user-defined sync scripts.

  • The synchronization script is written for and stored in the enterprise database, which allows developers to use SQL dialects that they are already familiar with.

  • Multiple communication protocols are supported for synchronization streams. Those protocols include TCP/IP, HTTP, and direct serial line.

  • Each synchronization operation is a transaction with guaranteed integrity.

  • Adjustable parameters (such as cache size, maximum number of threads) for performance tuning.

  • Strong security through user authentication and 128-bit encrypted synchronization data streams.


13.4.1 MobiLink via Standalone Native Clients

MobiLink can be manually invoked by native clients on devices. We go through the following steps to complete a synchronization cycle.

  1. Create a synchronization username and password through the Sybase Central administration panel.

  2. In the Sybase Central administration panel's MobiLink plug-in, select tables that are available for synchronization from backend databases.

  3. Write SQL scripts for each sync event. For example, an upload_insert event might trigger a simple SQL INSERT statement, while a download_cursor event might trigger an SQL SELECT statement. We can skip this step and use the default scripts if there is no custom synchronization logic.

  4. On the remote (mobile) database, create a publication containing tables to synchronize using the statement CREATE PUBLICATION.

  5. On the remote database, create a user and associate a publication with the user through a subscription. Related statements are CREATE SYNCHRONIZATION USER and CREATE SYNCHRONIZATION SUBSCRIPTION. The subscription also contains the synchronization server address and the data communication protocol (e.g., TCP/IP, HTTP, or serial link).

  6. Run the remote sync client.

Note

The concepts of publication and subscription in Sybase MobiLink is different from other synchronization engines.

13.4.2 Access MobiLink Programmatically

The MobiLink synchronization client can be invoked as a command-line utility. But for autogenerated UltraLite databases, the MobiLink API is already built in. The following code snippet (Listing 13.5) demonstrates how to synchronize the entire clientside Java UltraLite database.


Listing 13.5. Using MobiLink
UlSynchOptions synch_opts = new UlSynchOptions();

synch_opts.setUserName( "userid" );

synch_opts.setPassword( "passwd" );

synch_opts.setScriptVersion( "default" );

// We use TCP/IP socket as the sync transport.

synch_opts.setStream( new UlSocketStream() );

synch_opts.setStreamParms( "host=192.128.10.1" );

((JdbcConnection) conn) synchronize( synch_opts );

If you have a homogeneous environment containing only iAnywhere databases, Sybase has another synchronization product called SQL Remote. SQL Remote supports message-based asynchronous operation and is therefore scalable. Interested readers should refer to iAnywhere documentation for that technology.

13.5 Oracle9i Mobile Server

The Oracle9i Mobile Server is part of Oracle9i Application Server (AS). It works with native synchronization clients distributed with Oracle9i Lite databases. The 9i synchronization engine has the following features.

  • If our application was created using the Oracle9i Lite Mobile Development Kit, the Mobile Server could automatically generate synchronization logic for the application.
  • Oracle9i Lite supports synchronization over any TCP/IP-based network, including HTTP, CDPD, and 802.11b Wireless LAN. We may also add new transports by using the Mobile Server Open Transport APIs.
  • The synchronization server provides rich controls over conflict resolution, data subsetting, and security settings.
  • The Mobile Server supports asynchronous synchronization. During "rush hours," each device just submits the synchronization content in a queue and leaves. The server processes the queue when it is not busy. The client later retrieves updates from the server from another queue. This allows it to support a large number of mobile devices at a time.

The Oracle Mobile Server synchronization engine is also available programmatically through administration and clientside APIs. However, those APIs are available only in C/C++, not in Java yet.

13.6 The Synchronized Contact Manager

Now, we revisit the PointBase Contact Manager application discussed in Section 11.11. We can use PointBase's UniSync engine to simplify both the data backup and
database provisioning processes. Figure 13.5 demonstrates the synchronized Contact Manager application in action.
 

Synchronized Contact Manager peers in action

 


Using techniques and APIs we learned in Section 13.2, we use class ResetServer to set up the hub, publications, and spokes on the UniSync server. Listing 13.6 shows the relevant code snippets from class ResetServer.

Listing 13.6. Snippet from the ResetServer class to setup the UniSync server
// Connects to the backend server

manager = SyncManager.getInstance(caturl,catdriver, catuser,catpassword);

String dsname;

dsname=SyncDataSource.DEFAULT;



String hubname="Hub";

Hub hub=manager.createHub(hubname);



Publication pub;

String pubname;

SpokeConfig spoke;

Subscription sub;

String subname="SubNameCard";

String tablename="NAMECARD";

String[] tables=new String[]{tablename};



// publish the complete namecard table

pubname="PubNameCard";

pub=hub.newPublication(pubname,dsname,tables);

hub.publish(pub);



// create two spokes and subscribe

// to this publication

for(int i=1;i<=2;i++) {

  String name="Spoke"+i;

  spoke=hub.createSpokeConfig(name);

  spoke.savePassword("pass"+i);

  sub = spoke.newSubscription(subname, SyncDataSource.DEFAULT,pubname);

  spoke.subscribe(sub);

}



// publish the namecard table; without

// the picture column



pubname="PubNameCardNoPicture";

pub=hub.newPublication(pubname,dsname,tables);

SyncTable table=pub.getSyncTable(tablename);

table.dropSyncColumns(new String[]{"PICTURE"});

hub.publish(pub);



// create two spokes and subscribe to

// this publication

for(int i=3;i<=4;i++) {

  String name="Spoke"+i;

  spoke=hub.createSpokeConfig(name);

  spoke.savePassword("pass"+i);

  sub = spoke.newSubscription(subname, SyncDataSource.DEFAULT,pubname);

  spoke.subscribe(sub);

}

manager.close();

On the client side, since Contact Manager's database access layer is isolated, we mainly need to make changes to the DBManager class, and the rest of the application will automatically take advantage of the synchronization features. The following code snippet (Listing 13.7) from DBManager demonstrates how to obtain the spoke stub and process the synchronization on the device side. The comments embedded in the code illustrate the differences between the synchronized version and the local version of the application.


Listing 13.7. Refactor the DBManager class in the Contact Manager example to take advantage of database synchronization
// Import proprietary classes for sync

import com.pointbase.me.jdbc.*;



class DBManager {



  // In addition to JDBC connection variables

  // we also need to define variables for sync

  // ... ...

  private Spoke spoke;

  private String spokename;

  private int spoke_id;

  private int spoke_range_start,spoke_range_end;

  final static int ROWS_PER_SPOKE=1<<16;

  private String syncurl;

  private String syncpassword;





  private DBManager() {



    // get DB connection parameters

    // ... ...



    // get sync parameters

    syncurl = properties.getProperty("syncurl",

                                    "http://localhost:8124");

    String spokeid = properties.getProperty("spokeid", "1");

    spokename = properties.getProperty("spoke", "Spoke"+spokeid);

    syncpassword = properties.getProperty("syncpassword",

                                         "pass"+spokeid);

    url = properties.getProperty("url",

                "jdbc:pointbase:micro:pbdemo"+spokeid);



    connect();

  }



  // The complete connect method using

  // synchronization server

  private void connect() {

    try {

      // Connecting to the database...



      Class.forName(driver);



      // If the database doesn't exist,

      // create a new database.

      connection = DriverManager.getConnection(url, user, password);

      statement = connection.createStatement();



      // Check sync metadata and create tables

      loadMeta();



      // Creating prepared statements

      createStatement();



    } catch (Exception e) {

      e.printStackTrace();

      System.exit(1);

    }

    }



    // The complete newID method using the sync server

    private int getNewID() {

      try {

        ResultSet rs = statement.executeQuery(

         "SELECT MAX(ID)+1 FROM NameCard WHERE "+

         "ID>=" + spoke_range_start +

         " AND ID<"+spoke_range_end);

        rs.next();

        int id=rs.getInt(1);

        if(rs.wasNull()) {

          return spoke_range_start;

        } else {

          return id;

        }



      } catch (Exception e) {

        e.printStackTrace();

      }

      return 0;

    }



    // Create table and load metadata

    // from the sync hub

    void loadMeta() {

      try {

        SyncManager manager = SyncManager.getInstance(connection);

        spoke=manager.getSpoke(spokename);

        if(spoke==null) {

          System.out.println(

              "Loading MetaData from url "+syncurl+

              " for spoke "+spokename+

              " using password "+syncpassword);

          spoke=manager.createSpoke(spokename);

          spoke.savePassword(syncpassword);

          spoke.saveHubURL(syncurl);

          spoke.loadConfig();

          spoke.getSnapshot();

        }

        spoke_id = spoke.getSpokeId();

        System.out.println("SpokeID is "+spoke_id);

        spoke_range_start = ROWS_PER_SPOKE * spoke_id;

        spoke_range_end = spoke_range_start + ROWS_PER_SPOKE - 1;

     }  catch (SyncException e) {

        e.printStackTrace();

     }

   }



   // Synchronize spoke databases (mobile databases)

   // with the hub and backend databases

   void sync() {

     try {

       spoke.sync();

     } catch (SyncException e) {

       e.printStackTrace();

     }

  }



  // Other data access methods are the same as the

  // non-synced version.

}


 

Last modified on Wednesday, 23 September 2009 20:48
Vicky

Vicky

E-mail: This e-mail address is being protected from spambots. You need JavaScript enabled to view it