WSO2 Deployment Synchronizer - Sharing deployment artifacts across a product cluster

This post is about a new feature in WSO2 Carbon product platform. I will use the latest versions of WSO2 Governance Registry(WSO2-G-reg-4.1.0) and WSO2 Enterprise Service Bus (ESB-4.0.2) for the demonstration.

Before going through the configuration steps, lets look at the problem which is going to be addressed using Deployment Synchronizer.

Suppose we have a WSO2 ESB product cluster with a single READ-WRITE node and a several READ-ONLY nodes which shares a common configuration registry. When we deploy a proxy service from READ-WRITE node, in order for other nodes to be synced with that new service deployment, all READ-ONLY nodes have to be restarted. This is a painful concern in a large product cluster.

WSO2 Deployment Synchronizer feature has been implemented to resolve that concern. With that, once you deploy a service (or any deployable artifact such as ESB sequence, scheduled task etc..) from one node in a product cluster, the other nodes automatically get the changes and sync up with the master (or READ-WRITE) node.

The Deployment Synchronizer makes use of two different approaches for deployment artifact synchronization.

1. SVN based synchronizer

2. Registry based synchronizer

Lets look at each of these in detail.

Pre-Requisites:

Download WSO2 ESB-4.0.2 and WSO2 Governance Registry-4.1.0 binary distributions from wso2 oxygen tank

Product Clustering Setup

Lets proceed through setting up a two node WSO2 Carbon product cluster as shown above.

Step 1:

Extract the downloaded WSO2ESB-4.0.2.zip and make two copies of it as wso2esb-rw and wso2esb-ro
wso2esb-rw directory is used as the master node of the cluster and wso2esb-ro node will be used as the slave node.

Step 2:

Extract the downloaded WSO2greg-4.1.0.zip into a new directory. This will be used as the central governance and configuration registry in our ESB cluster.

Step 3:
The above WSO2 G-reg instance will run on a mysql DB instead of the default H2 database. Therefore, lets create a mysql DB first.
Open a mysql prompt in your server. Type the following commands to create a database and assign user privileges.

mysql>create database reg_db;
mysql>use reg_db;
mysql>grant all on reg_db.* TO regadmin@localhost identified by "regadmin";

Edit the CARBON_HOME/repository/conf/registry.xml of the WSO2 G-reg server as follows.

<currentDBConfig>mysql-reg</currentDBConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>

<dbConfig name="mysql-reg">
<url>jdbc:mysql://localhost:3306/reg_db</url>
<userName>regadmin</userName>
<password>regadmin</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>5</maxActive>
<maxWait>60000</maxWait>
<minIdle>50</minIdle>
<validationQuery>SELECT 1</validationQuery></dbConfig>

Now, copy mysql jdbc driver (mysql-connector-java-5.1.7-bin.jar or later) to CARBON_HOME/repository/component/lib directory of WSO2Greg server and start the server with -Dsetup switch.

sh wso2server.sh -Dsetup

This will start WSO2-Greg server on mysql.

Step 4

Now, we have started central governance and configuration registry instance of our WSO2 ESB product cluster. Now, lets proceed with configuring WSO2 ESB nodes.
Lets configure read-write node first.

We are going to run 3 carbon servers in the same machine. Therefore, we need to change the port index in CARBON_HOME/repository/conf/carbon.xml so that each of the WSO2 ESB nodes will run on their own ports without conflicting with each other.
In carbon.xml, change the following element in order to run the ESB read-write node in HTTP port 9764 and HTTPS port 9444.


<Offset>1</Offset>

We are going to store the configuration data of ESB nodes in the cluster in /_system/esbnodes space of the above registry. Also, the governance data will be stored in /_system/governance directory. Therefore, lets add the following registry mounts through CARBON_HOME/repository/conf/registry.xml.

<dbConfig name="mysql-reg">
<url>jdbc:mysql://localhost:3306/reg_db</url>
<userName>regadmin</userName>
<password>regadmin</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>5</maxActive>
<maxWait>60000</maxWait>
<minIdle>50</minIdle>
<validationQuery>SELECT 1</validationQuery></dbConfig>

<remoteInstance url="https://localhost:9443/registry">
<id>conf-gov-registry</id>
<dbConfig>mysql-reg</dbConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
</remoteInstance>

<!-- Governance data will be stored in /_system/governance collection of central registry instance --> 
<mount overwrite="true" path="/_system/governance">
<instanceId>conf-gov-registry</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>

<!-- Configuration data will be stored in /_system/esbnodes collection of central registry instance -->  
<mount overwrite="true" path="/_system/config">
<instanceId>conf-gov-registry</instanceId>
<targetPath>/_system/esbnodes</targetPath>
</mount>

Copy mysql jdbc driver (mysql-connector-java-5.1.7-bin.jar or later) to CARBON_HOME/repository/component/lib directory of WSO2 ESB read-write node and start the server.

sh wso2server.sh

Step 5

We have configured and started the READ-WRITE node of ESB cluster. Now, we can configure the READ-ONLY node. The configuration is almost same as READ-WRITE node except the highlighted elements given below.

First, change the port offset to 2 so that the ports will not be conflicted with the other server ports.

<Offset>2</Offset>

Change the default NIO HTTP and HTTPS ports in CARBON_HOME/repository/conf/axis2.xml as follows.

<transportReceiver class="org.apache.synapse.transport.nhttp.HttpCoreNIOListener" name="http">
<parameter locked="false" name="port">8281</parameter>

<transportReceiver class="org.apache.synapse.transport.nhttp.HttpCoreNIOSSLListener" name="https">
<parameter locked="false" name="port">8244</parameter>

Add the following registry mounts through CARBON_HOME/repository/conf/registry.xml of ESB READ-ONLY node.

<dbConfig name="mysql-reg">
<url>jdbc:mysql://localhost:3306/reg_db</url>
<userName>regadmin</userName>
<password>regadmin</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>5</maxActive>
<maxWait>60000</maxWait>
<minIdle>50</minIdle>
<validationQuery>SELECT 1</validationQuery></dbConfig>

<remoteInstance url="https://localhost:9443/registry">
<id>conf-gov-registry</id>
<dbConfig>mysql-reg</dbConfig>
<readOnly>true</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
</remoteInstance>

<!-- Governance data will be stored in /_system/governance collection of central registry instance --> 
<mount overwrite="true" path="/_system/governance">
<instanceId>conf-gov-registry</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>

<!-- Configuration data will be stored in /_system/esbnodes collection of central registry instance -->  
<mount overwrite="true" path="/_system/config">
<instanceId>conf-gov-registry</instanceId>
<targetPath>/_system/esbnodes</targetPath>
</mount>

Copy mysql jdbc driver (mysql-connector-java-5.1.7-bin.jar or later) to CARBON_HOME/repository/component/lib directory of WSO2 ESB read-only node and start the server.

sh wso2server.sh

Now we are done with the clustering setup but we have not done any configurations related to the deployment synchronizer yet.
We will look in to the registry based deployment synchronization first.

Step 6 - Registry based deployment synchronizer

In this mode, once you deploy an artifact from the READ-WRITE node, the artifacts will be stored in the relevant collection in the configuration registry. The other nodes of the cluster will use the registry checkin-checkout client and checkout the deployment artifacts to their file system. Then with the hot deployment functionality, the artifacts will get deployed in the cluster nodes.

Lets look at how we can use the registry based deployment synchronizer.

• Log in to management console of ESB READ-WRITE node (https://localhost:9444/carbon)
• Navigate to Configure --> Deployment Synchronizer UI

• Select Auto Commit option and click on Enable
• Log in to management console of ESB READ-ONLY node (https://localhost:5/carbon)
• Access Configure --> Deployment Synchronizer UI
• Select Auto Checkout option and click on Enable
  

• Create a proxy service (eg:-proxy1) in READ-WRITE node
• After 60 seconds (the default synchronization period), log in to the management console of the READ-ONLY node of the cluster.
• You will notice that the proxy1 is listed in the services list of READ-ONLY node
  

Step 7 - SVN based deployment synchronizer

Deployment synchronization can be achieved using a SVN repository as well. Lets look at SVN based deployment synchronizer.

In this mode, instead of a registry, we use a subversion repository as the deployment artifact store. As we check-in files to SVN, the synchronizer use a SVN client API and commit and update deployment artifacts periodically by using a SVN location.

• First, revert the registry based deployment synchronizer settings which we did above. (Disable deployment synchronizer in Configure --> Deployment Synchronizer UI)

• Have a proper SVN location. You can use an existing SVN location or create a new one.
• Add the following configuration in CARBON_HOME/repository/conf/carbon.xml of both ESB nodes. Make sure to specify correct SVN credentials and location URL according to your environment.
  

<DeploymentSynchronizer>
<Enabled>true</Enabled>
<AutoCommit>true</AutoCommit>
<AutoCheckout>true</AutoCheckout>
<RepositoryType>svn</RepositoryType>
<SvnUrl>https://svn.wso2.com/wso2/custom/projects/projects/qa/deployment-synchronizer/esb</SvnUrl>
<SvnUser>qasvn</SvnUser>
<SvnPassword>test</SvnPassword>
<SvnUrlAppendTenantId>false</SvnUrlAppendTenantId>
</DeploymentSynchronizer>

• Restart both ESB nodes.
  
• Deploy a proxy service from either one of the nodes
• Changes will get reflected into other nodes after 60 seconds (default synchronization period)
  

We looked at two different ways of sharing deployment artifacts among cluster nodes. If you come across any issues when configuring either one of the above approaches, please drop me a mail.