WSO2 Deployment Synchronizer - Sharing deployment artifacts across a product cluster

This post is about a new feature in WSO2 Carbon product platform. I will use the latest versions of WSO2 Governance Registry(WSO2-G-reg-4.1.0) and WSO2 Enterprise Service Bus (ESB-4.0.2) for the demonstration.
Before going through the configuration steps, lets look at the problem which is going to be addressed using Deployment Synchronizer.
Suppose we have a WSO2 ESB product cluster with a single READ-WRITE node and a several READ-ONLY nodes which shares a common configuration registry. When we deploy a proxy service from READ-WRITE node, in order for other nodes to be synced with that new service deployment, all READ-ONLY nodes have to be restarted. This is a painful concern in a large product cluster.
WSO2 Deployment Synchronizer feature has been implemented to resolve that concern. With that, once you deploy a service (or any deployable artifact such as ESB sequence, scheduled task etc..) from one node in a product cluster, the other nodes automatically get the changes and sync up with the master (or READ-WRITE) node.
The Deployment Synchronizer makes use of two different approaches for deployment artifact synchronization.

1. SVN based synchronizer
2. Registry based synchronizer

Lets look at each of these in detail.

Pre-Requisites:
Download WSO2 ESB-4.0.2 and WSO2 Governance Registry-4.1.0 binary distributions from wso2 oxygen tank

Product Clustering Setup


Lets proceed through setting up a two node WSO2 Carbon product cluster as shown above.

Step 1:

Extract the downloaded WSO2ESB-4.0.2.zip and make two copies of it as wso2esb-rw and wso2esb-ro
wso2esb-rw directory is used as the master node of the cluster and wso2esb-ro node will be used as the slave node.

Step 2:

Extract the downloaded WSO2greg-4.1.0.zip into a new directory. This will be used as the central governance and configuration registry in our ESB cluster.

Step 3:
The above WSO2 G-reg instance will run on a mysql DB instead of the default H2 database. Therefore, lets create a mysql DB first.
Open a mysql prompt in your server. Type the following commands to create a database and assign user privileges.

mysql>create database reg_db;
mysql>use reg_db;
mysql>grant all on reg_db.* TO regadmin@localhost identified by "regadmin";

Edit the CARBON_HOME/repository/conf/registry.xml of the WSO2 G-reg server as follows.
<currentDBConfig>mysql-reg</currentDBConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>

<dbConfig name="mysql-reg">
<url>jdbc:mysql://localhost:3306/reg_db</url>
<userName>regadmin</userName>
<password>regadmin</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>5</maxActive>
<maxWait>60000</maxWait>
<minIdle>50</minIdle>
<validationQuery>SELECT 1</validationQuery></dbConfig>



Now, copy mysql jdbc driver (mysql-connector-java-5.1.7-bin.jar or later) to CARBON_HOME/repository/component/lib directory of WSO2Greg server and start the server with -Dsetup switch.

sh wso2server.sh -Dsetup

This will start WSO2-Greg server on mysql.

Step 4

Now, we have started central governance and configuration registry instance of our WSO2 ESB product cluster. Now, lets proceed with configuring WSO2 ESB nodes.
Lets configure read-write node first.

We are going to run 3 carbon servers in the same machine. Therefore, we need to change the port index in CARBON_HOME/repository/conf/carbon.xml so that each of the WSO2 ESB nodes will run on their own ports without conflicting with each other.
In carbon.xml, change the following element in order to run the ESB read-write node in HTTP port 9764 and HTTPS port 9444.


<Offset>1</Offset>

We are going to store the configuration data of ESB nodes in the cluster in /_system/esbnodes space of the above registry. Also, the governance data will be stored in /_system/governance directory. Therefore, lets add the following registry mounts through CARBON_HOME/repository/conf/registry.xml.
<dbConfig name="mysql-reg">
<url>jdbc:mysql://localhost:3306/reg_db</url>
<userName>regadmin</userName>
<password>regadmin</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>5</maxActive>
<maxWait>60000</maxWait>
<minIdle>50</minIdle>
<validationQuery>SELECT 1</validationQuery></dbConfig>

<remoteInstance url="https://localhost:9443/registry">
<id>conf-gov-registry</id>
<dbConfig>mysql-reg</dbConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
</remoteInstance>

<!-- Governance data will be stored in /_system/governance collection of central registry instance -->
<mount overwrite="true" path="/_system/governance">
<instanceId>conf-gov-registry</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>

<!-- Configuration data will be stored in /_system/esbnodes collection of central registry instance -->
<mount overwrite="true" path="/_system/config">
<instanceId>conf-gov-registry</instanceId>
<targetPath>/_system/esbnodes</targetPath>
</mount>


Copy mysql jdbc driver (mysql-connector-java-5.1.7-bin.jar or later) to CARBON_HOME/repository/component/lib directory of WSO2 ESB read-write node and start the server.

sh wso2server.sh

Step 5

We have configured and started the READ-WRITE node of ESB cluster. Now, we can configure the READ-ONLY node. The configuration is almost same as READ-WRITE node except the highlighted elements given below.

First, change the port offset to 2 so that the ports will not be conflicted with the other server ports.

<Offset>2</Offset>

Change the default NIO HTTP and HTTPS ports in CARBON_HOME/repository/conf/axis2.xml as follows.

<transportReceiver class="org.apache.synapse.transport.nhttp.HttpCoreNIOListener" name="http">
<parameter locked="false" name="port">8281</parameter>

<transportReceiver class="org.apache.synapse.transport.nhttp.HttpCoreNIOSSLListener" name="https">
<parameter locked="false" name="port">8244</parameter>


Add the following registry mounts through CARBON_HOME/repository/conf/registry.xml of ESB READ-ONLY node.
<dbConfig name="mysql-reg">
<url>jdbc:mysql://localhost:3306/reg_db</url>
<userName>regadmin</userName>
<password>regadmin</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>5</maxActive>
<maxWait>60000</maxWait>
<minIdle>50</minIdle>
<validationQuery>SELECT 1</validationQuery></dbConfig>

<remoteInstance url="https://localhost:9443/registry">
<id>conf-gov-registry</id>
<dbConfig>mysql-reg</dbConfig>
<readOnly>true</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
</remoteInstance>

<!-- Governance data will be stored in /_system/governance collection of central registry instance -->
<mount overwrite="true" path="/_system/governance">
<instanceId>conf-gov-registry</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>

<!-- Configuration data will be stored in /_system/esbnodes collection of central registry instance -->
<mount overwrite="true" path="/_system/config">
<instanceId>conf-gov-registry</instanceId>
<targetPath>/_system/esbnodes</targetPath>
</mount>


Copy mysql jdbc driver (mysql-connector-java-5.1.7-bin.jar or later) to CARBON_HOME/repository/component/lib directory of WSO2 ESB read-only node and start the server.

sh wso2server.sh

Now we are done with the clustering setup but we have not done any configurations related to the deployment synchronizer yet.
We will look in to the registry based deployment synchronization first.

Step 6 - Registry based deployment synchronizer

In this mode, once you deploy an artifact from the READ-WRITE node, the artifacts will be stored in the relevant collection in the configuration registry. The other nodes of the cluster will use the registry checkin-checkout client and checkout the deployment artifacts to their file system. Then with the hot deployment functionality, the artifacts will get deployed in the cluster nodes.

Lets look at how we can use the registry based deployment synchronizer.

  • Log in to management console of ESB READ-WRITE node (https://localhost:9444/carbon)
  • Navigate to Configure --> Deployment Synchronizer UI

  • Select Auto Commit option and click on Enable
  • Log in to management console of ESB READ-ONLY node (https://localhost:5/carbon)
  • Access Configure --> Deployment Synchronizer UI
  • Select Auto Checkout option and click on Enable
  • Create a proxy service (eg:-proxy1) in READ-WRITE node
  • After 60 seconds (the default synchronization period), log in to the management console of the READ-ONLY node of the cluster.
  • You will notice that the proxy1 is listed in the services list of READ-ONLY node

Step 7 - SVN based deployment synchronizer

Deployment synchronization can be achieved using a SVN repository as well. Lets look at SVN based deployment synchronizer.

In this mode, instead of a registry, we use a subversion repository as the deployment artifact store. As we check-in files to SVN, the synchronizer use a SVN client API and commit and update deployment artifacts periodically by using a SVN location.

  • First, revert the registry based deployment synchronizer settings which we did above. (Disable deployment synchronizer in Configure --> Deployment Synchronizer UI)

  • Have a proper SVN location. You can use an existing SVN location or create a new one.
  • Add the following configuration in CARBON_HOME/repository/conf/carbon.xml of both ESB nodes. Make sure to specify correct SVN credentials and location URL according to your environment.
<DeploymentSynchronizer>
<Enabled>true</Enabled>
<AutoCommit>true</AutoCommit>
<AutoCheckout>true</AutoCheckout>
<RepositoryType>svn</RepositoryType>
<SvnUrl>https://svn.wso2.com/wso2/custom/projects/projects/qa/deployment-synchronizer/esb</SvnUrl>
<SvnUser>qasvn</SvnUser>
<SvnPassword>test</SvnPassword>
<SvnUrlAppendTenantId>false</SvnUrlAppendTenantId>
</DeploymentSynchronizer>


  • Restart both ESB nodes.
  • Deploy a proxy service from either one of the nodes
  • Changes will get reflected into other nodes after 60 seconds (default synchronization period)

We looked at two different ways of sharing deployment artifacts among cluster nodes. If you come across any issues when configuring either one of the above approaches, please drop me a mail.

Comments

Waruna said…
FYI: You need to add the svn based configuration into the carbon.xml file which resided in CARBON_HOME/repository/conf directory
Charitha said…
Thanks Waruna. I updated the config steps.
Anonymous said…
Hi Charitha,

I think there is a typo in the Step 4 and 5 where we mount config registry space.

We need to mount /_system/esbnodes of the G-Reg instance. Therefore /_system/nodes should be /_system/esbnodes, right?
Charitha said…
Thanks Harshana. I fixed it
Waruna said…
You can also configure the registry based deployment synchroniser in the carbon.xml


true
true
false
Gnanaguru said…
Hi Charitha,

Thanks. Its a great post.

And I want to ask you, Is it enough for the clustering setup ?

Or I need to enable some clustering configuration in axis.xml. Because WSO2 clustering guides says it so.

Please share your thought.

Regards
Guru
Charitha said…
Hi Guru,
Axis2 clustering is not required for deployment synchronization for WSO2 Carbon version which this post is based on (ESB-4.0.3). However, for the next Carbon platform release (Carbon-4.0.0), you must enable axis2 clustering in order for dep Sync to work
riccardo said…
Hi, I deployed the clustering configuration with 2 nodes sharing the configuration registry with G-Reg. When I add UsernameToken security on a proxy Service I cannot authenticate with users defined inside the G-REG but only with users defined in the ESB. I believed that sharing configuration with G-reg would imply that all users where stored in the same DB, or am I missing something? In other words: how can I share Users and Roles configurations among all WSO2 nodes and G-reg?
many thanks
riccardo said…
Hi, sorry for my previous question, I solved it by using a mysql db store. Anyway I wanted to ask you about the SVN based deployment synchronizer: from my log I have that it can't commit: "commit for tenant failed"
[2012-09-03 13:48:33,275] ERROR - CarbonDeploymentSchedulerTask Deployment synch
ronization commit for tenant -1234 failed
java.lang.RuntimeException: org.wso2.carbon.deployment.synchronizer.DeploymentSy
nchronizerException: ERROR - CarbonDeploymentSchedulerTask Deployment synch
ronization commit for tenant -1234 failed
java.lang.RuntimeException: org.wso2.carbon.deployment.synchronizer.DeploymentSy
nchronizerException: "Error while committing artifacts to the SVN repository at org.wso2.carbon.deployment.synchronizer.internal.DeploymentSynchroniz
erServiceImpl.commit(DeploymentSynchronizerServiceImpl.java:105)..."

Any suggestion?

Popular posts from this blog

Common mistakes to avoid in WSO2 ESB - 1 - "org.apache.axis2.AxisFault: The system cannot infer the transport information from the URL"

Working with HTTP multipart requests in soapUI

How to configure soapUI to send HTTP chunked encoded requests