Tips for a good ESB integration system

1. Build SOA Environment –

Share common logics around processes with exposed services.

Benefits:

Changes will be easy – If a common logic gets changed, then you would only need to deploy a service again and each process would work with the new logic.

Services logic would be shared around ALL applications – 

These days, almost every develop tool or language is able to consume a web-service. That means you can share logic easily around an organization. Use it in order to minimize duplicate code and maintenance.

2. Less-Code and Out Of The Box solutions –

More code = More complexity & maintenance. 

Use products for minimum coding, reliable solutions and support from professional vendors.
 

 

3. End-To-End Tracing –

A good BPM would always have tracing and information regarding messages being transferred in the wire.
 
Suppose message doing this route -
Application A --> Integration System --> Application B, Application C, Application D.
Make sure there's a field to identify the message in each step and each application writes to the same trace log.

Application A writes that message 12203 was created in a folder.
Integration System writes that message 12203 was taken and process started.
Integration System writes that message 12203 is now sent to Application B, Application C, Application D.
Application B writes that message 12203 was received.
etc.

Write important business and technical process data.

Many organizations use SQL Server for tracing but the innovative solution is to use ElasticSearch which includes free-text search on a huge mass of data.

Building reports for customers (based on those traces) is always a good thing because they can help with maintenance.

 

4. Cache layer for lookups –

Reading and writing to and from memory is faster than any I/O.

If many processes uses configurations or lookup values stored in SQL Server then consider an appropriate cache layer which uses memory to store data.

Redis & Memcached are classic for caching, but you can also use NoSQL solutions which stored data in memory first before being persisted (e.g couchbase).

5. XSLTs for Mapping -

DON'T write C#, Java or whatever language code in order to transform XML's.

Develop XML maps ONLY with XSLTs which can be reused in any integration tool which deals with XML transformation.

 

6. Use Stored Procedures for queries -

It's better for your application to execute stored procedure for queries, rather than sending those as text to the DB.

It's easier for a DBA to control & manage organization logic, and to avoid "BAD" queries.

 

7. Dynamic configuration –

Think deeply on every process configurations you should be able to modify on runtime without changing any code.

Don't exaggerate with dynamic configurations because it has a maintenance price. You do not want an interface with a huge number of configurations, if most of them are values which will be remain unchanged. 

 

8. Well-defined schemas –

It's important that schema nodes, elements and attributes would have real limitations (Min/Max occurs, Value types etc.).

It's possible to restrict schema field value types in many ways (int, string, regular expressions, enums etc.).

Benefits:

Data integrity – Well, that's the obvious reason. Data is more reliable on each point of entire process.

Less complexity in process – Suppose a process need to get an e-mail address from a request message.

If schema would enforce a valid e-mail regex in that field, then there is no need
to perform this validation on process.

 

9. ASync web-services is better than Sync web-services –

If there is a possibility of consuming asynchronous web-service – ALWAYS prefer it over a synchronous web-service.

Synchronic method (Two-Way) would open a session, send a request, wait for a response and only then closes the session.

Asynchrony method (One-Way) would open a session, send a request and closes the session immediately without waiting for a response.

More sessions = More CPU and more memory usage on server.

Take this tip in mind especially with offline interfaces. In those kind of processes the execution doesn't has be immediate so entire interface can last longer. 

 

10. Avoid direct-connection between Production & Test environments –

Some organizations tend to send messages from production to test\development environments.

That's really bad because if a resource isn't available in test\development then production would be affected.

 

 

11. Identical backup environment (or clustered in an active-passive mode) –

Maintain a backup (not-activated) environment for production (an exact copy of it).

Make sure backup environment is always up to date with the latest changes of the WORKING production environment.

 

12. Retry mechanism when using resources –

Consider using retry mechanism on a process which uses resources (SQL, Web-Services etc.).

Take notice you can implement retry only on OFFLINE processes/ASync services (i.e. when no one waits with an open session for an answer from that service).

 

13. Centralized production environments -

With the major communication progress during the last few years, geographic distance is not an issue as it used to be.

Less environments = Less maintenance.

A good centralized environment contains distributed servers which share data processing.
Some integration tools like "BizTalk" includes load balancing mechanism out of the box, but if your tool doesn't - a load balancing software would help.

 

14. Health and performance monitor system –

Monitor & alert crucial resources which is used by processes to early identify issues. "SCOM" and "Nagios" are popular products. Use "Watcher" for elasticsearch logs.

For instance: 

If a server CPU stays on 100% for the last 30 seconds – send an e-mail. 

If files are piling up in a directory which should be empty – send an SMS.

 

14. Archive source messages\requests –

Gives power for an integration system to rerun messages in case process went wrong and to be self-dependent from end-to-end.

 

15. Persistence –

Some integration tools (like BizTalk, which is DB-based) have persistence functionality built-in as part of their engine.
Products that put an emphasis on performance doesn't have it (e.g TIBCO) but it's possible to implement persistence logic inside a process.
Analyze each process before implementing and think when persistence is necessary if everything breaks down.

 

16. Parallel processing –

Use parallel processing if process logic allows it but beware from over-complex.
Keep in-mind that code should be thread-safe.

Using Couchbase NoSQL Database as a caching layer for our BizTalk Processes - Part 2

Previous Parts:

Part 1

Before I'll start, I would like to recommend on "Couchbase Essentials" book by John Lablocki.

That book goes through many important subjects of Couchbase Server, and also includes explanations about using it on .NET.
You can purchase the book on: http://bit.ly/1wtEbwl

"Failover" a node in Couchbase

Couchbase NoSQL concept is to have as many servers as possible in one Couchbase cluster.

Couchbase engine spread cluster data across the cluster nodes.

If one node (= server) goes down, part of the data would be unavailable until:

1. Failed node is up again
2. "Failover" is performed on that failed node.

Faling over a node would eliminate it from cluster, and other nodes would serve its data in memory (by promote replica documents of the failed node to "active" status. Replica documents are placed on disk of each one of the cluster nodes).
Replica documents gets updated when data changes, and used as a backup for in-memory data of each node.

Thumb rule: You can only "Failover" as many nodes as you have replicas.
For instance: If a cluster has 10 servers, and 1 server is down - 10% of the data would be unavailable until failing over that node (manually or automatically) or failed node is up again.

You can read more about Couchbase architechture and replication over here:
http://www.couchbase.com/couchbase-server/architecture

http://www.couchbase.com/docs/couchbase-manual-1.8/couchbase-introduction-architecture-replication.html

Scenario Architecture

Our BizTalk environment contains only 2 servers.
The decision was to keep them in separate clusters for two reasons:
1. If one of those nodes is down, 50% of the cache is unavailable. That is too much.
2. Automatic "Failover" takes 30 seconds (which is too much for us), and can only be configured on clusters which contains 3 nodes or more.

Each of those servers contains its own Couchbase Cluster (= cache), and each cluster backs up the other.

For instance:

If Server A (= cluster A) goes down, Server B is independent and all requests coming from Server B process infrastructure would go to Server B cache.
More than that, if Server A Couchbase windows service is down for some reason (= NoSQL database is down), all requests from process infrastructure on Server A would be routed to Server B (= cluster B). After a configured period of time, requests should hand over again to Server A.

In order to implement backup functionality for clusters, it was needed to write a cache access layer which is built on top of Couchbase Client Library.
In order to keep the cache data identical on both clusters, i've configured an uni-directional XDCR (Cross Datacenter Replication) from Server A to Server B (That means that only Server A loads data from SQL).

To Summarize:

Implement Cache Access with Cluster Backup functionality

First, I've download Couchbase .NET Client Library from http://www.couchbase.com/communities/net/getting-started

I've extracted 3 Dll's from the downloaded zip:
1. Couchbase.dll - Couchbase client dll in order to access Couchbase NoSQL database
2. Enyim.Caching.dll - Memcached dll. Couchbase NoSQL is built on top of it.
3. Newtonsoft.Json.dll - Json.NET is a popular high-performance JSON framework for .NET. I use it to convert JSON documents (from cache) to .NET Dictionary object.

I've added those 3 Dll's to my project, and wrote "CouchbaseClientManager" class:

public class CouchbaseClientManager

    {

        private const string DEAFULT_CLIENT = "couchbase";

        private const string BACKUP_CLIENT = "backup";

        private const string TEST_KEY = "test";

        private Timer timer { get; set; }  // Timestamp for backup to replce the default  

        private string currentClient; // Current client type to create instance

       

        public CouchbaseClientManager()

        {

            this.currentClient = DEAFULT_CLIENT;

            this.timer = new System.Timers.Timer(TimeSpan.Parse(ConfigurationManager.AppSettings["backupTime"]).TotalMilliseconds);

            this.timer.AutoReset = true;           

        }

        public T Get<T>(string key)

        {

            T row;

            if (this.currentClient == DEAFULT_CLIENT)

            {

                if (this.CheckConnectivity(this.currentClient, TEST_KEY))

                {

                    row = this.Get<T>(this.currentClient, key);

                }

                else

                {

                    this.SetCurrentClient(BACKUP_CLIENT); // Setting backup client as the current client

                    this.timer.Elapsed += (sender, arguments) => this.SetCurrentClient(DEAFULT_CLIENT); // When the timer is up, set the current client to default client

                    this.timer.Start();

                    row = this.Get<T>(BACKUP_CLIENT, key);

                }

            }

            else // if currentClient set to BACKUP_CLIENT

            {

                row = this.Get<T>(BACKUP_CLIENT, key);

            }

            return row;

        }

        private T Get<T>(string clientType, string key)

        {

            T row;

            using (var client = new CouchbaseClient(clientType))

            {

                row = client.Get<T>(key);

            }

            return row;

        }

        private void SetCurrentClient(string clientType)

        {

            this.currentClient = clientType;

        }

        private bool CheckConnectivity(string clientType,string key)

        {

            if (System.String.IsNullOrEmpty(this.Get<string>(clientType,key)))

                return false;

            return true;

        }

    }

Few things about the code above:

1. Code reads configurations from config file (Default and Backup cluster configurations and backup time period).

2. Getting a cache row in Json is done by: client.Get<T>(key);

3. Connectivity Check is done by using a "test" key I've places in each cluster.

Here is a flow chart of the code:

Let's take a look on Operations class, which provides a layer for getting data from cache.

public class Operations

    {

        private static readonly Operations instance = new Operations();

        private string urlCacheConfig;

        private List<SyncEntity> syncEntities;

        private CacheConfig.Operations cacheConfigOperations;

        private CouchbaseClientManager couchbaseClientManager;

        static Operations()

        {

        }

        private Operations()

        {

            this.urlCacheConfig = ConfigurationManager.AppSettings["cacheConfigFile"];     

            cacheConfigOperations = new CacheConfig.Operations(this.urlCacheConfig);

            LoadCacheEntities();

            cacheConfigOperations.fileConfigChanged += new System.IO.FileSystemEventHandler(fileConfigChanged);

            this.couchbaseClientManager = new CouchbaseClientManager();

        }

        public static Operations Instance

        {

            get

            {

                return instance;

            }

        }

        private void LoadCacheEntities()

        {            

            syncEntities = cacheConfigOperations.GetCacheItems();           

        }

        private void fileConfigChanged(object sender, System.IO.FileSystemEventArgs e)

        {

            LoadCacheEntities();

        }

        public IDictionary GetDictionary(string table, string key)

        {

            SyncEntity tableSyncEntity = syncEntities.Where(se => se.tableName.Equals(table)).FirstOrDefault();

            if (tableSyncEntity == null)

                return null;

            string column = tableSyncEntity.keys[0];

            string fullKey = table + "_" + column + "_" + key;

            Dictionary<string, string> dicRow;

            string row = this.couchbaseClientManager.Get<string>(fullKey);

            System.Web.Script.Serialization.JavaScriptSerializer js = new System.Web.Script.Serialization.JavaScriptSerializer();

            dicRow = js.Deserialize<Dictionary<string, string>>(row);

            return dicRow;

        }

    }

Operations class keeps entities (List<SyncEntity>) from config in memory.
If config file is changed, Operations loads it again. BizTalk process infrastructure would use "GetDictionary" method in order to get a row from cache.
Each table has a constant key column, and we needed to get data by that column value.

For instance:

If we have defined "Name" as key column, then running GetDictionary("Products","Phone"); would return a key-value dictionary which contains:

table - Cached table name.

key - Value of column.

Client configuration file look like this one, with minor changes (like server and database names etc.):

<?xml version="1.0"?>

<configuration>

  <configSections>   

    <section name="couchbase" type="Couchbase.Configuration.CouchbaseClientSection, Couchbase, Version=1.2.6.0, Culture=neutral, PublicKeyToken=12b9c6b5a9ec94c3"/>

    <section name="backup" type="Couchbase.Configuration.CouchbaseClientSection, Couchbase, Version=1.2.6.0, Culture=neutral, PublicKeyToken=12b9c6b5a9ec94c3"/>

  </configSections>

  <connectionStrings>

    <add name="connectionString" connectionString="Persist Security Info=False;Integrated Security=True;Initial Catalog=DBName;server=SQLServerAddress" />

  </connectionStrings>

  <appSettings>

    <add key="cacheConfigFile" value="C:\CouchbaseCache\BTSCachingTasksConfiguration.xml"/>

    <add key="backupTime" value="00:01:00"/>

  </appSettings>

  <couchbase>

    <servers bucket="default" bucketPassword="private">

      <add uri="http://currentServer:8091/pools/default"/>

    </servers>

  </couchbase>

  <backup>

    <servers bucket="default" bucketPassword="private">

      <add uri="http://otherServer:8091/pools/default"/>

    </servers>

  </backup>

  <startup>

    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/>

  </startup>

</configuration>

Here are some remarks regarding config file:

1. 

BTSCachingTasksConfiguration.xml - Contains table names and columns keys to cache.

For instance:

<?xml version="1.0" encoding="utf-8"?>

<ArrayOfCachingTask>

  <entity>

   <connectionKey>NotRelevant</connectionKey>  

   <tableName>Products</tableName>

   <keys>Name</keys>

   <timer>12:00:00</timer> 

  </entity>

</ArrayOfCachingTask>

As you can see, cached table name is "Products", and column key is: "Name".

"timer" is an interval value to load that table again (in order to refresh it on cache).
That load isn't done by that client library. It's done by a windows service I wrote that runs on background.

2. Connection String to SQL database is used when BTSCachingTasksConfiguration.xml is changed, and data gets load again from SQL to Couchbase NoSQL (cache). The load process is done by CacheConfig.Operations class.

3. BizTalk is working only with signed dll's placed on GAC, therefore I put all relevant assemblies there.

What's next?

Part 3 would speak about a windows service I wrote to initialize cache, and refresh it on a configured time interval. I would also go into detail of Coucbase Console Administration and Map-Reduce functions.