Case scenario at work:
We needed a caching system in order to save some key-value information for our BizTalk processes, which are SOA-based. (BizTalk processes are getting configuration during Run Time).
Cache should:
1. Hold a SQL table which contains that information, and use polling for refresh.
1. Hold a SQL table which contains that information, and use polling for refresh.
2. We need to control when and which table gets refreshed.
The previous solution of the cache problem was an old, "home-made" windows service component.
Reload without any cache downtime
Our cache needs to be accessible during the refresh process. That means we can't remove/flush all keys and reload them because it will cause a cache downtime.
Instead, here are the steps to perform a reload:
1. Removed keys - Iterate each key on the cache and a check whether it still exists in that SQL table. If it doesn't - remove it from cache.
2. Update old keys & Insert new keys - Iterate each key on the SQL table and update it on the cache layer.
How Couchbase Server is a good fit for a cache solution ?
What is Couchbase Server?
Couchbase NoSQL is both a key-value store and a document database (documents are saved in JSON format).
NoSQL solutions are suitable for high-performance, and it's generally faster than relational database.
Couchbase also have client providers for many platforms: .NET, JAVA, C etc.
If you want to read more about Couchbase features:
Back to the cache subject, if you are using multiple servers to serve data, Couchbase replicates it across those nodes. If one node goes down, other node contains the requested data in disk - and will load it to memory. Each Couchbase client holds a cluster map, so each knows which node currently holds the requested data in memory, and therefore knows where to send the request.
As written in Couchbase site:
"Read and write operations first go to the in-memory object-managed cache – if a document being read is not in the cache, it is fetched from disk. Updates to documents are first made in the in-memory cache and later eventually persisted to disk."
In our scenario, we just need to write a windows service which read keys & values from SQL and loads them to Couchbase (which is the cache itself).
Both BizTalk servers would run on two different couchbase cluster, and data gets replicated from one to another (uni-directional replication).
In our scenario, we just need to write a windows service which read keys & values from SQL and loads them to Couchbase (which is the cache itself).
Both BizTalk servers would run on two different couchbase cluster, and data gets replicated from one to another (uni-directional replication).
Why not just use memcached ?
Memcached is a distributed memory object caching system, which appeared to be the perfect solution for our cache. Looking at our reload process - iterating each key in memcached is not possible.
The good news is that Couchbase database is built over memcached.
It has the memcached performance with more important features that Couchbase database provides.
With Couchbase you also get a nice administration site for monitoring and configuring the server. This site also contains our loaded documents in cache and information regard them.
You can read more about the advantages over memcached cache tier here:
http://www.couchbase.com/memcached
http://www.couchbase.com/memcached
Installing Couchbase Server
1. Download Couchbase Server for Windows - http://www.couchbase.com/download and install it:
2. Click on the "Couchbase Console" icon on your desktop and you'll get:
3. Start new cluster by:
4. If you wish, you can have some samples to get started:
5. In that step, choose Couchbase bucket over Memcached bucket.
That would enable you to use map/reduce functions (e.g in order to receive all keys), and will also allow you to view documents.
Persistence and Replication is also enabled in Couchbase bucket.
You can also choose the number of replicas (for each document) on disk.
The replication process takes place between servers in each cluster, and also between servers across cluster, as shown here:
Also tick "Flush" option, which allows to delete all data from that bucket.
6.
7. Configure user name and password for your server:
8. That's it ! Your Couchbase server is up & running. You should have been navigated to the administration console:
Download Couchbase client
Currently we are using .NET platform, so we downloaded .NET client from:
http://www.couchbase.com/communities/all-client-libraries
And followed this wonderful tutorial on how to use it in .NET applications:
And followed this wonderful tutorial on how to use it in .NET applications:
What's next?
In part 2, I will get more specific about our caching tier.
