Redis as Messaging Middleware

I needed a simple middleware for fluxua, my Hadoop workflow engine project on github. In fluxua, you define a set of Hadoop jobs running as an workflow. The dependency between the jobs is represented as a directed acyclic graph (DAG). The workflow engine will execute the Hadoop jobs in proper order.

Since the Hadopp jobs take significant amount of time for execution, I needed a simple async messaging middleware for the client application to communicate with the workflow engine. I evaluated RabbitMQ, which seemed like an overkill for my problem. I finally settled on Redis, using it’s list data structure as the messaging queue.

Note on Redis

Redis is not just another key, value store. What is interesting about Redis is that it imposes structure on the value.The value could be a List, Map or a Set.

If you insert at one end of a list and remove from the other end, it becomes a queue and Redis morphs itself to a messaging service. Here are some of the notable characteristics of Redis

  • It operates primarily in memory and is blazing  fast. It periodically checkpoints data on the disk, but there is no guarantee against data loss.
  • Redis is more appropriate where performance and flexibility is more important than persistence and data integrity. Persistence can even be disabled.
  • Keys are strings. Values can be String, List, Set and Map. Redis is also known as a data structure server.
  • The elements of List, Set and Map could be any string, including JSON serialized objects.
  • Check pointing interval is configurable. Please bear in mind that data loss is possible in Redis.
  • Multiple operations can be performed atomically with transactional boundary. Redis also allows optimistic concurrency control.
  • Redis has master slave replication, which can be used for fail over and load balancing read queries.
  • There is no support for sharding. But it should be easy to build a sharding layer by hashing the key and distributing keys across multiple instances of Redis

Hadoop Workflow Service

The workflow engine API  is divided into two sets. The job control API is for executing Hadoop workflow and the admin API is for starting and stopping the workflow engine. The different message exchange protocols are as follows. Messages are transmitted as objects serialized with JSON

Type Flow Description
job request client to server client sends job request
job request ack server to client server sends message acknowledging job start
job status query client to server client quries for job status
job status return server to client server returns job status on successful completion or failure
server shutdown request admin to server admin sends shutdown request to server

As mentioned before, I am using Redis List as queue and and the key corresponding to the List object is the queue name. There is one queue to receive job requests and another to receive admin requests. After a shut down request is received, the server checks if there are workflows in in progress and if so waits until they are all completed and then quits. During that time it does accept any other request.

Messaging Service

Here is the messaging service implementation in Redis that gets used by the workflow service. it uses the jedis, which is a java client for Redis. The API consists of reading   job request and admin request queues and writing to corresponding response queues. There are also two methods for failed jobs to save to and retrieve from Redis Map.

public String receiveJobRequest() {
  String requestSt = jedis.rpop(requestQueue);
  return requestSt;

public void replyJobRequest(String repChannel, String response) {
  jedis.lpush(repChannel, response);

public String receiveAdminRequest() {
  String requestSt = jedis.rpop(adminQueueIn);
  return requestSt;

public void replyAdminRequest( String response) {
   jedis.lpush(adminQueueOut, response);
public void saveFailedJob (String requestID, String requestSt) {
  jedis.hset(failedJobsStore,  requestID, requestSt);

public String  getFailedJob (String requestID) {
  return jedis.hget(failedJobsStore, requestID);

public void close() {

Redis Client

I am using Redis client in java called Jedi for the workflow server side. There are also Redis client available for ruby, PHP and other languages. The list of clients can be found here. To build a web front end for fluxua, ruby or  PHP clients could be used.

Wrapping up

Redis has been called the swiss army knife of key value store. It truly lives up to that reputation. It’s small and fast. It’s been used a distributed cache also and many consider to be good replacement of Memcache.

For commercial support for any solution in my github repositories, please talk to ThirdEye Data Science Services. Support is available for Hadoop or Spark deployment on cloud including installation, configuration and testing,


About Pranab

I am Pranab Ghosh, a software professional in the San Francisco Bay area. I manipulate bits and bytes for the good of living beings and the planet. I have worked with myriad of technologies and platforms in various business domains for early stage startups, large corporations and anything in between. I am an active blogger and open source project owner. I am passionate about technology and green and sustainable living. My technical interest areas are Big Data, Distributed Processing, NOSQL databases, Machine Learning and Programming languages. I am fascinated by problems that don't have neat closed form solution.
This entry was posted in Key Value Store, Messaging, Redis and tagged , . Bookmark the permalink.

3 Responses to Redis as Messaging Middleware

  1. Pingback: Redis as Messaging Middleware |

  2. Pingback: Real Time Fraud Detection with Sequence Mining | Mawazo

  3. Pingback: Real Time Fraud Detection with Sequence Mining | Big Data Cloud

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s