Integrating Apache Hadoop and Apache Flume with IBM MQ

By | May 30, 2015

Over the past 2 years of working with Apache Hadoop a few things have come up folks wanting to use Apache Kafka which definitely has it’s place in the Hadoop Big Data and Next Generation of Technology spheres. But there is also the need to integrate what a lot of folks would consider Legacy Messaging (IBM MQ) with Apache Hadoop and Apache Flume. I personally don’t feel IBM MQ is a legacy technology it actually performs much better and integrates with almost every platform out there especially when using bindings. So this brings me to what I decided was a good idea after working with IBM MQ and WebSphere Application Server for a very long time 15 years.  I saw the need to integrate Apache Flume and IBM MQ specifically in regards to IBM MQ and Local Queue Managers that can utilize bindings to connect which will provide extremely high performant messaging with Legacy Technologies and Hadoop.  Here is documentation from IBM stating Bindings performs better than Clients.

The project is hosted over at github. What this code enables is the ability to define a source or sink using IBM MQ Local QMGR and Queues as an endpoint.  Btw I provide no guarantees and no liability you are welcome to ask for bug fixes via github issues.

 

So how to use this most likely you will have a license to utilize IBM MQ from IBM or you can download the trial version and the MQ Advanced Developer version which are free for developers.

After installing IBM MQ you must configure your local QMGR to allow the “flume” group or whatever group or user-id you are using to run the local flume process on the same server as your QMGR.

 

How Configure IBM MQ to work with Flume in Bindings Mode:

su – mqm
# Source Local QMGR Environment Variables
. /opt/ibm/mqm/bin/setmqenv
# Create a Local QMGR if it doesn’t already exist
crtmqm GSSMQP1
# Start the Local QMGR if it’s not already started
strmqm GSSMQP1
# Create Local Queues if none are defined for testing
runmqsc GSSMQP1
define qlocal(GSS.REQUEST.REPLY.QUEUE)
    end

#ACL the Local QMGR for Flume and the Queue(s) being used by Flume
setmqaut -m GSSMQP1 -t qmgr -g flume +connect +inq +dsp
setmqaut -m GSSMQP1 -t queue -n GSS.REQUEST.REPLY.QUEUE -g flume +connect +inq +browse +get +put +dsp

# Refresh Security on the Local QMGR
runmqsc GSSMQP1
    refresh security
end

 

#Configure Flume as described in the github readme

 

 

 

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *