I recently came across a situation that required me to be able to iterate over a JSON message payload similar to what can be done with JDOM in regards to XML similar to what I do within my Stax XML Mapreduce InputFormat. So basically in this case you need to treat JSONArray’s similar to XML…
Read more
XmlStaxInputFormat / XmlStaxFileRecordReader Github Project – https://github.com/gss2002/xml-stax-mr After some time it seemed like a gap that existed with Hadoop MapReduce and Spark that the existing XmlInputFormat classes from Mahout were using fseek and searching for strings as the file is read in from HDFS. The ability to break up a large Xml file becomes extremely important…
Read more
I’ve been working on securing Apache SolrCloud with kerberos. This includes configuring Zookeeper. So after struggling and lots of searching I came up with a working kerberized solution for SolrCloud, with Zookeeper, and Apache Ranger for Authorization. First I tried to secure a standalone Solr instance by updating to the Solr 6x branch which is a SNAPSHOT…
Read more
After working many years with IBM WebSphere Application Server on Solaris, Linux on PSeries, XSeries and ZSeries and Z/OS. I came to realize the IBM version of Java has much better debug tools and documentation available to perform debugging and performance tuning. Examples of these features are the IBM AOT Ahead of Time Compiler which…
Read more
After successfully running a Large Hadoop Cluster for a period of time. I started to notice strange things occurring initially with the MapReduce PI example task where tasks would be marked as failed. When looking more closely and attempting to logon/su/ssh to a machine with the userid that was running the job the sshd/su would return: -bash:…
Read more
Integrating Apache Hadoop and Flume with IBM MQ Over the past 2 years of working with Apache Hadoop a few things have come up folks wanting to use Apache Kafka which definitely has it’s place in the Hadoop Big Data and Next Generation of Technology spheres. But there is also the need to integrate with…
Read more