Category: Linux

Technology Blog

RHEL 7.x and SSSD and /etc/resolv.conf

Ran into an interesting situation with /etc/resolv.conf with “options rotate timeout:1” set with SSSD in regards to DNS lookups and nameservers not being up and SSSD marking an entire domain down. With this specific situation the last server in /etc/resolv.conf had been left down by accident following a reconfiguration of VMWare. When the servers were…
Read more

RHEL 7 and NFSv4 with Kerberos

Over the past year I have been tasked with building out a large Secure NFSv4 Environment using DRBD, Corosync and Pacemaker and ran into a plethora of issues which included gotcha’s with setting up NFSv4 Server and Client Security settings related to gssproxy/rpc-gssd, how to enforce quotas remotely with rpc-rquotad, to setting up idmapd or…
Read more

Removing DRBD Devices and Volumes from Highly Available NFS

In working through attempting to add a new volume without causing an outage with a highly available NFS setup I had to come up with a methodology to remove the DRBD device to effectively rinse and repeat until we came up with the right steps. Below outlines the steps to remove a DRBD block device/volume.…
Read more

Adding a block device to DRBD with Corosync and Pacemaker for use with Highly Available NFS

Over the past few weeks I have been working with DRBD, Corosync and Pacemaker and adding addition block devices to DRBD to create new volumes and noticed that there was no solid methodology or steps to complete this task but after some trial and error I came up with steps to avoid taking an outage…
Read more

Integrating Apache Nifi with IBM MQ

Integrating Apache Nifi with IBM MQ This would be a continuation of the IBM MQ and Hadoop integration article I first posted a few years ago. This explains how to integrate IBM MQ with Apache Nifi or Hortonworks HDF. IBM MQ is extremely important when attempting to integrate new technologies with legacy environments specifically mainframe environments…
Read more

Apache SolrCloud Kerberos Configuration

I’ve been working on securing Apache SolrCloud with kerberos. This includes configuring Zookeeper. So after struggling and lots of searching I came up with a working kerberized solution for SolrCloud, with Zookeeper, and Apache Ranger for Authorization. First I tried to secure a standalone Solr instance by updating to the Solr 6x branch which is a SNAPSHOT…
Read more

Benefits of using IBM Java and JDK features

After working many years with IBM WebSphere Application Server on Solaris, Linux on PSeries, XSeries and ZSeries and Z/OS. I came to realize the IBM version of Java has much better debug tools and documentation available to perform debugging and performance tuning. Examples of these features are the IBM AOT Ahead of Time Compiler which…
Read more

Hadoop, Java and HTTPD and /etc/security/limits.d/ nproc/pid-max

After successfully running a Large Hadoop Cluster for a period of time. I started to notice strange things occurring initially with the MapReduce PI example task where tasks would be marked as failed. When looking more closely and attempting to logon/su/ssh to a machine with the userid that was running the job the sshd/su would return: -bash:…
Read more

Hadoop and ip_conntrack: table full, dropping packet

I’m pretty sure many folks have seen this specific error across multiple different linux systems specifically when iptables is enabled and the OS has thousands of connections coming in second. In my case I ran into this Examples of this are with Hadoop NameNode. Someone accidentally executed iptables -L to try to get a list…
Read more

Hadoop and Redhat System Tuning /etc/sysctl.conf

Hadoop and Redhat System Tuning /etc/sysctl.conf One of the most overlooked things after building out a Hadoop cluster is the operating system tuning. This post will cover how to tune settings in /etc/sysctl.conf also known as Linux Kernel Settings. /etc/sysctl.conf ## ALWAYS INCREASE KERNEL SEMAPHORES especially IF using IBM JDK with SharedClassCache also a separate…
Read more