Deploying and Configuring Kubernetes (K8S) on Centos 8

This document will explain how to setup Kubernetes aka K8S on Centos 8 with the following components providing network capabilities: Tigera Calico for the Network Stack, MetalLB for the Load Balancer and Nginx Ingress Controller for Inbound Traffic (HTTP, HTTPS). Just a note these nodes need at least 2 CPUs and at least 4GB of Memory to setup. 

• Setup Hosts and their DNS Entries both A Records and PTR Records.
  • cplane.k8s.senia.org – 10.69.68.30 (Control Plane Node)
  • worker3.k8s.senia.org – 10.69.68.31 (Worker Node)
  • worker4.k8s.senia.org – 10.69.68.31 (Worker Node)
• Install and Configure Centos 8 on Each Host
  • Utilize a Centos 8 Repo: http://mirror.net.cen.ct.gov/centos/8/BaseOS/x86_64/os/
  • Perform “Minimal Install” and “Headless Management”
• After Centos 8 is installed the following Repos, Packages and settings are needed
  • Exclude Redhat/Centos “runc” rpm from Centos8-AppStream repo
    • echo “exclude=runc” >> /etc/yum.repos.d/CentOS-Linux-AppStream.repo
  • Remove Redhat/Centos “runc”
    • yum erase runc -y
    • Setup Kubernetes Repo
      cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
      [kubernetes]
      name=Kubernetes
      baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
      enabled=1
      gpgcheck=1
      repo_gpgcheck=1
      gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
      exclude=kubelet kubeadm kubectl
      EOF
  • Install Docker
    • yum install -y yum-utils
    • yum-config-manager –add-repo https://download.docker.com/linux/centos/docker-ce.repo
    • yum install docker-ce -y
  • Disable SELINUX
    • setenforce 0
    • sed -i ‘s/^SELINUX=enforcing$/SELINUX=permissive/’ /etc/selinux/config
  • Install Kubernetes
    • yum install -y kubelet kubeadm kubectl –disableexcludes=kubernetes
    • yum install iproute-tc -y
  • Install “git” on Control Plane Nodes only:
    • yum install git -y
  • Update Docker to use “systemd” vs “cgroupfs” for cgroup driver
    • vi /usr/lib/systemd/system/docker.service
      -ExecStart=/usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock
      +ExecStart=/usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock –exec-opt native.cgroupdriver=systemd
  • Disable swap and remove it via /etc/fstab on each node
    • swapoff -a
    • vi /etc/fstab
      • #/dev/mapper/cl_cplane-swap none swap defaults 0 0
  • Disable firewalld (Would not disable in production for testing purposes only)
    • systemctl stop firewalld ; systemctl disable firewalld
  • Enable br_netfilter kernel module
    • modprobe br_netfilter
    • cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
      br_netfilter
      EOF
    • cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
      net.bridge.bridge-nf-call-ip6tables = 1
      net.bridge.bridge-nf-call-iptables = 1
      EOF
    • sudo sysctl –system
  • On Each Host Start Docker and Kublet
    • systemctl start docker
    • systemctl start kubelet
  • Initialize the Control Plane Node with a CIDR Block
    • Note: Internal POD Communication this subnet is NOT used outside of inter-pod-communication
    • kubeadm init –pod-network-cidr=192.168.0.0/16
  • Save the join command into a safe place to use for joining nodes
    • kubeadm join cplane.k8s.senia.org:6443 –token wgxxxxxx.sh2xxxxxvwwk \
      –discovery-token-ca-cert-hash sha256:d099xxxxxxxx0e424a793f205663556d92a29af707e1f9ccf8459804f77ec 
  • Create a copy of the K8S Admin.conf into Home Directory for utilizing “kubectl”
    • mkdir -p $HOME/.kube
    • cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    • chown $(id -u):$(id -g) $HOME/.kube/config
  • Verify if “kubectl” works
    • kubectl get nodes
  • Setup the Calico Kubernetes Network Stack
    • What is Calico: https://projectcalico.docs.tigera.io/about/about-calico
    • kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
    • kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml
  • Execute the join commands on the Worker Nodes
    • kubeadm join cplane.k8s.senia.org:6443 –token wggbwy.shxxxxxx
      –discovery-token-ca-cert-hash sha256:d0995xxxxxx29af707e1f9ccf8459804f77ec 
  • Install and Configure MetalLB
    
    • What is MetalLB – https://metallb.universe.tf
    • Find and Verify if strictARP = true – If not set Change it from false to true
      • kubectl edit configmap -n kube-system kube-proxy
    • Create MetalLB Namespace using “kubectl apply”
      • kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/namespace.yaml
    • Install MetalLB using “kubectl apply”
      • kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/metallb.yaml
    • Create MetalLB ConfigMap with External IP Addresses to be used for Load Balancer External IPs
      • vi metallb.configmap.yaml
        

        apiVersion: v1 kind: ConfigMap metadata:    namespace: metallb-system    name: config data:    config: |         address-pools:         – name: default            protocol: layer2            addresses:            – 10.69.68.70-10.69.68.79

      • Apply the metallb.configmap.yaml
        • kubectl apply -f metallb.configmap.yaml
      • Install Krew for two useful plugins
        
        • https://krew.sigs.k8s.io/docs/user-guide/setup/install/
      • Add to .bash_profile
        
        • export PATH=”${KREW_ROOT:-$HOME/.krew}/bin:$PATH”
      • Install Ingress Nginx Plugin
        • kubectl install ingress-nginx
      • Install calicoctl plugin
        • https://projectcalico.docs.tigera.io/maintenance/clis/calicoctl/install#install-calicoctl-as-a-kubectl-plugin-on-a-single-host
        • cd /root/.krew/bin
        • curl -L https://github.com/projectcalico/calico/releases/download/v3.21.4/calicoctl-linux-amd64 -o kubectl-calico
        • chmod 755 $HOME/.krew/bin/kubectl-calico
      • Verify calicoctl plugin works
        • kubectl calico node status
          

          Calico process is running. IPv4 BGP status +————–+——————-+——-+———-+————-+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +————–+——————-+——-+———-+————-+ | 10.69.68.32 | node-to-node mesh | up | 04:15:00 | Established | | 10.69.68.31 | node-to-node mesh | up | 04:16:27 | Established | +————–+——————-+——-+———-+————-+ IPv6 BGP status No IPv6 peers found.

        • Install Nginx Ingress Controller
          • kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml
          • kubectl ingress-nginx logs -n ingress-nginx