Quick install of a Kubernetes self hosted cluster

Updated October 27, 2018: Added disable chronyd before enabling ntpd

This how to explains the steps I went through to install Kubernetes on a bare metal self hosted cluster in our data center.

All your servers will have the same setup, with the exception of the first server you’ll run and init and then have the others join the cluster.

Install Operating System

I installed our servers with CentOS 7.x, this is our standardized operating system we use for all our managed customers.

After the operating system is installed I recommend making some standardized settings afterwards.  These are things I’ve found make the most sense for our configuration.

Configure OS

If you’re going to run Ceph, something we use for our storage in the cluster, you need to make sure all nodes have their time in sync. I found that you first need to disable chronyd before you enable ntpd.

yum -y install ntpd
systemctl stop chronyd
systemctl disable chronyd
systemctl start ntpd
systemctl enable ntpd

Next, if you make any DNS changes, and reboot the Network Manager will revert the changes, so this may not be necessary for you, but it does help in our situation.

systemctl disable NetworkManager.service
systemctl stop NetworkManager.service

After that we need to make sure the firewall is turned off since nodes and services will be using different ports, as well as our cluster is behind a pfSense firewall, so we’re good to open all ports on the severs.

systemctl disable firewalld
systemctl stop firewalld

Now that we’ve got basic operating system settings done, we go through the Kubernetes install and setup process.

Kubernetes Installation on master

These are the required package to get kubeadm and kubectl installed to run Kubernetes

First install docker

yum install -y docker
systemctl enable docker && systemctl start docker

Next we’ll add the Kubernetes repo

cat < /etc/yum.repos.d/kubernetes.repo
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

Then we apply some kernel changes

setenforce 0

cat <  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

sysctl --system

After that we need to turn off and disable swap on the server.

swapoff /dev/sda2

Next edit fstab and commend out the swap partition

vi /etc/fstab

Upgrade the operating system.

yum -y upgrade

Install kubeadm and kubectl

yum -y install kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet

Once all those are successful you’re ready to create your first cluster.

kubeadm init --pod-network-cidr=

I’m going to install Calico for my node networking, so we needed to include the extra pod-network-cidr argument at the time we init the cluster.

Secondly, it’ll give you the full command to have other nodes join into the cluster, make a note of your join command.  Here’s my example:

kubeadm join --token TOKEN --discovery-token-ca-cert-hash sha256:SHA_HASH

Next I installed Calico

kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubeadm/1.7/calico.yaml

That should complete the install on your master node.

Setting up nodes

Now that you’re master node is finished you’ll want to go through all the same steps for setting up each of your nodes.  Just remember stop after the kubectl and kubeadm installation and enable.

You’ll run the command from above which is for each of your nodes to join into the cluster. 

kubeadm join --token TOKEN --discovery-token-ca-cert-hash sha256:SHA_HASH

One thing to remember that when I first did this, I still forgot.  The token is only good for 24 hrs.  So if you need to add more nodes tomorrow or the next day, you’ll have to generate a new token and replace it in the TOKEN from above

kubeadm token create

That should get your master and nodes setup.

Next steps

Configure desktop to connect to cluster.

Install Ingress Nginx controller for incoming traffic.

Install Rook + Ceph for our pod storage.

Install Prometheus for monitoring the cluster. (Coming Soon)