Quick install of a Kubernetes self hosted cluster
Updated October 27, 2018: Added disable chronyd before enabling ntpd
This how to explains the steps I went through to install Kubernetes on a bare metal self hosted cluster in our data center.
All your servers will have the same setup, with the exception of the first server you’ll run and init and then have the others join the cluster.
Install Operating System
I installed our servers with CentOS 7.x, this is our standardized operating system we use for all our managed customers.
After the operating system is installed I recommend making some standardized settings afterwards. These are things I’ve found make the most sense for our configuration.
Configure OS
If you’re going to run Ceph, something we use for our storage in the cluster, you need to make sure all nodes have their time in sync. I found that you first need to disable chronyd before you enable ntpd.
yum -y install ntpd systemctl stop chronyd systemctl disable chronyd systemctl start ntpd systemctl enable ntpd
Next, if you make any DNS changes, and reboot the Network Manager will revert the changes, so this may not be necessary for you, but it does help in our situation.
systemctl disable NetworkManager.service systemctl stop NetworkManager.service
After that we need to make sure the firewall is turned off since nodes and services will be using different ports, as well as our cluster is behind a pfSense firewall, so we’re good to open all ports on the severs.
systemctl disable firewalld systemctl stop firewalld
Now that we’ve got basic operating system settings done, we go through the Kubernetes install and setup process.
Kubernetes Installation on master
These are the required package to get kubeadm and kubectl installed to run Kubernetes
First install docker
yum install -y docker systemctl enable docker && systemctl start docker
Next we’ll add the Kubernetes repo
cat </etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF
Then we apply some kernel changes
setenforce 0 cat </etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sysctl --system
After that we need to turn off and disable swap on the server.
swapoff /dev/sda2
Next edit fstab and commend out the swap partition
vi /etc/fstab
Upgrade the operating system.
yum -y upgrade
Install kubeadm and kubectl
yum -y install kubelet kubeadm kubectl systemctl enable kubelet && systemctl start kubelet
Once all those are successful you’re ready to create your first cluster.
kubeadm init --pod-network-cidr=192.168.0.0/16
I’m going to install Calico for my node networking, so we needed to include the extra pod-network-cidr argument at the time we init the cluster.
Secondly, it’ll give you the full command to have other nodes join into the cluster, make a note of your join command. Here’s my example:
kubeadm join 10.230.107.137:6443 --token TOKEN --discovery-token-ca-cert-hash sha256:SHA_HASH
Next I installed Calico
kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubeadm/1.7/calico.yaml
That should complete the install on your master node.
Setting up nodes
Now that you’re master node is finished you’ll want to go through all the same steps for setting up each of your nodes. Just remember stop after the kubectl and kubeadm installation and enable.
You’ll run the command from above which is for each of your nodes to join into the cluster.
kubeadm join 10.230.107.137:6443 --token TOKEN --discovery-token-ca-cert-hash sha256:SHA_HASH
One thing to remember that when I first did this, I still forgot. The token is only good for 24 hrs. So if you need to add more nodes tomorrow or the next day, you’ll have to generate a new token and replace it in the TOKEN from above
kubeadm token create
That should get your master and nodes setup.
Next steps
Configure desktop to connect to cluster.
Install Ingress Nginx controller for incoming traffic.
Install Rook + Ceph for our pod storage.
Install Prometheus for monitoring the cluster. (Coming Soon)