Kubernetes 1.11 API certificate expired

Kubernetes

What is Kubernetes?  Kubernetes is an open-source orchestration software for deploying, managing, and scaling containerized workloads on a cluster of servers.  Those servers can be Baremetal or virtualized servers.  

Certificate has expired or is not yet valid

We’ve had a cluster running for the last year using Kubernetes v1.11.x, and recently received an error that the certificate had expired. I looked online and found a number of different ways to resolve the problem, but right now I don’t want to upgrade the cluster. I only want to fix the certificate and get everything back into working order then schedule the upgrade later. I’ll do a post with our sets when upgrading from v1.11 to v1.16 which is the current release.

This is the error message we’re getting:

[root@node1 kubernetes]# kubectl get nodes
Unable to connect to the server: x509: certificate has expired or is not yet valid

We can test and verify that the certificate has expired a few days ago:

[root@node1 ~]# echo -n | openssl s_client -connect localhost:6443 2>&1 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | openssl x509 -text -noout | grep Not
Not Before: Sep 17 03:50:24 2018 GMT
Not After : Sep 17 03:50:25 2019 GMT

Generate certificates for Kubernetes

Now that we confirmed that’s really our problem, the certificate has expired. We need to generate a new cert. First step is going to backup and remove the old certificates.

[root@node1 pki]# cd /etc/kubernetes/pki/

[root@node1 pki]# rm {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt}

[root@node1 pki]# kubeadm alpha phase certs all
[certificates] Using the existing ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [node1.example.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.230.107.137]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Using the existing sa key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [node1.example.com localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [node1.example.com localhost] and IPs [10.230.107.137 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[root@node1 pki]#

Now that we’ve got new certificates, next we need to remove and regenerate the config files.

[root@node1 kubernetes]# cd /etc/kubernetes
[root@node1 kubernetes]# rm admin.conf controller-manager.conf kubelet.conf scheduler.conf
[root@node1 kubernetes]# kubeadm alpha phase kubeconfig all
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"

Now that generating certificates and configurations are successful lets restart Kubelet and Docker so we can get everything back online.

[root@node1 .kube]# systemctl status kubelet
[root@node1 .kube]# systemctl restart kubelet
[root@node1 .kube]# systemctl  restart docker

Next we can do a quick test and get running pods in the default namespace:

[root@node1 ~]# kubectl  get pods
NAME                               READY     STATUS    RESTARTS   AGE
clusterfsvolume-5f95b59f8c-2cs29   1/1       Running   0          96d
fsbackup-66757dc6b6-488sj          1/1       Running   0          96d

Now everything is back to normal and developers are able to manage pods as well as view pod logs now.