Kubernetes 1.11 API certificate expired
Kubernetes
What is Kubernetes? Kubernetes is an open-source orchestration software for deploying, managing, and scaling containerized workloads on a cluster of servers. Those servers can be Baremetal or virtualized servers.
Certificate has expired or is not yet valid
We’ve had a cluster running for the last year using Kubernetes v1.11.x, and recently received an error that the certificate had expired. I looked online and found a number of different ways to resolve the problem, but right now I don’t want to upgrade the cluster. I only want to fix the certificate and get everything back into working order then schedule the upgrade later. I’ll do a post with our sets when upgrading from v1.11 to v1.16 which is the current release.
This is the error message we’re getting:
[root@node1 kubernetes]# kubectl get nodes Unable to connect to the server: x509: certificate has expired or is not yet valid
We can test and verify that the certificate has expired a few days ago:
[root@node1 ~]# echo -n | openssl s_client -connect localhost:6443 2>&1 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | openssl x509 -text -noout | grep Not Not Before: Sep 17 03:50:24 2018 GMT Not After : Sep 17 03:50:25 2019 GMT
Generate certificates for Kubernetes
Now that we confirmed that’s really our problem, the certificate has expired. We need to generate a new cert. First step is going to backup and remove the old certificates.
[root@node1 pki]# cd /etc/kubernetes/pki/
[root@node1 pki]# rm {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt}
[root@node1 pki]# kubeadm alpha phase certs all
[certificates] Using the existing ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [node1.example.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.230.107.137]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Using the existing sa key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [node1.example.com localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [node1.example.com localhost] and IPs [10.230.107.137 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[root@node1 pki]#
Now that we’ve got new certificates, next we need to remove and regenerate the config files.
[root@node1 kubernetes]# cd /etc/kubernetes [root@node1 kubernetes]# rm admin.conf controller-manager.conf kubelet.conf scheduler.conf [root@node1 kubernetes]# kubeadm alpha phase kubeconfig all [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
Now that generating certificates and configurations are successful lets restart Kubelet and Docker so we can get everything back online.
[root@node1 .kube]# systemctl status kubelet [root@node1 .kube]# systemctl restart kubelet [root@node1 .kube]# systemctl restart docker
Next we can do a quick test and get running pods in the default namespace:
[root@node1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE clusterfsvolume-5f95b59f8c-2cs29 1/1 Running 0 96d fsbackup-66757dc6b6-488sj 1/1 Running 0 96d
Now everything is back to normal and developers are able to manage pods as well as view pod logs now.