Prometheus kubelet metrics server returned HTTP status 403 Forbidden
Recently been working a lot with Kubernetes and needed to install some monitoring to better profile the cluster and it’s components. After doing that then doing an installation using the their deploy script. It all looked good, no errors, all the pods and services were running. I went ahead and launched the Prometheus web interface and saw there was a bunch of errors and it was not able to access Kubelet metrics.
Setup
I first cloned the project for prometheus-operator
git clone git@github.com:coreos/prometheus-operator.git
After that I then ran the deployment for cluster monitoring.
$ cd prometheus-operator/
$ contrib/kube-prometheus/hack/cluster-monitoring/deploy
After everything was installed, I created my own ingress. We’re running ingress-nginx, so I created a couple of quick ingress configuration files and imported them into the cluster so I could access remotely.
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: alertmanager-ingress namespace: monitoring labels: app: alertmanager-main spec: rules: - host: alert.example.com http: paths: - backend: serviceName: alertmanager-main servicePort: 9093 path: / tls: - hosts: - alert.example.com secretName: example-com-tls -- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: grafana-ingress namespace: monitoring labels: app: grafana spec: rules: - host: gra.example.com http: paths: - backend: serviceName: grafana servicePort: 3000 path: / tls: - hosts: - gra.example.com secretName: example-com-tls -- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: prometheus-ingress namespace: monitoring labels: app: prometheus-operated spec: rules: - host: prom.example.com http: paths: - backend: serviceName: prometheus-operated servicePort: 9090 path: / tls: - hosts: - prom.example.com secretName: example-com-tls
You can save that as prometheus-ingress.yaml, then import.
$ kubectl apply -f prometheus-ingress.yaml
Now that you’ve got your Prometheus running and ingress routing setup, we can do a test. Launch a web browser and go to
https://prom.example.com/targets
Once that opens, check and make sure if you have any errors. Everything should look good, except for kubelet section. This is where I had errors on all the metrics collections. The error I had
HTTP status 403 Forbidden
After doing some research I determined, that after you complete the pod installation there was another setup when using kubeadm.
Follow the instruction you’ll need to modify some configuration files on your master node.
Fist update the kubelet service to include webhook and restart.
KUBEADM_SYSTEMD_CONF=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf sed -e "/cadvisor-port=0/d" -i "$KUBEADM_SYSTEMD_CONF" if ! grep -q "authentication-token-webhook=true" "$KUBEADM_SYSTEMD_CONF"; then sed -e "s/--authorization-mode=Webhook/--authentication-token-webhook=true --authorization-mode=Webhook/" -i "$KUBEADM_SYSTEMD_CONF" fi systemctl daemon-reload systemctl restart kubelet
Next, modify the kube controller and kube scheduler to allow for reading data.
sed -e "s/- --address=127.0.0.1/- --address=0.0.0.0/" -i /etc/kubernetes/manifests/kube-controller-manager.yaml sed -e "s/- --address=127.0.0.1/- --address=0.0.0.0/" -i /etc/kubernetes/manifests/kube-scheduler.yaml
After you make those changes they will restart automatically after about a minute or so.
Once you do these changes, now go back to your prometheus targets page and after a minute you should see all the error messages go away.