• Share
• Tweet
• Share
• Email

Prometheus kubelet metrics server returned HTTP status 403 Forbidden

Recently been working a lot with Kubernetes and needed to install some monitoring to better profile the cluster and it’s components.  After doing that then doing an installation using the their deploy script.  It all looked good, no errors, all the pods and services were running.  I went ahead and launched the Prometheus web interface and saw there was a bunch of errors and it was not able to access Kubelet metrics.

Setup

I first cloned the project for prometheus-operator

git clone git@github.com:coreos/prometheus-operator.git

After that I then ran the deployment for cluster monitoring.

$ cd prometheus-operator/
$ contrib/kube-prometheus/hack/cluster-monitoring/deploy

After everything was installed, I created my own ingress.  We’re running ingress-nginx, so I created a couple of quick ingress configuration files and imported them into the cluster so I could access remotely.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: alertmanager-ingress
  namespace: monitoring
  labels:
    app: alertmanager-main
spec:
  rules:
  - host: alert.example.com
    http:
      paths:
      - backend:
          serviceName: alertmanager-main
          servicePort: 9093
        path: /
  tls:
  - hosts:
    - alert.example.com
    secretName: example-com-tls
--
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: grafana-ingress
  namespace: monitoring
  labels:
    app: grafana
spec:
  rules:
  - host: gra.example.com
    http:
      paths:
      - backend:
          serviceName: grafana
          servicePort: 3000
        path: /
  tls:
  - hosts:
    - gra.example.com
    secretName: example-com-tls
--
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prometheus-ingress
  namespace: monitoring
  labels:
    app: prometheus-operated
spec:
  rules:
  - host: prom.example.com
    http:
      paths:
      - backend:
          serviceName: prometheus-operated
          servicePort: 9090
        path: /
  tls:
  - hosts:
    - prom.example.com
    secretName: example-com-tls

You can save that as prometheus-ingress.yaml, then import.

$ kubectl apply -f prometheus-ingress.yaml

Now that you’ve got your Prometheus running and ingress routing setup, we can do a test.  Launch a web browser and go to

https://prom.example.com/targets

Once that opens, check and make sure if you have any errors.  Everything should look good, except for kubelet section.  This is where I had errors on all the metrics collections.  The error I had

HTTP status 403 Forbidden

After doing some research I determined, that after you complete the pod installation there was another setup when using kubeadm.

Follow the instruction you’ll need to modify some configuration files on your master node.

Fist update the kubelet service to include webhook and restart.

KUBEADM_SYSTEMD_CONF=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
sed -e "/cadvisor-port=0/d" -i "$KUBEADM_SYSTEMD_CONF"
if ! grep -q "authentication-token-webhook=true" "$KUBEADM_SYSTEMD_CONF"; then
  sed -e "s/--authorization-mode=Webhook/--authentication-token-webhook=true --authorization-mode=Webhook/" -i "$KUBEADM_SYSTEMD_CONF"
fi
systemctl daemon-reload
systemctl restart kubelet

Next, modify the kube controller and kube scheduler to allow for reading data.

sed -e "s/- --address=127.0.0.1/- --address=0.0.0.0/" -i /etc/kubernetes/manifests/kube-controller-manager.yaml
sed -e "s/- --address=127.0.0.1/- --address=0.0.0.0/" -i /etc/kubernetes/manifests/kube-scheduler.yaml

After you make those changes they will restart automatically after about a minute or so.

Once you do these changes, now go back to your prometheus targets page and after a minute you should see all the error messages go away.

• Share
• Tweet
• Share
• Email