Kubernetes error: namespaces “gitlab-managed-apps” is forbidden: User “system:serviceaccount:gitlab-managed-apps:gitlab-sa” cannot get namespaces in the namespace “gitlab-managed-apps”
Updated 9/19/2018: Please take a look at this article, one of my co-workers explains Microservices Workflow and includes Gitlab Setup process.
Today I’ve been working on getting a Gitlab CE server connected to Kubernetes for deployments and testing, but having some issues.
I’m running a stock Kubernetes installation on CentOS 7.4 with these versions
kubeadm-1.9.3-0.x86_64
kubelet-1.9.3-0.x86_64
kubernetes-cni-0.6.0-0.x86_64
kubectl-1.9.3-0.x86_64
gitlab-ce-10.5.2-ce.0.el6.x86_64
This cluster has 1 controller and 2 nodes to start. After putting all the settings in to Gitlab, we then attempt to install Helm. This is the error we’re getting when attempting to install Helm Tiller through the Gitlab web interface on a newly added Kubernetes cluster.
Kubernetes error: namespaces "gitlab-managed-apps" is forbidden: User "system:serviceaccount:gitlab-managed-apps:gitlab-sa" cannot get namespaces in the namespace "gitlab-managed-apps"
This is how we set up Gitlab, and how we got that error, and what we did to a work around for the moment.
So I went into a project in Gitlab, and selected project > CI/CD > Kubernetes
Create Kubernetes cluster, and we’re not using GKE, we have our own self hosted installation of Kubernetes on a group VPS of servers for testing.
Enter Cluster Name, API URL, CA Certificate, Token and Project namespace
I’ll explain how to get each of these on your self-hosted stock Kubernetes cluster.
Cluster name that is something you call your cluster, enter what you want here.
API URL can be seen by logging into your cluster master and typing
kubectl cluster-info
Output should look something like
Kubernetes master is running at https://10.81.236.201:6443 KubeDNS is running at https://10.81.236.201:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
The API URL would be https://10.81.236.201:6443
CA Certificate is in the configuration directory, we’re using a self signed certificate so you’ll need to grab the certificate info from the cluster.
cat /etc/kubernetes/pki/ca.crt
Token now this is tricky one, honestly I still think we’re a little off how this is configured, so this will probably change in my next blog.
Create gitlab-sa.yaml file with the contents
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: gitlab-sa
namespace: gitlab-managed-apps
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: gitlab-role
namespace: gitlab-managed-apps
rules:
- apiGroups:
- ""
- extensions
resources:
- '*'
verbs:
- '*'
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: gitlab-rb
namespace: gitlab-managed-apps
subjects:
- kind: ServiceAccount
name: gitlab-sa
namespace: gitlab-managed-apps
roleRef:
kind: Role
name: gitlab-role
apiGroup: rbac.authorization.k8s.io
The file will create a Service Account, then create a Role then bind the role to the Service Account.
Now execute the following commands
kubectl create namespace gitlab-managed-apps
kubectl create -f gitlab-sa.yaml
That will create a name space, then create the gitlab-sa service account in that name space along with it’s role settings.
After that we need to get the service accounts token
kubectl get secret --namespace=gitlab-managed-apps
You’ll see the token name like gitlab-sa-token-xxxxx, you’re will look a little different than mine:
kubectl describe secrets/gitlab-sa-token-x2qtg --namespace=gitlab-manage-apps
Now put that long token string into your Gitlab and Save changes
Now you’ll want to attempt to install Helm Tiller through Gitlab. Click the Install button and after 10 seconds you get an error message which will look something like this that I mentioned at the beginning.
Kubernetes error: namespaces "gitlab-managed-apps" is forbidden: User "system:serviceaccount:gitlab-managed-apps:gitlab-sa" cannot get namespaces in the namespace "gitlab-managed-apps"
After doing a bunch of research we tried a bunch of different ways of setting up our Service Account with Roles, and essentially kept getting the same error. One thing I did see after reading Connecting GitLab with Kubernetes was someone who was using GKE and said to enable insecure on your GKE setup, however we’re not using GKE.
After continuing research found a thread discussing installing Helm with default RBAC rules since we’re trying to install Helm Tiller on Gitlab. I saw someone mentioned about setting insecure, by disabling RBAC using the command
kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --user=admin --user=kubelet --group=system:serviceaccounts
After making that change our Helm Tiller, Ingress and Prometheus module all installed from Gitlab.
However keep in mind, this is my first attempt and clearly it’s not good practice to be disabling RBAC, but this is a test.