We have some users who are trying to push Docker containers in to a Gitlab registry and their push is being rejected because of an invalid certificate. This was working last week before doing yum update, upgrading from Gitlab 10.2.x, and enabling HTTPS on the Gitlab web interface using WeEncrypt certificates.
We’re running the following software versions on the Gitlab server
Gitlab CE 10.5.2
Comodo Wildcard Certificate
CentOS 6.9
Client workstation
CentOS 7.4
Docker CE 17.20.0
and attempting to do docker login with
docker login -u jeff@example.com -p PASSWORD registry.example.com:5050
This is the error the developer was getting:
Error response from daemon: Get https://registry.example.com:5050/v2/: Get https://gitlab.example.com/jwt/auth?account=jeff%40example.com&client_id=docker&offline_token=true&service=container_registry: x509: certificate signed by unknown authority
There was a secondary issue as well that started happening, normal users trying to check out code were now forced to use SSL which is what we want
git clone https://gitlab.example.com/project/mysite-web.git
however, they are where getting an error and not able to successfully clone the repo
fatal: unable to access 'https://gitlab.example.com/project/mysite-web.git/': Peer's Certificate issuer is not recognized.
So I assumed it was the WeEncrypt certificate not working for some reason, and replaced it with our wildcard certificate. I did some tests using curl
curl -l -v https://registry.example.com:5050
which looks something like this:
* About to connect() to gitlab.example.com port 443 (#0) * Trying 10.96.132.13... * Connected to gitlab.example.com (10.96.132.13) port 443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 * Server certificate: * subject: CN=*.example.com,OU=PositiveSSL Wildcard,OU=Domain Control Validated * start date: Sep 19 00:00:00 2017 GMT * expire date: Sep 19 23:59:59 2018 GMT * common name: *.example.com * issuer: CN=COMODO RSA Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB
It looks okay, but not completely working yet.
Basically it seems that the SSL certificate settings on Gitlab needed to be changed. I need to make sure both the registry and the repo were using a pem file.
Placed all my files in /etc/gitlab/ssl
__example.com.ca-bundle
__example.com.crt
star.example.com.key
Then created a pem file by combining the ca and crt
cat __example.com.crt __example.com.ca-bundle > star.example.com.pem
Now I updated the configuration for Gitlab server
vi /etc/gitlab/gitlab.rb
Ensure you have the following lines, I added mine at the bottom after all the commented out examples just so I can see all my settings in a common location
external_url 'https://gitlab.example.com' nginx['redirect_http_to_https'] = true nginx['redirect_http_to_https_port'] = 80 nginx['ssl_certificate'] = "/etc/gitlab/ssl/star.example.com.pem" nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/star.example.com.key" registry['enable'] = true registry_external_url 'https://registry.example.com:5050' gitlab_rails['registry_api_url'] = "http://localhost:5000" registry['registry_http_addr'] = "localhost:5000" registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/star.example.com.pem" registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/star.example.com.key"
Next reconfigure Gitlab settings
gitlab-ctl reconfigure
Then restart the two services we modified
gitlab-ctl restart registry
gitlab-ctl restart nginx
Now test by running the docker login and git clone command again.
# docker login -u jeff@example.com -p PASSWORD registry.example.com:5050 WARNING! Using --password via the CLI is insecure. Use --password-stdin. Login Succeeded # git clone https://gitlab.example.com/project/mysite-web.git Cloning into 'mysite-web'... Username for 'https://gitlab.example.com': jeff@example.com Password for 'https://jeff@example.com@gitlab.example.com': remote: Counting objects: 86729, done. remote: Compressing objects: 100% (24177/24177), done. Receiving objects: 83% (71986/86729), 109.71 MiB | 13.37 MiB/s
Looks like the both work, problem fixed.