Creating a Ceph storage cluster on Kubernetes with Rook
We’ve got our new Kubernetes self hosted cluster running, and now have a need for pod storage. You were probably wondering like me what is the right solution for storing files. Something that works well with the idea of Kubernetes (k8s for short). It needs to be something that will allow the pod to move between nodes if something happens while at the same time not killing the data storage if the pod is moved. There are a few options available, and we chose to go with Ceph.
After struggling with the Ceph Docker container project, I came across Rook. Rook is a cloud native storage service that works with Ceph. In the future they say other storage systems, but Ceph is what we needed.
Versions we are using
Centos 7.4
Kubernetes 1.9.6
Ceph 12.2.4
These are the steps we went through to get our installation up and running.
Create Rook Installation
First we need to clone the Rook Github repository, on my desktop where I’ve got kubectl installed I did the following
$ git checkout git@github.com:rook/rook.git
$ cd rook/cluster/examples/kubernetes/
Then we’re going to install Rook using the examples
$ kubectl create -f rook-operator.yaml
$ kubectl create -f rook-cluster.yaml
That gets you up and running you should see something like these pods all running in the Rook namespace
$ kubectl -n rook get pods NAME READY STATUS RESTARTS AGE rook-ceph-mds-clusterfs-585d54994f-dqxn8 1/1 Running 1 1d rook-ceph-mds-clusterfs-585d54994f-jhwtc 1/1 Running 0 1d rook-ceph-mgr0-cfccfd6b8-hv7kz 1/1 Running 0 3d rook-ceph-mon1-5hmvx 1/1 Running 0 3d rook-ceph-mon2-zkrjj 1/1 Running 0 3d rook-ceph-mon3-wqzkg 1/1 Running 0 21h rook-ceph-osd-2wzjd 1/1 Running 0 3d rook-ceph-osd-f5j57 1/1 Running 0 3d rook-ceph-osd-lldnz 1/1 Running 1 3d rook-ceph-osd-x48f7 1/1 Running 0 3d
You’ve got the Monitors (mon), Storage (osd), Manager (mgr) all running now.
Creating Block Storage
Next you’ll need to apply the settings for doing block storage. If you’re using a pod and want to do a persistent volume (pv and pvc) you’ll need to have block storage enabled, otherwise your pod’s storage is tied to a node which can be bad. This do this you’ll need to create a storage class. Before running this example file, I changed the setting replicated size = 3, we want all of our storage to be redundant in case a single nodes has a problem.
kubectl create -f rook-storageclass.yaml
Now you’re able to use a volume type of “rook-block”.
Testing Block Storage
Lets test it out using the Rook WordPress example setup
$ kubectl create -f mysql.yaml $ kubectl create -f wordpress.yaml
I personally changed both of mine from the resources requests storage to 20GB of storage to 1GB, that’s overkill for a simple test WP installation. You can check to make sure it worked
$ kubectl get pvc
And you’ll see your persistent volume claims being setup.
One thing about volumes and pods, they are one to one. Meaning it can only be mounted on a single pod, doesn’t allow you to have multiple pods all pointing to the same file structure.
Creating a File System
Because of that we needed to have an ftp, rsync, and registry pods all on the same file structure. We did this by setting up a CephFS file system. A big no no, I’m going to tell you now that we got hung up on for a while, do not create multiple file systems. This is features is experimental, where it does work, but multiple fs will cause your mons to crash.
To create a file system, you can run the rook-filesystem. I personally edited mine, updated the name and most important to have replicated size = 3, so we have redundancy in the data being store in case any single node goes out.
$ kubectl create -f rook-filesystem.yaml
After that has been completed you can now mount the file system on your pods. We have different groups use different sub-directories. Remember you’ll need to pre-create the sub-directories on the fs before you can use that path and mount on a pod. We did this by creating a fsvolume container that all it does is mount the file system to easily browse files.
Testing the File System
Another way to test is using Rook Toolbox and mount the volume.
kubectl create -f rook-tools.yaml
Make sure the pod is running
kubectl -n rook get pod rook-tools
Next shell into the pod and execute commands to test your installation
kubectl -n rook exec -it rook-tools bash
Once you’re logged in you can execute the following commands
root@rook-tools:/# mon_endpoints=$(grep mon_host /etc/ceph/ceph.conf | awk '{print $3}')
root@rook-tools:/# my_secret=$(grep key /etc/ceph/keyring | awk '{print $3}')
root@rook-tools:/# mount -t ceph -o mds_namespace=myfs,name=admin,secret=$my_secret $mon_endpoints:/ /mnt
root@rook-tools:/# ls -la /mnt/
total 4
drwxr-xr-x 1 root root 1 Apr 19 21:40 .
drwxr-xr-x 1 root root 4096 Apr 19 20:41 ..
drwxr-xr-x 1 1001 1001 3 Apr 19 22:12 example
You should be now be ready to use block storage and file system storage on your Ceph Rook installation.