Kubernetes pod getting rbd mount error
Today I had a couple of our Kubernetes nodes get rebooted in the data center after they came back up, our Elastic Search pods were stuck in Init or Error states. Looking checking out the error logs on the pods it said that it was getting an rbd mount error.
ceph-volumeattacher: failed rbd single_major check, assuming it's unsupported: failed to check for rbd module single_major param: Failed to complete 'check kmod param': exit status 1. modinfo: ERROR: could not get modinfo from 'rbd': Exec format error
Load the rbd driver for Ceph
I logged into the node as root and loaded the kernel driver
modprobe rbd
After fixing the issue for the current running kernel, we need to add the kernel module to load automatically when the server reboots to eliminate this problem in the future, create a file in modules-load.d directory like this
vi /etc/modules-load.d/rbd.conf
Next enter contents of the file:
# Load Ceph RBD Kernel Driver rbd
Save the changes.
Now after a reboot, my system will automatically load the rbd driver for mounting Ceph volumes.