Kubernetes pod getting rbd mount error

Today I had a couple of our Kubernetes nodes get rebooted in the data center after they came back up, our Elastic Search pods were stuck in Init or Error states.  Looking checking out the error logs on the pods it said that it was getting an rbd mount error.

ceph-volumeattacher: failed rbd single_major check, assuming it's unsupported: failed to check for rbd module single_major param: Failed to complete 'check kmod param': exit status 1. modinfo: ERROR: could not get modinfo from 'rbd': Exec format error

Load the rbd driver for Ceph

I logged into the node as root and loaded the kernel driver

modprobe rbd

After fixing the issue for the current running kernel, we need to add the kernel module to load automatically when the server reboots to eliminate this problem in the future, create a file in modules-load.d directory like this

vi /etc/modules-load.d/rbd.conf

Next enter contents of the file:

# Load Ceph RBD Kernel Driver
rbd

Save the changes.

Now after a reboot, my system will automatically load the rbd driver for mounting Ceph volumes.