Connecting Kubernetes To A Ceph Cluster Using Rook

Using Kubernetes for container orchestration is great but if you are going to scale our deployment into a multi-node cluster, you will likely find a new challenge quickly. Using local storage for your containers will quickly break the value of orchestration as your containers change nodes from time to time. This is where network based file storage systems become very valuable when used in-conjunction with a Kubernetes cluster. There are numerous different network file storage systems that are readily available for integration into Kubernetes. For this tutorial, I am going to focus on Ceph as it is often found in the on-premises environments that my tutorials are focused towards.

Ceph offers three types of storage that can all be used in Kubernetes which are object storage (rados), block storage (RBD), and file systems (Ceph FS). This tutorial will focus on the latter two as that is all that is needed to hit the ground running with a number of common scenarios. It’s also worth mentioning that setting up the Ceph object storage system rados is a fairly intensive process that would at least justify it’s own blog post. You are probably already familiar with Ceph’s object storage system rados though as it’s a cross-compatible equivalent of Amazon Web Service’s S3 product. Anywhere you can use AWS’s S3 for storage, you can alternatively use Ceph’s rados gateway to achieve the same outcome.

There are a number of different ways to connect a Kubernetes cluster to Ceph to achieve network based data storage but I will be focusing on a method that uses Rook. This method seems to be the most popular choice currently for ease of use and efficiency. Before you get started, you will need a command shell in an environment that has kubectl available with the appropriate configuration to allow administrative access to the cluster you wish to connect to Ceph. You will also need administrative access to the Ceph cluster which you intend to connect to the Kubernetes cluster. This tutorial will focus on using CLI based access to Ceph in order to complete the steps but if you know what you’re doing, you can achieve some of the same goals using a GUI based approach such as the controls provided in Proxmox Virtualization Environment.

I have not widely tested this implementation with various combinations of Ceph and Rook but you should be aware there is a history of specific releases having known issues that require some manual intervention. My test case is based off of a Ceph 16.2.7 deployment running on top of Proxmox Virtualization Environment 7.1-7 with Rook version 1.8.1.

Let’s start by gathering some data from the Ceph environment that will be needed to connect Kubernetes to the Ceph environment. Execute the following commands in the Ceph environment with a user that has the appropriate permissions to access sensitive Ceph configuration data (such as root) and then copy the output so that it can be executed in your Kubernetes environment:

ROOK_EXTERNAL_FSID=$(ceph fsid)

ROOK_EXTERNAL_CEPH_MON_DATA=$(ceph mon dump -f json 2>/dev/null|jq --raw-output .mons[0].name)=$(ceph mon dump -f json 2>/dev/null|jq --raw-output .mons[0].public_addrs.addrvec[0].addr)

ROOK_EXTERNAL_ADMIN_SECRET=$(ceph auth get-key client.admin)

clear
echo 'export NAMESPACE=rook-ceph-external'
echo 'export ROOK_EXTERNAL_FSID='"$ROOK_EXTERNAL_FSID"
echo 'export ROOK_EXTERNAL_CEPH_MON_DATA='"$ROOK_EXTERNAL_CEPH_MON_DATA"
echo 'export ROOK_EXTERNAL_ADMIN_SECRET='"$ROOK_EXTERNAL_ADMIN_SECRET"

This should produce a result similar to the following:

export NAMESPACE=rook-ceph-external
export ROOK_EXTERNAL_FSID=XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
export ROOK_EXTERNAL_CEPH_MON_DATA=NODE_HOSTNAME=000.000.000.000:3300
export ROOK_EXTERNAL_ADMIN_SECRET=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Copy all four of those lines and stash them in a text file for use later in this tutorial. Now you need to grab a copy of the Rook project files so execute the following commands in your kubectl enabled environment:

git clone -b v1.8.1 https://github.com/rook/rook.git
cd rook/deploy/examples

Now it’s time to load the Kubernetes cluster with all of the Rook / Ceph dependencies needed to connect the two environments. Execute the following command:

kubectl create -f crds.yaml -f common.yaml -f operator.yaml -f common-external.yaml

You can check that all is well and completed by running the following command:

kubectl get pods -n rook-ceph

This command should yield an output similar to the following:

NAME                                 READY   STATUS    RESTARTS   AGE
rook-ceph-operator-b89545b4f-psh7j   1/1     Running   0          2m56s

Now it’s time to setup some security and configuration related objects in Kubernetes to support the connection. Make a copy of the command output from the earlier step that you stashed away and execute those four commands in your Kubernetes environment.

The rest of this step is pretty easy as Rook has provided a bash script to do the work for you by using the values set in the environment variables from the previous step. Execute the following command:

bash import-external-cluster.sh

This next step shouldn’t be necessary but there is still a bit of a lingering oversight in the Rook’s implementation design that results in an issue where a blank Ceph username and secret value are stored into the associated Kubernetes configuration. Execute the following command:

kubectl edit secret rook-ceph-mon -n rook-ceph-external

This should launch a VIM style editor in the CLI with some YAML formatted data visible. You need to delete the following two lines from this view and then save changes and exit:

  ceph-secret: ""
  ceph-username: ""

Now you’re ready to actually connect the Ceph environment with Kubernetes by applying a YAML configuration that contains the CephCluster object. Execute the following command:

kubectl apply -f cluster-external.yaml

This step may take some time to fully complete depending on your environment so you can run the following command to watch for the status:

watch -n 2 kubectl get cephcluster -n rook-ceph-external

When everything is ready and all went well, you should see the “Phase” column value as “Connected”.

Now let’s move on to getting Ceph configured with a new Ceph file system to support the Kubernetes cluster. Of course, you can re-use an existing one if you like to conserve growth capacity given that each Ceph file system requires it’s own metadata server (MDS) and only one MDS can be ran on a Ceph node. Be aware though, if you re-use the same Ceph file system you may have setup for use in a PVE cluster, you need to be extra cautious with actions you take in the future as to not accidentally damage or destroy a mission critical storage system for your PVE cluster.

Update the following list of commands to reflect the name of the CephFS you would like to create or use the appropriate names of an existing CephFS that you would like to re-use for your Kubernetes environment. Once you have updated the list of commands, execute them in your kubectl environment. If you are creating a new CephFS instead of re-using an existing one, execute these commands in your Ceph environment as well.

cephfs_name=k8s-cephfs
cephfs_metadata_pool=k8s-cephfs_metadata
cephfs_data_pool=k8s-cephfs_data

If you are creating a new CephFS instead of re-using an existing one, execute the following commands in your Ceph environment:

ceph osd pool create $cephfs_metadata_pool
ceph osd pool create $cephfs_data_pool
ceph fs new $cephfs_name $cephfs_metadata_pool $cephfs_data_pool

Now that you have a Ceph file system ready for use, it’s time to setup two storage classes in your Kubernetes environment to provide containers with both block and file system storage options. You need to update the storage class templates before you apply them to the Kubernetes environment by executing the following commands:

sed -i 's/clusterID\:\srook-ceph/clusterID\: rook-ceph-external/g' csi/rbd/storageclass.yaml
sed -i 's/clusterID\:\srook-ceph/clusterID\: rook-ceph-external/g' csi/cephfs/storageclass.yaml
sed -i 's/fsName\:\smyfs/fsName\: '"$cephfs_name"'/g' csi/cephfs/storageclass.yaml
sed -i 's/pool\:\smyfs\-replicated/pool\: '"$cephfs_data_pool"'/g' csi/cephfs/storageclass.yaml
sed -i 's/csi.storage.k8s.io\/provisioner-secret-namespace\:\srook-ceph/csi.storage.k8s.io\/provisioner-secret-namespace\: rook-ceph-external/g' csi/cephfs/storageclass.yaml
sed -i 's/csi.storage.k8s.io\/controller-expand-secret-namespace\:\srook-ceph/csi.storage.k8s.io\/controller-expand-secret-namespace\: rook-ceph-external/g' csi/cephfs/storageclass.yaml
sed -i 's/csi.storage.k8s.io\/node-stage-secret-namespace\:\srook-ceph/csi.storage.k8s.io\/node-stage-secret-namespace\: rook-ceph-external/g' csi/cephfs/storageclass.yaml

Now you can apply the storage class templates to the Kubernetes environment by executing the following command:

kubectl create -f csi/rbd/storageclass.yaml -f csi/cephfs/storageclass.yaml

The final step to this process is to set a default storage class for scenarios where a storage class is not specifically defined. I typically choose to use the RBD storage class as I most commonly use that. I use the CephFS storage class for scenarios where data needs to be consumed by more than one container simultaneously such as web application files in a high availability configuration. Execute the following command to set the default storage class to the new Ceph RBD option:

kubectl patch storageclass rook-ceph-block -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

That’s it! You’re now ready to start deploying containers that consume storage directly from your Ceph cluster.

I recommend you check out this post next to take your Kubernetes deployment to the next level.