Ceph offers three types of storage that can all be used in Kubernetes which are object storage (rados), block storage (RBD), and file systems (Ceph FS). This tutorial will focus on the latter two as that is all that is needed to hit the ground running with a number of common scenarios. It’s also worth mentioning that setting up the Ceph object storage system rados is a fairly intensive process that would at least justify it’s own blog post. You are probably already familiar with Ceph’s object storage system rados though as it’s a cross-compatible equivalent of Amazon Web Service’s S3 product. Anywhere you can use AWS’s S3 for storage, you can alternatively use Ceph’s rados gateway to achieve the same outcome.

There are a number of different ways to connect a Kubernetes cluster to Ceph to achieve network based data storage but I will be focusing on a method that uses Rook. This method seems to be the most popular choice currently for ease of use and efficiency. Before you get started, you will need a command shell in an environment that has kubectl available with the appropriate configuration to allow administrative access to the cluster you wish to connect to Ceph. You will also need administrative access to the Ceph cluster which you intend to connect to the Kubernetes cluster. This tutorial will focus on using CLI based access to Ceph in order to complete the steps but if you know what you’re doing, you can achieve some of the same goals using a GUI based approach such as the controls provided in Proxmox Virtualization Environment.

I have not widely tested this implementation with various combinations of Ceph and Rook but you should be aware there is a history of specific releases having known issues that require some manual intervention. My test case is based off of a Ceph 16.2.7 deployment running on top of Proxmox Virtualization Environment 7.1-7 with Rook version 1.8.1.

Let’s start by gathering some data from the Ceph environment that will be needed to connect Kubernetes to the Ceph environment. Execute the following commands in the Ceph environment with a user that has the appropriate permissions to access sensitive Ceph configuration data (such as root) and then copy the output so that it can be executed in your Kubernetes environment:

ROOK_EXTERNAL_FSID=$(ceph fsid)

ROOK_EXTERNAL_CEPH_MON_DATA=$(ceph mon dump -f json 2>/dev/null|jq --raw-output .mons[0].name)=$(ceph mon dump -f json 2>/dev/null|jq --raw-output .mons[0].public_addrs.addrvec[0].addr)

ROOK_EXTERNAL_ADMIN_SECRET=$(ceph auth get-key client.admin)

clear
echo 'export NAMESPACE=rook-ceph-external'
echo 'export ROOK_EXTERNAL_FSID='"$ROOK_EXTERNAL_FSID"
echo 'export ROOK_EXTERNAL_CEPH_MON_DATA='"$ROOK_EXTERNAL_CEPH_MON_DATA"
echo 'export ROOK_EXTERNAL_ADMIN_SECRET='"$ROOK_EXTERNAL_ADMIN_SECRET"

This should produce a result similar to the following:

export NAMESPACE=rook-ceph-external
export ROOK_EXTERNAL_FSID=XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
export ROOK_EXTERNAL_CEPH_MON_DATA=NODE_HOSTNAME=000.000.000.000:3300
export ROOK_EXTERNAL_ADMIN_SECRET=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Copy all four of those lines and stash them in a text file for use later in this tutorial. Now you need to grab a copy of the Rook project files so execute the following commands in your kubectl enabled environment:

git clone -b v1.8.1 https://github.com/rook/rook.git
cd rook/deploy/examples

Now it’s time to load the Kubernetes cluster with all of the Rook / Ceph dependencies needed to connect the two environments. Execute the following command:

kubectl create -f crds.yaml -f common.yaml -f operator.yaml -f common-external.yaml

You can check that all is well and completed by running the following command:

kubectl get pods -n rook-ceph

This command should yield an output similar to the following:

NAME                                 READY   STATUS    RESTARTS   AGE
rook-ceph-operator-b89545b4f-psh7j   1/1     Running   0          2m56s

Now it’s time to setup some security and configuration related objects in Kubernetes to support the connection. Make a copy of the command output from the earlier step that you stashed away and execute those four commands in your Kubernetes environment.

The rest of this step is pretty easy as Rook has provided a bash script to do the work for you by using the values set in the environment variables from the previous step. Execute the following command:

bash import-external-cluster.sh

This next step shouldn’t be necessary but there is still a bit of a lingering oversight in the Rook’s implementation design that results in an issue where a blank Ceph username and secret value are stored into the associated Kubernetes configuration. Execute the following command:

kubectl edit secret rook-ceph-mon -n rook-ceph-external

This should launch a VIM style editor in the CLI with some YAML formatted data visible. You need to delete the following two lines from this view and then save changes and exit:

  ceph-secret: ""
  ceph-username: ""

Now you’re ready to actually connect the Ceph environment with Kubernetes by applying a YAML configuration that contains the CephCluster object. Execute the following command:

kubectl apply -f cluster-external.yaml

This step may take some time to fully complete depending on your environment so you can run the following command to watch for the status:

watch -n 2 kubectl get cephcluster -n rook-ceph-external

When everything is ready and all went well, you should see the “Phase” column value as “Connected”.

Now let’s move on to getting Ceph configured with a new Ceph file system to support the Kubernetes cluster. Of course, you can re-use an existing one if you like to conserve growth capacity given that each Ceph file system requires it’s own metadata server (MDS) and only one MDS can be ran on a Ceph node. Be aware though, if you re-use the same Ceph file system you may have setup for use in a PVE cluster, you need to be extra cautious with actions you take in the future as to not accidentally damage or destroy a mission critical storage system for your PVE cluster.

Update the following list of commands to reflect the name of the CephFS you would like to create or use the appropriate names of an existing CephFS that you would like to re-use for your Kubernetes environment. Once you have updated the list of commands, execute them in your kubectl environment. If you are creating a new CephFS instead of re-using an existing one, execute these commands in your Ceph environment as well.

cephfs_name=k8s-cephfs
cephfs_metadata_pool=k8s-cephfs_metadata
cephfs_data_pool=k8s-cephfs_data

If you are creating a new CephFS instead of re-using an existing one, execute the following commands in your Ceph environment:

ceph osd pool create $cephfs_metadata_pool
ceph osd pool create $cephfs_data_pool
ceph fs new $cephfs_name $cephfs_metadata_pool $cephfs_data_pool

Now that you have a Ceph file system ready for use, it’s time to setup two storage classes in your Kubernetes environment to provide containers with both block and file system storage options. You need to update the storage class templates before you apply them to the Kubernetes environment by executing the following commands:

sed -i 's/clusterID\:\srook-ceph/clusterID\: rook-ceph-external/g' csi/rbd/storageclass.yaml
sed -i 's/clusterID\:\srook-ceph/clusterID\: rook-ceph-external/g' csi/cephfs/storageclass.yaml
sed -i 's/fsName\:\smyfs/fsName\: '"$cephfs_name"'/g' csi/cephfs/storageclass.yaml
sed -i 's/pool\:\smyfs\-replicated/pool\: '"$cephfs_data_pool"'/g' csi/cephfs/storageclass.yaml
sed -i 's/csi.storage.k8s.io\/provisioner-secret-namespace\:\srook-ceph/csi.storage.k8s.io\/provisioner-secret-namespace\: rook-ceph-external/g' csi/cephfs/storageclass.yaml
sed -i 's/csi.storage.k8s.io\/controller-expand-secret-namespace\:\srook-ceph/csi.storage.k8s.io\/controller-expand-secret-namespace\: rook-ceph-external/g' csi/cephfs/storageclass.yaml
sed -i 's/csi.storage.k8s.io\/node-stage-secret-namespace\:\srook-ceph/csi.storage.k8s.io\/node-stage-secret-namespace\: rook-ceph-external/g' csi/cephfs/storageclass.yaml

Now you can apply the storage class templates to the Kubernetes environment by executing the following command:

kubectl create -f csi/rbd/storageclass.yaml -f csi/cephfs/storageclass.yaml

The final step to this process is to set a default storage class for scenarios where a storage class is not specifically defined. I typically choose to use the RBD storage class as I most commonly use that. I use the CephFS storage class for scenarios where data needs to be consumed by more than one container simultaneously such as web application files in a high availability configuration. Execute the following command to set the default storage class to the new Ceph RBD option:

kubectl patch storageclass rook-ceph-block -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

That’s it! You’re now ready to start deploying containers that consume storage directly from your Ceph cluster.

I recommend you check out this post next to take your Kubernetes deployment to the next level.

To get started, you need at least one clean Ubuntu 20.04 environment, ideally at least two Ubuntu 20.04 environments to separate your control plane and worker nodes. For high availability environments, you should have at least three control plane nodes and at least two worker nodes. You don’t have to create all the additional nodes at once as adding them later isn’t much of a chore. The instructions that follow will assume you’re working with at least two nodes, one for your control plane, and one for your workload. It is very important that every node you intend to join to the same cluster has a unique hostname configured. It does not matter if two nodes are located in different network search domains, their hostname must specifically be unique to the cluster.

Planning Cluster Configuration

Before you start executing commands, you need to get all your ducks in a row with a few pieces of configuration information. The following is an example of the configuration used in the Azorian Solutions production Kubernetes cluster:

             Cluster Domain: k8s.azorian.solutions
     Control Plane Endpoint: cp.k8s.azorian.solutions
          Pod Networks CIDR: 10.255.0.0/16
      Service Networks CIDR: 10.10.10.0/23
            Kubelet Version: 1.22.0-00
            Kubeadm Version: 1.22.0-00
            Kubectl Version: 1.22.0-00

The first configuration item is the cluster domain. Think of this much like you would a network search domain although it isn’t necessarily used in the same way.

The next configuration item is the control plane endpoint. This is typically a domain name containing an A record corresponding to the IP address of each control plane node that will operate in the cluster. Initially, this domain should only contain a single A record that points to the IP address of the first control plane node that you intend to initialize the Kubernetes cluster on. If you miss this detail during setup, it could lean to painful results forcing you to start the entire process all over again (assuming you don’t have great proficiency in Kubernetes clusters to fix it).

Moving on, the pod networks CIDR should be a fairly large private network allocation as this will be carved up into many smaller networks for each pod that gets deployed to the cluster. Remember, this could easily equate to one network per container deployed depending on how you deploy your services so don’t get stingy on address space. This is not something that is easily changed after the fact. The allocation chosen here should not be your easiest to remember as you likely won’t spend much if any time directly interacting with the IP addresses that are assigned from these networks. It’s also important to remember that the addresses automatically assigned from this network are ephemeral in nature in accordance with the destruction and recreation of pods that happen automatically from time to time.

Next up is the service networks CIDR. Until you get familiar with exactly how Kubernetes works, I recommend starting with a /23 private network as this will ensure you don’t likely end up in a situation where you run out of space from deploying services to the cluster. Services in Kubernetes are like a public facing gateway to your various pods (which contain one or more containers). You can think of a service similar to how you would think of a load balancer. It will receive communication requests to specific IP addresses and ports which are then automatically routed to any pods that have been linked to the service.

Finally, you have your Kubelet, Kubeadm, and Kubectl versions. You can select any of the available Kubernetes release versions here. It is considered best practice for compatibility reasons to keep all three of these packages in sync with at least major and minor versions. If you know what you’re doing, you can install different versions of each but you may experience problems by doing so if there is a breaking change between two selected versions. Version 1.23 hasn’t been out long and I haven’t tested it so for this tutorial, we’ll stick with version 1.22.0-00 which is what I am running today.

Preparing The Nodes

Let’s start by running a number of commands to get each node of the Kubernetes cluster prepared for deployment. Begin by opening up a command shell to each of the nodes with a non-root user that has super user (sudo) privileges.

First, you need to disable the use of swap partitions. Execute the following commands on each node:

sudo sed -i 's/^\/swap/#\/swap/g' /etc/fstab
sudo swapoff -a

Next, you need to update the OS to the latest packages for the current distribution. Execute the following commands on each node:

sudo apt update
sudo apt upgrade -y

Now it’s time to install some basic dependencies to support the remainder of the installation process. Execute the following command on each node:

sudo apt install -y apt-transport-https ca-certificates curl gnupg lsb-release net-tools containerd

Now you need to load the “overlay” and “br_netfilter” Linux kernel modules. Execute the following commands on each node:

sudo modprobe overlay
sudo modprobe br_netfilter

The previous commands only load kernel modules temporarily. You need to add a configuration file to ensure that these modules are loaded after each machine reboot. Execute the following command on each node:

echo 'overlay
br_netfilter' | sudo tee /etc/modules-load.d/k8s.conf > /dev/null

Now you need to add a kernel configuration file to tweak three networking settings. Execute the following command on each node:

echo 'net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1' | sudo tee /etc/sysctl.d/99-k8s.conf > /dev/null

The first of the three aforementioned settings instruct the kernel to accept and forward IP packets not intended for the host machine, but instead one or more of the guests running on the host. The second and third settings control whether or not packets passing through a Linux networking bridge are passed to iptables for processing.

Now you need to instruct the kernel to load the new kernel configuration file to apply the changes. Execute the following command on each node:

sudo sysctl --system

Now you need to install the GPG key for the Google APT repository that contains the Kubernetes packages you’ll need. Execute the following command on each node:

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

Now you need to add a configuration file for the APT package manager to make the system aware of the Kubernetes package repository hosted by Google. Execute the following command on each node:

echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list > /dev/null

Now you need to update APT’s package cache to download the latest package information from the Kubernetes repository. Execute the following command on each node:

sudo apt update

Now you need to set some temporary environment variables that will be used during the Kubernetes package installation process. Execute the following commands on each node, taking care to update each variable with the appropriate versions that you chose for Kubelet, Kubeadm, and Kubectl at the beginning of this tutorial:

kubelet_version=1.22.0-00
kubeadm_version=1.22.0-00
kubectl_version=1.22.0-00

Now you are finally ready to install the Kubernetes packages. Execute the following commands on each node:

KUBELET_EXTRA_ARGS='--feature-gates="AllAlpha=false,RunAsGroup=true"
  --container-runtime=remote --cgroup-driver=systemd
  --runtime-request-timeout=5m
  --container-runtime-endpoint="unix:///var/run/containerd/containerd.sock"'

sudo apt install -y --allow-change-held-packages \
  kubelet=$kubelet_version kubeadm=$kubeadm_version \
  kubectl=$kubectl_version kubernetes-cni

Now there is one final dependency to install which is the control script for the Calico CNI since this tutorial will be making use of Calico as the container networking interface (CNI). Execute the following commands on each node:

export calico_version=$(curl --silent "https://api.github.com/repos/projectcalico/calico/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')

curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" -o kubectl-calico -L https://github.com/projectcalico/calico/releases/download/$calico_version/calicoctl-linux-amd64

sudo chown root:root kubectl-calico
sudo chmod +x kubectl-calico
sudo mv kubectl-calico /usr/local/bin/
sudo ln -s /usr/local/bin/kubectl-calico /usr/local/bin/calicoctl

Alright, that completes the node preparation phase of the deployment! I promise the next phase will be much quicker.

Initializing The Cluster

It’s finally time to initialize the Kubernetes cluster now that you have all of the nodes in a prepared state. If you are running the nodes in virtualization environment like Proxmox or ESXI, you may want to consider doing a quick snapshot of each node just in case something doesn’t go right with the initialization process. This will give you a relatively simple way to revert the coming changes and try again without having to completely rebuild the cluster nodes all over again.

Let’s start by setting some additional temporary environment variables to assist with building a cluster configuration file. Execute the following commands on the first control plane node only! Take care to update each variable to the corresponding configuration value chosen at the beginning of this tutorial.

cluster_domain=k8s.azorian.solutions
control_plane_endpoint=cp.k8s.azorian.solutions
pod_networks_cidr=10.255.0.0/16
service_networks_cidr=10.10.10.0/23

Now you need to create a cluster initialization file in YAML format. Execute the following command on the first control plane node only!

echo 'kind: InitConfiguration
apiVersion: kubeadm.k8s.io/v1beta2
nodeRegistration:
  criSocket: /var/run/containerd/containerd.sock
---
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta2
controlPlaneEndpoint: '"$control_plane_endpoint"':6443
networking:
  dnsDomain: '"$cluster_domain"'
  podSubnet: '"$pod_networks_cidr"'
  serviceSubnet: '"$service_networks_cidr"'
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
---
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1' | tee /tmp/kubeadm-config.yaml

Now you can finally initialize the first node of the cluster! Execute the following command on the first control plane node only!

sudo kubeadm init --config /tmp/kubeadm-config.yaml --upload-certs

If all goes well with the cluster initialization, then the aforementioned command should produce some output that provides two different commands beginning with “kubeadm join” that you need to capture for use a little later in this tutorial.

Now you need to copy the admin security configuration file of the cluster to the appropriate location in the current user’s home directory so that you can execute various cluster administration commands as the current user without the user of super user (sudo) privileges. Execute the following command on the first control plane node only!

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You now have a bare bones Kubernetes cluster! You aren’t done quiet yet though as this cluster doesn’t have any container networking interface. Read on.

Installing Calico CNI

This step should prove to be the easiest. Execute the following command on the first control plane node only!

curl https://docs.projectcalico.org/manifests/calico.yaml | kubectl apply -f -

Alright, now the cluster has a container network interface installed and is ready for use if you are only working with a single node. Otherwise, read on.

Initializing The Remaining Nodes

If you are setting up more than one node for your Kubernetes environment, then you need to execute the “kubeadm join” commands that you captured during cluster initialization in an earlier step. For each of the worker nodes you are going to join to the cluster, execute the shorter of the two commands (the one that does not contain the –control-plane parameter) on the node.

If you are setting up more than one control plane nodes for your Kubernetes environment, then you need to execute the longer of the two commands (the one that does contain the –control-plane parameter) on each remaining control plane nodes. Additionally, for ease of operation, also execute the following commands on each of the additional control plane nodes:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

There is one final step to ensure that the Kubernetes cluster is ready to start deploying components. Each node that will be handling workloads needs to have a Kubernetes worker role label attached. Run the following command from a control plane node for each worker node in the cluster taking care to substitute the “HOSTNAME-HERE” string with the node’s configured hostname. If you aren’t sure of the node’s hostname, you can determine this by executing the “hostname” command on the node in question.

kubectl label node HOSTNAME-HERE node-role.kubernetes.io/worker=worker

That’s it! You now have an operational Kubernetes cluster environment. I don’t want to toot your horn, but you’re kind of a big deal now.

I recommend you check out this post next to take your Kubernetes deployment to the next level.