Ceph Configuration

The first step to this process involves configuring Ceph and extracting relevant information needed to setup your virtual machine client. Open a command shell to one of the hosts running Ceph and elevate your access to a user that has permissions to execute Ceph configuration commands.

Once you have an active command shell to the Ceph server, copy and paste the following script block into the terminal and then execute it with a return as required. This will install the APT package “jq” which is used for extracting some of the configuration settings.

(
sudo apt install -y jq

FS_NAME_DEF="cephfs"
FS_PATH_DEF="/"
FS_PERM_DEF="rw"
CLIENT_NAME_DEF="vm-client"
FS_NAME=""
FS_PATH=""
FS_PERM=""
CLIENT_NAME=""

get_fs_name () {
  clear
  echo 'Enter the name of the Ceph file system you wish to use and then press return ['"$FS_NAME_DEF"']:'
  read FS_NAME
  FS_NAME=$(echo "$FS_NAME" | xargs)
  if [[ ${#FS_NAME} -eq 0 ]]; then
    FS_NAME="$FS_NAME_DEF"
  fi
}

get_fs_path () {
  clear
  echo 'Enter the absolute path within the Ceph file system you wish to use and then press return ['"$FS_PATH_DEF"']:'
  read FS_PATH
  FS_PATH=$(echo "$FS_PATH" | xargs)
  if [[ ${#FS_PATH} -eq 0 ]]; then
    FS_PATH="$FS_PATH_DEF"
  fi
}

get_fs_perm () {
  clear
  echo 'Enter the access permissions for the previously provided path you wish to use and then press return ['"$FS_PERM_DEF"']:'
  read FS_PERM
  FS_PERM=$(echo "$FS_PERM" | xargs)
  if [[ ${#FS_PERM} -eq 0 ]]; then
    FS_PERM="$FS_PERM_DEF"
  fi
}

get_client_name () {
  clear
  echo 'Enter the name of the Ceph client node and then press return ['"$CLIENT_NAME_DEF"']:'
  read CLIENT_NAME
  CLIENT_NAME=$(echo "$CLIENT_NAME" | xargs)
  if [[ ${#CLIENT_NAME} -eq 0 ]]; then
    CLIENT_NAME="$CLIENT_NAME_DEF"
  fi
}

get_fs_name
get_fs_path
get_fs_perm
get_client_name
ceph fs authorize $FS_NAME client.$CLIENT_NAME $FS_PATH $FS_PERM > /dev/null
CLIENT_SECRET=$(ceph auth get-or-create-key client.$CLIENT_NAME)
CEPH_CONF=$(ceph config generate-minimal-conf)
MON_TARGET=$(ceph mon dump -f json 2>/dev/null|jq --raw-output .mons[0].public_addrs.addrvec[0].addr)
readarray -d : -t MON_IP<<<"$MON_TARGET"

clear
echo 'export FS_NAME='"$FS_NAME"
echo 'export FS_PATH='"$FS_PATH"
echo 'export CLIENT_NAME='"$CLIENT_NAME"
echo 'export CLIENT_SECRET='"$CLIENT_SECRET"
echo 'export MON_IP='"$MON_IP"
echo 'sudo tee /etc/ceph/ceph.conf &> /dev/null <<EOF
'"$CEPH_CONF"'
EOF
'
)

This script will prompt you for the following information;

• Ceph file system name that should be used.
• Ceph file system path that should be used. This should be an absolute path within the given file system.
• Ceph file system permissions that should be granted for the previously given file system path. This can be one of “r” for read-only, “w” for write-only, or “rw” for both read and write permissions.
• Ceph client name that should be used. This is what will be used to create a user for Ceph authentication so it should only contain alphanumeric characters as well as underscores and hyphens.

The rest of the required information will be automatically extracted by the script. If everything executes correctly, you should be left with a Bash script that looks something like this:

export FS_NAME=cephfs
export FS_PATH=/vm/docker-stacks
export CLIENT_NAME=docker-node1
export CLIENT_SECRET=AQC4YUhjP+NbOxAAH+PViHPlzP4joKruS3qkXg==
export MON_IP=172.22.1.4
sudo tee /etc/ceph/ceph.conf &> /dev/null <<EOF
# minimal ceph.conf for 20fc650f-25a4-42c6-957c-efb594ce1f01
[global]
	fsid = 20fc650f-25a4-42c6-957c-efb594ce1f01
	mon_host = [v2:172.22.1.2:3300/0,v1:172.22.1.2:6789/0] [v2:172.22.1.3:3300/0,v1:172.22.1.3:6789/0] [v2:172.22.1.4:3300/0,v1:172.22.1.4:6789/0]
EOF

Copy the Bash script to a text editor for use in the next step.

Virtual Machine Configuration

For the remaining steps, open a command shell to the virtual machine that will act as a Ceph client for your application. Once you have an active command shell to the virtual machine, paste the Bash script that was generated from the previous step into the command shell and then execute it with a return as required.

Now the virtual machine should be ready for configuration. Using the same command shell that the previous Bash script was executed in, paste the following script and then execute it with a return as required.

(
MNT_PATH_DEF="/mnt/cephfs"
MNT_PATH=""

create_mount_path () {
  echo "The given path $MNT_PATH does not exist. Would you like to create it automatically? Enter yes or no:"
  read CREATE_PATH
  CREATE_PATH=$(echo "$CREATE_PATH" | xargs)
  if [[ "$CREATE_PATH" == "yes" ]] || [[ "$CREATE_PATH" == "y" ]]; then
    sudo mkdir $MNT_PATH
  else
    clear
    echo "The given path $MNT_PATH was not automatically created so you must specify an existing directory to use!"
    echo ''
    get_mount_path
  fi
}

get_mount_path () {
  echo 'Enter the absolute path of the location you wish to mount the Ceph file system to and then press return ['"$MNT_PATH_DEF"']:'
  read MNT_PATH
  MNT_PATH=$(echo "$MNT_PATH" | xargs)
  if [[ ${#MNT_PATH} -eq 0 ]]; then
    MNT_PATH="$MNT_PATH_DEF"
  fi
  if [[ ! -d $MNT_PATH ]] && [[ ! -f $MNT_PATH ]]; then
    clear
    create_mount_path
  elif [[ -f $MNT_PATH ]]; then
    clear
    echo "The given path $MNT_PATH is not a directory!"
    echo ''
    get_mount_path
  fi
}

clear
get_mount_path

sudo apt update && sudo apt install -y software-properties-common

wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -

echo "deb https://download.ceph.com/debian-pacific/ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/ceph.list > /dev/null

echo "# deb-src https://download.ceph.com/debian-pacific/ $(lsb_release -cs) stable" | sudo tee -a /etc/apt/sources.list.d/ceph.list > /dev/null  

sudo apt update

sudo apt install -y ceph-common

sudo tee /etc/ceph/ceph.client.$CLIENT_NAME.keyring &> /dev/null <<EOF
[client.${CLIENT_NAME}]
    key = ${CLIENT_SECRET}
EOF

sudo tee /etc/ceph/ceph.client.$CLIENT_NAME.keyring.value &> /dev/null <<EOF
${CLIENT_SECRET}
EOF

echo "$MON_IP:$FS_PATH $MNT_PATH ceph name=$CLIENT_NAME,fs=$FS_NAME,recover_session=clean,_netdev 0 2" | sudo tee -a /etc/fstab &> /dev/null

sudo mount $MNT_PATH
)

This script will prompt you for the following information:

• Absolute mount path that should be used for mounting the network file system on the virtual machine.

Once you have provided a valid answer to the previous prompt, the script will proceed to setup the required APT repository, install the required Ceph packages, create the Ceph client credential files, setup an fstab entry, and then mount the configured file system at the given mount path.

If everything executed as expected, you should now find that the Ceph file system is mounted and ready for use!

Ceph offers three types of storage that can all be used in Kubernetes which are object storage (rados), block storage (RBD), and file systems (Ceph FS). This tutorial will focus on the latter two as that is all that is needed to hit the ground running with a number of common scenarios. It’s also worth mentioning that setting up the Ceph object storage system rados is a fairly intensive process that would at least justify it’s own blog post. You are probably already familiar with Ceph’s object storage system rados though as it’s a cross-compatible equivalent of Amazon Web Service’s S3 product. Anywhere you can use AWS’s S3 for storage, you can alternatively use Ceph’s rados gateway to achieve the same outcome.

There are a number of different ways to connect a Kubernetes cluster to Ceph to achieve network based data storage but I will be focusing on a method that uses Rook. This method seems to be the most popular choice currently for ease of use and efficiency. Before you get started, you will need a command shell in an environment that has kubectl available with the appropriate configuration to allow administrative access to the cluster you wish to connect to Ceph. You will also need administrative access to the Ceph cluster which you intend to connect to the Kubernetes cluster. This tutorial will focus on using CLI based access to Ceph in order to complete the steps but if you know what you’re doing, you can achieve some of the same goals using a GUI based approach such as the controls provided in Proxmox Virtualization Environment.

I have not widely tested this implementation with various combinations of Ceph and Rook but you should be aware there is a history of specific releases having known issues that require some manual intervention. My test case is based off of a Ceph 16.2.7 deployment running on top of Proxmox Virtualization Environment 7.1-7 with Rook version 1.8.1.

Let’s start by gathering some data from the Ceph environment that will be needed to connect Kubernetes to the Ceph environment. Execute the following commands in the Ceph environment with a user that has the appropriate permissions to access sensitive Ceph configuration data (such as root) and then copy the output so that it can be executed in your Kubernetes environment:

ROOK_EXTERNAL_FSID=$(ceph fsid)

ROOK_EXTERNAL_CEPH_MON_DATA=$(ceph mon dump -f json 2>/dev/null|jq --raw-output .mons[0].name)=$(ceph mon dump -f json 2>/dev/null|jq --raw-output .mons[0].public_addrs.addrvec[0].addr)

ROOK_EXTERNAL_ADMIN_SECRET=$(ceph auth get-key client.admin)

clear
echo 'export NAMESPACE=rook-ceph-external'
echo 'export ROOK_EXTERNAL_FSID='"$ROOK_EXTERNAL_FSID"
echo 'export ROOK_EXTERNAL_CEPH_MON_DATA='"$ROOK_EXTERNAL_CEPH_MON_DATA"
echo 'export ROOK_EXTERNAL_ADMIN_SECRET='"$ROOK_EXTERNAL_ADMIN_SECRET"

This should produce a result similar to the following:

export NAMESPACE=rook-ceph-external
export ROOK_EXTERNAL_FSID=XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
export ROOK_EXTERNAL_CEPH_MON_DATA=NODE_HOSTNAME=000.000.000.000:3300
export ROOK_EXTERNAL_ADMIN_SECRET=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Copy all four of those lines and stash them in a text file for use later in this tutorial. Now you need to grab a copy of the Rook project files so execute the following commands in your kubectl enabled environment:

git clone -b v1.8.1 https://github.com/rook/rook.git
cd rook/deploy/examples

Now it’s time to load the Kubernetes cluster with all of the Rook / Ceph dependencies needed to connect the two environments. Execute the following command:

kubectl create -f crds.yaml -f common.yaml -f operator.yaml -f common-external.yaml

You can check that all is well and completed by running the following command:

kubectl get pods -n rook-ceph

This command should yield an output similar to the following:

NAME                                 READY   STATUS    RESTARTS   AGE
rook-ceph-operator-b89545b4f-psh7j   1/1     Running   0          2m56s

Now it’s time to setup some security and configuration related objects in Kubernetes to support the connection. Make a copy of the command output from the earlier step that you stashed away and execute those four commands in your Kubernetes environment.

The rest of this step is pretty easy as Rook has provided a bash script to do the work for you by using the values set in the environment variables from the previous step. Execute the following command:

bash import-external-cluster.sh

This next step shouldn’t be necessary but there is still a bit of a lingering oversight in the Rook’s implementation design that results in an issue where a blank Ceph username and secret value are stored into the associated Kubernetes configuration. Execute the following command:

kubectl edit secret rook-ceph-mon -n rook-ceph-external

This should launch a VIM style editor in the CLI with some YAML formatted data visible. You need to delete the following two lines from this view and then save changes and exit:

  ceph-secret: ""
  ceph-username: ""

Now you’re ready to actually connect the Ceph environment with Kubernetes by applying a YAML configuration that contains the CephCluster object. Execute the following command:

kubectl apply -f cluster-external.yaml

This step may take some time to fully complete depending on your environment so you can run the following command to watch for the status:

watch -n 2 kubectl get cephcluster -n rook-ceph-external

When everything is ready and all went well, you should see the “Phase” column value as “Connected”.

Now let’s move on to getting Ceph configured with a new Ceph file system to support the Kubernetes cluster. Of course, you can re-use an existing one if you like to conserve growth capacity given that each Ceph file system requires it’s own metadata server (MDS) and only one MDS can be ran on a Ceph node. Be aware though, if you re-use the same Ceph file system you may have setup for use in a PVE cluster, you need to be extra cautious with actions you take in the future as to not accidentally damage or destroy a mission critical storage system for your PVE cluster.

Update the following list of commands to reflect the name of the CephFS you would like to create or use the appropriate names of an existing CephFS that you would like to re-use for your Kubernetes environment. Once you have updated the list of commands, execute them in your kubectl environment. If you are creating a new CephFS instead of re-using an existing one, execute these commands in your Ceph environment as well.

cephfs_name=k8s-cephfs
cephfs_metadata_pool=k8s-cephfs_metadata
cephfs_data_pool=k8s-cephfs_data

If you are creating a new CephFS instead of re-using an existing one, execute the following commands in your Ceph environment:

ceph osd pool create $cephfs_metadata_pool
ceph osd pool create $cephfs_data_pool
ceph fs new $cephfs_name $cephfs_metadata_pool $cephfs_data_pool

Now that you have a Ceph file system ready for use, it’s time to setup two storage classes in your Kubernetes environment to provide containers with both block and file system storage options. You need to update the storage class templates before you apply them to the Kubernetes environment by executing the following commands:

sed -i 's/clusterID\:\srook-ceph/clusterID\: rook-ceph-external/g' csi/rbd/storageclass.yaml
sed -i 's/clusterID\:\srook-ceph/clusterID\: rook-ceph-external/g' csi/cephfs/storageclass.yaml
sed -i 's/fsName\:\smyfs/fsName\: '"$cephfs_name"'/g' csi/cephfs/storageclass.yaml
sed -i 's/pool\:\smyfs\-replicated/pool\: '"$cephfs_data_pool"'/g' csi/cephfs/storageclass.yaml
sed -i 's/csi.storage.k8s.io\/provisioner-secret-namespace\:\srook-ceph/csi.storage.k8s.io\/provisioner-secret-namespace\: rook-ceph-external/g' csi/cephfs/storageclass.yaml
sed -i 's/csi.storage.k8s.io\/controller-expand-secret-namespace\:\srook-ceph/csi.storage.k8s.io\/controller-expand-secret-namespace\: rook-ceph-external/g' csi/cephfs/storageclass.yaml
sed -i 's/csi.storage.k8s.io\/node-stage-secret-namespace\:\srook-ceph/csi.storage.k8s.io\/node-stage-secret-namespace\: rook-ceph-external/g' csi/cephfs/storageclass.yaml

Now you can apply the storage class templates to the Kubernetes environment by executing the following command:

kubectl create -f csi/rbd/storageclass.yaml -f csi/cephfs/storageclass.yaml

The final step to this process is to set a default storage class for scenarios where a storage class is not specifically defined. I typically choose to use the RBD storage class as I most commonly use that. I use the CephFS storage class for scenarios where data needs to be consumed by more than one container simultaneously such as web application files in a high availability configuration. Execute the following command to set the default storage class to the new Ceph RBD option:

kubectl patch storageclass rook-ceph-block -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

That’s it! You’re now ready to start deploying containers that consume storage directly from your Ceph cluster.

I recommend you check out this post next to take your Kubernetes deployment to the next level.