Ceph Configuration

The first step to this process involves configuring Ceph and extracting relevant information needed to setup your virtual machine client. Open a command shell to one of the hosts running Ceph and elevate your access to a user that has permissions to execute Ceph configuration commands.

Once you have an active command shell to the Ceph server, copy and paste the following script block into the terminal and then execute it with a return as required. This will install the APT package “jq” which is used for extracting some of the configuration settings.

(
sudo apt install -y jq

FS_NAME_DEF="cephfs"
FS_PATH_DEF="/"
FS_PERM_DEF="rw"
CLIENT_NAME_DEF="vm-client"
FS_NAME=""
FS_PATH=""
FS_PERM=""
CLIENT_NAME=""

get_fs_name () {
  clear
  echo 'Enter the name of the Ceph file system you wish to use and then press return ['"$FS_NAME_DEF"']:'
  read FS_NAME
  FS_NAME=$(echo "$FS_NAME" | xargs)
  if [[ ${#FS_NAME} -eq 0 ]]; then
    FS_NAME="$FS_NAME_DEF"
  fi
}

get_fs_path () {
  clear
  echo 'Enter the absolute path within the Ceph file system you wish to use and then press return ['"$FS_PATH_DEF"']:'
  read FS_PATH
  FS_PATH=$(echo "$FS_PATH" | xargs)
  if [[ ${#FS_PATH} -eq 0 ]]; then
    FS_PATH="$FS_PATH_DEF"
  fi
}

get_fs_perm () {
  clear
  echo 'Enter the access permissions for the previously provided path you wish to use and then press return ['"$FS_PERM_DEF"']:'
  read FS_PERM
  FS_PERM=$(echo "$FS_PERM" | xargs)
  if [[ ${#FS_PERM} -eq 0 ]]; then
    FS_PERM="$FS_PERM_DEF"
  fi
}

get_client_name () {
  clear
  echo 'Enter the name of the Ceph client node and then press return ['"$CLIENT_NAME_DEF"']:'
  read CLIENT_NAME
  CLIENT_NAME=$(echo "$CLIENT_NAME" | xargs)
  if [[ ${#CLIENT_NAME} -eq 0 ]]; then
    CLIENT_NAME="$CLIENT_NAME_DEF"
  fi
}

get_fs_name
get_fs_path
get_fs_perm
get_client_name
ceph fs authorize $FS_NAME client.$CLIENT_NAME $FS_PATH $FS_PERM > /dev/null
CLIENT_SECRET=$(ceph auth get-or-create-key client.$CLIENT_NAME)
CEPH_CONF=$(ceph config generate-minimal-conf)
MON_TARGET=$(ceph mon dump -f json 2>/dev/null|jq --raw-output .mons[0].public_addrs.addrvec[0].addr)
readarray -d : -t MON_IP<<<"$MON_TARGET"

clear
echo 'export FS_NAME='"$FS_NAME"
echo 'export FS_PATH='"$FS_PATH"
echo 'export CLIENT_NAME='"$CLIENT_NAME"
echo 'export CLIENT_SECRET='"$CLIENT_SECRET"
echo 'export MON_IP='"$MON_IP"
echo 'sudo tee /etc/ceph/ceph.conf &> /dev/null <<EOF
'"$CEPH_CONF"'
EOF
'
)

This script will prompt you for the following information;

• Ceph file system name that should be used.
• Ceph file system path that should be used. This should be an absolute path within the given file system.
• Ceph file system permissions that should be granted for the previously given file system path. This can be one of “r” for read-only, “w” for write-only, or “rw” for both read and write permissions.
• Ceph client name that should be used. This is what will be used to create a user for Ceph authentication so it should only contain alphanumeric characters as well as underscores and hyphens.

The rest of the required information will be automatically extracted by the script. If everything executes correctly, you should be left with a Bash script that looks something like this:

export FS_NAME=cephfs
export FS_PATH=/vm/docker-stacks
export CLIENT_NAME=docker-node1
export CLIENT_SECRET=AQC4YUhjP+NbOxAAH+PViHPlzP4joKruS3qkXg==
export MON_IP=172.22.1.4
sudo tee /etc/ceph/ceph.conf &> /dev/null <<EOF
# minimal ceph.conf for 20fc650f-25a4-42c6-957c-efb594ce1f01
[global]
	fsid = 20fc650f-25a4-42c6-957c-efb594ce1f01
	mon_host = [v2:172.22.1.2:3300/0,v1:172.22.1.2:6789/0] [v2:172.22.1.3:3300/0,v1:172.22.1.3:6789/0] [v2:172.22.1.4:3300/0,v1:172.22.1.4:6789/0]
EOF

Copy the Bash script to a text editor for use in the next step.

Virtual Machine Configuration

For the remaining steps, open a command shell to the virtual machine that will act as a Ceph client for your application. Once you have an active command shell to the virtual machine, paste the Bash script that was generated from the previous step into the command shell and then execute it with a return as required.

Now the virtual machine should be ready for configuration. Using the same command shell that the previous Bash script was executed in, paste the following script and then execute it with a return as required.

(
MNT_PATH_DEF="/mnt/cephfs"
MNT_PATH=""

create_mount_path () {
  echo "The given path $MNT_PATH does not exist. Would you like to create it automatically? Enter yes or no:"
  read CREATE_PATH
  CREATE_PATH=$(echo "$CREATE_PATH" | xargs)
  if [[ "$CREATE_PATH" == "yes" ]] || [[ "$CREATE_PATH" == "y" ]]; then
    sudo mkdir $MNT_PATH
  else
    clear
    echo "The given path $MNT_PATH was not automatically created so you must specify an existing directory to use!"
    echo ''
    get_mount_path
  fi
}

get_mount_path () {
  echo 'Enter the absolute path of the location you wish to mount the Ceph file system to and then press return ['"$MNT_PATH_DEF"']:'
  read MNT_PATH
  MNT_PATH=$(echo "$MNT_PATH" | xargs)
  if [[ ${#MNT_PATH} -eq 0 ]]; then
    MNT_PATH="$MNT_PATH_DEF"
  fi
  if [[ ! -d $MNT_PATH ]] && [[ ! -f $MNT_PATH ]]; then
    clear
    create_mount_path
  elif [[ -f $MNT_PATH ]]; then
    clear
    echo "The given path $MNT_PATH is not a directory!"
    echo ''
    get_mount_path
  fi
}

clear
get_mount_path

sudo apt update && sudo apt install -y software-properties-common

wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -

echo "deb https://download.ceph.com/debian-pacific/ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/ceph.list > /dev/null

echo "# deb-src https://download.ceph.com/debian-pacific/ $(lsb_release -cs) stable" | sudo tee -a /etc/apt/sources.list.d/ceph.list > /dev/null  

sudo apt update

sudo apt install -y ceph-common

sudo tee /etc/ceph/ceph.client.$CLIENT_NAME.keyring &> /dev/null <<EOF
[client.${CLIENT_NAME}]
    key = ${CLIENT_SECRET}
EOF

sudo tee /etc/ceph/ceph.client.$CLIENT_NAME.keyring.value &> /dev/null <<EOF
${CLIENT_SECRET}
EOF

echo "$MON_IP:$FS_PATH $MNT_PATH ceph name=$CLIENT_NAME,fs=$FS_NAME,recover_session=clean,_netdev 0 2" | sudo tee -a /etc/fstab &> /dev/null

sudo mount $MNT_PATH
)

This script will prompt you for the following information:

• Absolute mount path that should be used for mounting the network file system on the virtual machine.

Once you have provided a valid answer to the previous prompt, the script will proceed to setup the required APT repository, install the required Ceph packages, create the Ceph client credential files, setup an fstab entry, and then mount the configured file system at the given mount path.

If everything executed as expected, you should now find that the Ceph file system is mounted and ready for use!

Enter BGP peering for Kubernetes using Calico CNI. Using this method, you can have all of your Kubernetes nodes communicate directly with an external network router via BGP in order to distribute routes to your cluster services and pods. Once this has been setup, any services you configure (properly) in your cluster will become immediately accessible to your network as routes are injected immediately upon activation.

In order to apply the YAML formatted configuration files needed to setup BGP peering, you will need to have the Calico CNI control script installed in an environment that has proper permissions to manage the cluster. If you used the tutorial I provided in this blog for deploying Kubernetes, then this control script will have already been installed to each of your control plane nodes with the alias “kubectl-calico” and “calicoctl” so you’re already good to get started.

If you setup your Kubernetes cluster some other way and do not currently have the Calico CNI control script installed, execute the following commands from an environment you have setup with kubectl for the cluster:

export calico_version=$(curl --silent "https://api.github.com/repos/projectcalico/calico/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')

curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" -o kubectl-calico -L https://github.com/projectcalico/calico/releases/download/$calico_version/calicoctl-linux-amd64

sudo chown root:root kubectl-calico
sudo chmod +x kubectl-calico
sudo mv kubectl-calico /usr/local/bin/
sudo ln -s /usr/local/bin/kubectl-calico /usr/local/bin/calicoctl

At this point, you should be ready to start building the basic configuration file required to update the Calico CNI in order to establish BGP peering both with your network router and between nodes (route reflection). The example I provide in this tutorial will assume a very basic setup that assumes only one availability zone with a single top-of-rack (ToR) router. If you are building something with even better redundancy, it’s not complicated to pivot from this configuration to one that supports multiple availability zones and/or multiple ToR routers.

Let’s start by applying some additional labels to all of your cluster nodes. These labels will be used to selectively enable route reflection via BGP. Execute the following command for each node in your cluster by updating the “HOSTNAME-HERE” string to match the hostname configured on each node. If you’re unsure of a node’s hostname, just execute the “hostname” command on that node.

kubectl label node HOSTNAME-HERE route-reflector=true

After this has been completed, you can verify that everything looks correct by executing the following command:

kubectl get nodes --show-labels

Now let’s create a YAML formatted configuration file using the following command:

echo 'apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false
  serviceClusterIPs:
  - cidr: 10.10.10.0/23
  serviceExternalIPs:
  - cidr: 10.10.0.0/23
  - cidr: 172.21.0.0/24
  listenPort: 179
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: as-rtr1
spec:
  peerIP: 172.22.100.254
  asNumber: 64512
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: as-node
spec:
  nodeSelector: all()
  peerSelector: route-reflector == \'true\'' | tee /tmp/calico-bgp-configuration.yaml

Before you apply this configuration file to the cluster, some of the values will probably need modified to reflect appropriate settings that match your environment.

The first item to update is the “serviceClusterIPs” entry. This should contain the CIDR that you configured for your service networks CIDR when initializing the cluster. If you followed my tutorial for deploying the Kubernetes cluster initially, all of this should have some familiarity already.

  serviceClusterIPs:
  - cidr: 10.10.10.0/23

The next item to update is the “serviceExternalIPs” entry. This should contain one or more CIDR entries to reflect any IP ranges that you intend to assign external service IP addresses from. For example, if you were going to deploy a DNS resolver for your network with an IP address of “10.10.0.1” then you would add a CIDR entry that includes that particular IP address such as “- cidr: 10.10.0.0/24”. IP addresses assigned to services from the ranges configured in this entry will be announced as individual /32 routes via BGP. This can be very useful if you need to cherry pick addresses in-between existing network allocations or if you need to temporarily “hijack” the traffic of another service on your network. A good example of this would be doing brief testing of new services that will replace existing deployments elsewhere on your network.

  serviceExternalIPs:
  - cidr: 10.10.0.0/23
  - cidr: 172.21.0.0/24

Lastly, you may need to update the “peerIP” and “asNumber” entries to reflect the appropriate IP address and ASN of your ToR router that the cluster nodes should peer to.

spec:
  peerIP: 172.22.100.254
  asNumber: 64512

Now that the configuration file is ready, it’s time to apply it to the cluster. Don’t forget that you will need to update your ToR router with the appropriate configuration to support BGP peering from each of your cluster nodes. The remote ASN used by the cluster nodes in this configuration will be the same as the ASN assigned in the previous section. To apply the configuration file to the cluster, execute the following command:

calicoctl apply -f /tmp/calico-bgp-configuration.yaml

At this point, if your ToR router has already been properly configured for BGP peering to your cluster nodes, you should see the peering relationships begin to establish after no more than one to two minutes typically, often quicker depending on your ToR router’s BGP configuration.

That’s it! Wow, wasn’t that simple? Your Kubernetes cluster is now ready to begin announcing routes for any services and pods that you deploy. Once the relationships have been established, you should see some announcements to cover what is already deployed in the cluster. You will only see /32 announcements for services which have been configured with one or more external IP addresses. None of the default services deployed for a fresh cluster define this setting so don’t expect to see any out of the gate if you have not already configured something this way.

There are some important details to note about this particular configuration (which can be alternated to work slightly differently). With this configuration, the node-to-node mesh networking is disabled which means that traffic for a given service IP will be routed only to the cluster nodes which are actively hosted an instance of the service. This is really handy if any of your services have a need to see the source IP of ingress traffic (such as for security purposes). Depending on your use case, this might not be the most ideal configuration though since additional external configuration would be required on your router to properly balance ingress traffic to each node hosting a service instance.

When you use the node-to-node mesh networking feature, you can direct ingress traffic for your services to any mesh node in the cluster even if the service doesn’t have an instance running on a particular node. If the service isn’t running locally, the node will forward the traffic to a different node that is running the service. This approach is great if you want ingress traffic to be automatically balanced across all available nodes running a particular service. The caveat here is that when using mesh networking, your service may not always see the source IP address of the ingress traffic since the packets will be rewritten with a source IP address that reflects the cluster node that forwarded the traffic via the mesh network.

I recommend you check out this post next to take your Kubernetes deployment to the next level.