Enter BGP peering for Kubernetes using Calico CNI. Using this method, you can have all of your Kubernetes nodes communicate directly with an external network router via BGP in order to distribute routes to your cluster services and pods. Once this has been setup, any services you configure (properly) in your cluster will become immediately accessible to your network as routes are injected immediately upon activation.

In order to apply the YAML formatted configuration files needed to setup BGP peering, you will need to have the Calico CNI control script installed in an environment that has proper permissions to manage the cluster. If you used the tutorial I provided in this blog for deploying Kubernetes, then this control script will have already been installed to each of your control plane nodes with the alias “kubectl-calico” and “calicoctl” so you’re already good to get started.

If you setup your Kubernetes cluster some other way and do not currently have the Calico CNI control script installed, execute the following commands from an environment you have setup with kubectl for the cluster:

export calico_version=$(curl --silent "https://api.github.com/repos/projectcalico/calico/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')

curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" -o kubectl-calico -L https://github.com/projectcalico/calico/releases/download/$calico_version/calicoctl-linux-amd64

sudo chown root:root kubectl-calico
sudo chmod +x kubectl-calico
sudo mv kubectl-calico /usr/local/bin/
sudo ln -s /usr/local/bin/kubectl-calico /usr/local/bin/calicoctl

At this point, you should be ready to start building the basic configuration file required to update the Calico CNI in order to establish BGP peering both with your network router and between nodes (route reflection). The example I provide in this tutorial will assume a very basic setup that assumes only one availability zone with a single top-of-rack (ToR) router. If you are building something with even better redundancy, it’s not complicated to pivot from this configuration to one that supports multiple availability zones and/or multiple ToR routers.

Let’s start by applying some additional labels to all of your cluster nodes. These labels will be used to selectively enable route reflection via BGP. Execute the following command for each node in your cluster by updating the “HOSTNAME-HERE” string to match the hostname configured on each node. If you’re unsure of a node’s hostname, just execute the “hostname” command on that node.

kubectl label node HOSTNAME-HERE route-reflector=true

After this has been completed, you can verify that everything looks correct by executing the following command:

kubectl get nodes --show-labels

Now let’s create a YAML formatted configuration file using the following command:

echo 'apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false
  serviceClusterIPs:
  - cidr: 10.10.10.0/23
  serviceExternalIPs:
  - cidr: 10.10.0.0/23
  - cidr: 172.21.0.0/24
  listenPort: 179
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: as-rtr1
spec:
  peerIP: 172.22.100.254
  asNumber: 64512
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: as-node
spec:
  nodeSelector: all()
  peerSelector: route-reflector == \'true\'' | tee /tmp/calico-bgp-configuration.yaml

Before you apply this configuration file to the cluster, some of the values will probably need modified to reflect appropriate settings that match your environment.

The first item to update is the “serviceClusterIPs” entry. This should contain the CIDR that you configured for your service networks CIDR when initializing the cluster. If you followed my tutorial for deploying the Kubernetes cluster initially, all of this should have some familiarity already.

  serviceClusterIPs:
  - cidr: 10.10.10.0/23

The next item to update is the “serviceExternalIPs” entry. This should contain one or more CIDR entries to reflect any IP ranges that you intend to assign external service IP addresses from. For example, if you were going to deploy a DNS resolver for your network with an IP address of “10.10.0.1” then you would add a CIDR entry that includes that particular IP address such as “- cidr: 10.10.0.0/24”. IP addresses assigned to services from the ranges configured in this entry will be announced as individual /32 routes via BGP. This can be very useful if you need to cherry pick addresses in-between existing network allocations or if you need to temporarily “hijack” the traffic of another service on your network. A good example of this would be doing brief testing of new services that will replace existing deployments elsewhere on your network.

  serviceExternalIPs:
  - cidr: 10.10.0.0/23
  - cidr: 172.21.0.0/24

Lastly, you may need to update the “peerIP” and “asNumber” entries to reflect the appropriate IP address and ASN of your ToR router that the cluster nodes should peer to.

spec:
  peerIP: 172.22.100.254
  asNumber: 64512

Now that the configuration file is ready, it’s time to apply it to the cluster. Don’t forget that you will need to update your ToR router with the appropriate configuration to support BGP peering from each of your cluster nodes. The remote ASN used by the cluster nodes in this configuration will be the same as the ASN assigned in the previous section. To apply the configuration file to the cluster, execute the following command:

calicoctl apply -f /tmp/calico-bgp-configuration.yaml

At this point, if your ToR router has already been properly configured for BGP peering to your cluster nodes, you should see the peering relationships begin to establish after no more than one to two minutes typically, often quicker depending on your ToR router’s BGP configuration.

That’s it! Wow, wasn’t that simple? Your Kubernetes cluster is now ready to begin announcing routes for any services and pods that you deploy. Once the relationships have been established, you should see some announcements to cover what is already deployed in the cluster. You will only see /32 announcements for services which have been configured with one or more external IP addresses. None of the default services deployed for a fresh cluster define this setting so don’t expect to see any out of the gate if you have not already configured something this way.

There are some important details to note about this particular configuration (which can be alternated to work slightly differently). With this configuration, the node-to-node mesh networking is disabled which means that traffic for a given service IP will be routed only to the cluster nodes which are actively hosted an instance of the service. This is really handy if any of your services have a need to see the source IP of ingress traffic (such as for security purposes). Depending on your use case, this might not be the most ideal configuration though since additional external configuration would be required on your router to properly balance ingress traffic to each node hosting a service instance.

When you use the node-to-node mesh networking feature, you can direct ingress traffic for your services to any mesh node in the cluster even if the service doesn’t have an instance running on a particular node. If the service isn’t running locally, the node will forward the traffic to a different node that is running the service. This approach is great if you want ingress traffic to be automatically balanced across all available nodes running a particular service. The caveat here is that when using mesh networking, your service may not always see the source IP address of the ingress traffic since the packets will be rewritten with a source IP address that reflects the cluster node that forwarded the traffic via the mesh network.

I recommend you check out this post next to take your Kubernetes deployment to the next level.

To get started, you need at least one clean Ubuntu 20.04 environment, ideally at least two Ubuntu 20.04 environments to separate your control plane and worker nodes. For high availability environments, you should have at least three control plane nodes and at least two worker nodes. You don’t have to create all the additional nodes at once as adding them later isn’t much of a chore. The instructions that follow will assume you’re working with at least two nodes, one for your control plane, and one for your workload. It is very important that every node you intend to join to the same cluster has a unique hostname configured. It does not matter if two nodes are located in different network search domains, their hostname must specifically be unique to the cluster.

Planning Cluster Configuration

Before you start executing commands, you need to get all your ducks in a row with a few pieces of configuration information. The following is an example of the configuration used in the Azorian Solutions production Kubernetes cluster:

             Cluster Domain: k8s.azorian.solutions
     Control Plane Endpoint: cp.k8s.azorian.solutions
          Pod Networks CIDR: 10.255.0.0/16
      Service Networks CIDR: 10.10.10.0/23
            Kubelet Version: 1.22.0-00
            Kubeadm Version: 1.22.0-00
            Kubectl Version: 1.22.0-00

The first configuration item is the cluster domain. Think of this much like you would a network search domain although it isn’t necessarily used in the same way.

The next configuration item is the control plane endpoint. This is typically a domain name containing an A record corresponding to the IP address of each control plane node that will operate in the cluster. Initially, this domain should only contain a single A record that points to the IP address of the first control plane node that you intend to initialize the Kubernetes cluster on. If you miss this detail during setup, it could lean to painful results forcing you to start the entire process all over again (assuming you don’t have great proficiency in Kubernetes clusters to fix it).

Moving on, the pod networks CIDR should be a fairly large private network allocation as this will be carved up into many smaller networks for each pod that gets deployed to the cluster. Remember, this could easily equate to one network per container deployed depending on how you deploy your services so don’t get stingy on address space. This is not something that is easily changed after the fact. The allocation chosen here should not be your easiest to remember as you likely won’t spend much if any time directly interacting with the IP addresses that are assigned from these networks. It’s also important to remember that the addresses automatically assigned from this network are ephemeral in nature in accordance with the destruction and recreation of pods that happen automatically from time to time.

Next up is the service networks CIDR. Until you get familiar with exactly how Kubernetes works, I recommend starting with a /23 private network as this will ensure you don’t likely end up in a situation where you run out of space from deploying services to the cluster. Services in Kubernetes are like a public facing gateway to your various pods (which contain one or more containers). You can think of a service similar to how you would think of a load balancer. It will receive communication requests to specific IP addresses and ports which are then automatically routed to any pods that have been linked to the service.

Finally, you have your Kubelet, Kubeadm, and Kubectl versions. You can select any of the available Kubernetes release versions here. It is considered best practice for compatibility reasons to keep all three of these packages in sync with at least major and minor versions. If you know what you’re doing, you can install different versions of each but you may experience problems by doing so if there is a breaking change between two selected versions. Version 1.23 hasn’t been out long and I haven’t tested it so for this tutorial, we’ll stick with version 1.22.0-00 which is what I am running today.

Preparing The Nodes

Let’s start by running a number of commands to get each node of the Kubernetes cluster prepared for deployment. Begin by opening up a command shell to each of the nodes with a non-root user that has super user (sudo) privileges.

First, you need to disable the use of swap partitions. Execute the following commands on each node:

sudo sed -i 's/^\/swap/#\/swap/g' /etc/fstab
sudo swapoff -a

Next, you need to update the OS to the latest packages for the current distribution. Execute the following commands on each node:

sudo apt update
sudo apt upgrade -y

Now it’s time to install some basic dependencies to support the remainder of the installation process. Execute the following command on each node:

sudo apt install -y apt-transport-https ca-certificates curl gnupg lsb-release net-tools containerd

Now you need to load the “overlay” and “br_netfilter” Linux kernel modules. Execute the following commands on each node:

sudo modprobe overlay
sudo modprobe br_netfilter

The previous commands only load kernel modules temporarily. You need to add a configuration file to ensure that these modules are loaded after each machine reboot. Execute the following command on each node:

echo 'overlay
br_netfilter' | sudo tee /etc/modules-load.d/k8s.conf > /dev/null

Now you need to add a kernel configuration file to tweak three networking settings. Execute the following command on each node:

echo 'net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1' | sudo tee /etc/sysctl.d/99-k8s.conf > /dev/null

The first of the three aforementioned settings instruct the kernel to accept and forward IP packets not intended for the host machine, but instead one or more of the guests running on the host. The second and third settings control whether or not packets passing through a Linux networking bridge are passed to iptables for processing.

Now you need to instruct the kernel to load the new kernel configuration file to apply the changes. Execute the following command on each node:

sudo sysctl --system

Now you need to install the GPG key for the Google APT repository that contains the Kubernetes packages you’ll need. Execute the following command on each node:

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

Now you need to add a configuration file for the APT package manager to make the system aware of the Kubernetes package repository hosted by Google. Execute the following command on each node:

echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list > /dev/null

Now you need to update APT’s package cache to download the latest package information from the Kubernetes repository. Execute the following command on each node:

sudo apt update

Now you need to set some temporary environment variables that will be used during the Kubernetes package installation process. Execute the following commands on each node, taking care to update each variable with the appropriate versions that you chose for Kubelet, Kubeadm, and Kubectl at the beginning of this tutorial:

kubelet_version=1.22.0-00
kubeadm_version=1.22.0-00
kubectl_version=1.22.0-00

Now you are finally ready to install the Kubernetes packages. Execute the following commands on each node:

KUBELET_EXTRA_ARGS='--feature-gates="AllAlpha=false,RunAsGroup=true"
  --container-runtime=remote --cgroup-driver=systemd
  --runtime-request-timeout=5m
  --container-runtime-endpoint="unix:///var/run/containerd/containerd.sock"'

sudo apt install -y --allow-change-held-packages \
  kubelet=$kubelet_version kubeadm=$kubeadm_version \
  kubectl=$kubectl_version kubernetes-cni

Now there is one final dependency to install which is the control script for the Calico CNI since this tutorial will be making use of Calico as the container networking interface (CNI). Execute the following commands on each node:

export calico_version=$(curl --silent "https://api.github.com/repos/projectcalico/calico/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')

curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" -o kubectl-calico -L https://github.com/projectcalico/calico/releases/download/$calico_version/calicoctl-linux-amd64

sudo chown root:root kubectl-calico
sudo chmod +x kubectl-calico
sudo mv kubectl-calico /usr/local/bin/
sudo ln -s /usr/local/bin/kubectl-calico /usr/local/bin/calicoctl

Alright, that completes the node preparation phase of the deployment! I promise the next phase will be much quicker.

Initializing The Cluster

It’s finally time to initialize the Kubernetes cluster now that you have all of the nodes in a prepared state. If you are running the nodes in virtualization environment like Proxmox or ESXI, you may want to consider doing a quick snapshot of each node just in case something doesn’t go right with the initialization process. This will give you a relatively simple way to revert the coming changes and try again without having to completely rebuild the cluster nodes all over again.

Let’s start by setting some additional temporary environment variables to assist with building a cluster configuration file. Execute the following commands on the first control plane node only! Take care to update each variable to the corresponding configuration value chosen at the beginning of this tutorial.

cluster_domain=k8s.azorian.solutions
control_plane_endpoint=cp.k8s.azorian.solutions
pod_networks_cidr=10.255.0.0/16
service_networks_cidr=10.10.10.0/23

Now you need to create a cluster initialization file in YAML format. Execute the following command on the first control plane node only!

echo 'kind: InitConfiguration
apiVersion: kubeadm.k8s.io/v1beta2
nodeRegistration:
  criSocket: /var/run/containerd/containerd.sock
---
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta2
controlPlaneEndpoint: '"$control_plane_endpoint"':6443
networking:
  dnsDomain: '"$cluster_domain"'
  podSubnet: '"$pod_networks_cidr"'
  serviceSubnet: '"$service_networks_cidr"'
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
---
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1' | tee /tmp/kubeadm-config.yaml

Now you can finally initialize the first node of the cluster! Execute the following command on the first control plane node only!

sudo kubeadm init --config /tmp/kubeadm-config.yaml --upload-certs

If all goes well with the cluster initialization, then the aforementioned command should produce some output that provides two different commands beginning with “kubeadm join” that you need to capture for use a little later in this tutorial.

Now you need to copy the admin security configuration file of the cluster to the appropriate location in the current user’s home directory so that you can execute various cluster administration commands as the current user without the user of super user (sudo) privileges. Execute the following command on the first control plane node only!

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You now have a bare bones Kubernetes cluster! You aren’t done quiet yet though as this cluster doesn’t have any container networking interface. Read on.

Installing Calico CNI

This step should prove to be the easiest. Execute the following command on the first control plane node only!

curl https://docs.projectcalico.org/manifests/calico.yaml | kubectl apply -f -

Alright, now the cluster has a container network interface installed and is ready for use if you are only working with a single node. Otherwise, read on.

Initializing The Remaining Nodes

If you are setting up more than one node for your Kubernetes environment, then you need to execute the “kubeadm join” commands that you captured during cluster initialization in an earlier step. For each of the worker nodes you are going to join to the cluster, execute the shorter of the two commands (the one that does not contain the –control-plane parameter) on the node.

If you are setting up more than one control plane nodes for your Kubernetes environment, then you need to execute the longer of the two commands (the one that does contain the –control-plane parameter) on each remaining control plane nodes. Additionally, for ease of operation, also execute the following commands on each of the additional control plane nodes:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

There is one final step to ensure that the Kubernetes cluster is ready to start deploying components. Each node that will be handling workloads needs to have a Kubernetes worker role label attached. Run the following command from a control plane node for each worker node in the cluster taking care to substitute the “HOSTNAME-HERE” string with the node’s configured hostname. If you aren’t sure of the node’s hostname, you can determine this by executing the “hostname” command on the node in question.

kubectl label node HOSTNAME-HERE node-role.kubernetes.io/worker=worker

That’s it! You now have an operational Kubernetes cluster environment. I don’t want to toot your horn, but you’re kind of a big deal now.

I recommend you check out this post next to take your Kubernetes deployment to the next level.