Given that the general focus of many of my blog posts will be containerization using OCI compliant images, I am not going to go into great detail about the LXC system but some of the benefits I outline about OCI containers will also be true for LXC.

So why bother with containers anyways when you have perfectly great environments for running virtual machines? Well, it seems like the list of reasons is endless but I will try to cover many of the reasons I know and find relevant to most of what I do. So let’s jump right in to it and I will explain many of the high level areas of benefits.

Resource Efficiency

If you have ever created a number of virtual machines and then taken notice to their idle resource utilization before you have configured them for their particular roles, you may have noticed that the utilization is not exactly tiny for a machine that isn’t really doing anything. This is because the Linux kernel while very efficient compared to that POS Windows, is still a bit heavy once loaded into memory. This isn’t a bad thing since after all, it offers some pretty amazing abilities. This is however often wasteful in a number of environments that don’t really need to have a large number of Linux kernel copies running just to service one or two primary processes per virtual machine.

There are some scenarios that justify the use of separate kernels still for truly mission critical security guarantees such as those maintained by nation states. This isn’t to say that the Linux kernel is insecure as it provides all of the appropriate isolation facilities for containers to run fully isolated from one another. However, humans make mistakes and every once in awhile, a security bug may come along that could put this isolation at risk. This is exactly why every application should be considered on a case by case basis. If you’re just running some everyday applications like database servers, DNS servers, and others alike that don’t hold truly sensitive data (like the kind that could contribute to the loss of millions of dollars or human life) then you really don’t need to achieve paranoid level security policies.

This is where containers really shine in a noticeable way as the overhead for a container is more or less just the overhead of the process or processes that it runs as opposed to creating entire copies of the Linux kernel. Instead, containers on a container host simply share the resources of the host machine through the user of kernel cgroups and namespaces. Namespaces allow for the creation of virtual hardware in the container and act as a sort of gateway to the underlying host hardware that is to be shared between containers. Kernel control groups (or cgroups for short) are used to apply limits, prioritization, accounting and control of various system resources such as memory, CPU, and file systems. You can can read more about Kernel cgroups here.

Portability

Containers provide a really effective means of packaging applications for distribution and hosting. If you have ever managed traditional bare metal or virtual machine based servers back in the day, you have almost certainly ran into scenarios where you were trying to deploy multiple applications to the same machine only to find out the hard way that the applications depend on different versions of a shared package. A good example of this would be a web application that requires a newer version of a language interpreter like PHP but the other application might be fairly outdated and require a much older version to work. This can be a real PITA to make work on the same machine in a clean way that doesn’t make maintaining ongoing updates a real nightmare.

Containers solve this problem by creating an isolated environment for each process as desired which means you can now run an arbitrary amount of applications on the same host machine, all requiring a different version of a common dependency without any concern. Furthermore, this design also isolates the package updates of one packaged application from another that might be loosely coupled together. An example of this could be a web application that also has a separate application to provide an API.

Deployment

Another great thing that containers solve is the ability to quickly roll out additional copies of an application for testing or updates. Since every container can maintain it’s own entire set of dependencies, this means you can create multiple copies of the same application with different versions of dependencies installed. This is incredibly useful for mission critical processes that demand zero downtime. With this model, you can easily roll out a copy of an application with the latest updates for dependencies to test.

I know what some of you are thinking at this point, “I could do that before with virtual machines through the use of snapshots and cloning.” You are correct about that but the speed and efficiency at which that can be done has been rivaled by the equivalent process in containers. Depending on the operational requirements of the application, you may not be able to keep both versions actively running side by side thus forcing you to do a form of fast swapping. This process becomes much slower if you’re dealing in the speed of VM snapshot restoration or rebooting.

With containers, since you don’t have the time it takes to load the entire kernel and all supporting startup processes, you can move immediately to starting the actual primary process from the moment you say go. This is great if you have a scenario where a database is too big to run multiple copies for testing but an application that relies on that database would be destructive if two copies were using the same database instance (such as network monitoring for example).

Scalability

Since the general design intention of a container is to one a single primary process such as a database server, an HTTP server, or maybe an SSH server as examples, this by design lends itself very well to the modern micro-service architecture that has taken hold in the application development world. A micro-service architecture is seemingly the most ideal design choice for most environments today because not only does it better isolate application component failures thereby limiting the effects of an outage, it also makes it much easier to scale applications more efficiently.

Consider the design of the original eBay architecture. It followed the old school principal of monolithic design where you had one ginormous application that couldn’t be ran on multiple machines very well. This meant that as user demand grew, the underlying hardware running the application had to grow as well. At different points in history, this quickly became a problem because hardware has only ever been so powerful at any given time. If the best CPU at the time only offered 8 cores and the best motherboards could only handle 4 CPUs, then you have quickly found a limit to just how far that application can scale. This scenario illustrated one of the many great examples why micro-service architecture had to become the new approach moving forward. It was the only realistic way to eventually to to the scale that some websites today run at such as eBay, Facebook, Twitter, Instagram, etc…

Possibly to your surprise, just about every major tech company today is making use of not only micro-service architecture but containerization as well. Examples include all of the major video streaming companies like Netflix, Hulu, Amazon Prime, etc.. as well as some of the biggest telecommunications providers in the world like Comcast, Cox, Charter, Verizon, AT&T, etc… This new model of application design has really changed the landscape for efficient scalability.

Even before containers, large organizations would use automation to grow and shrink the underlying resources of applications to handle spikes and valleys in user demand. Imagine having an application that would see a 5000% traffic growth at roughly the same time every day on a daily basis. Think about people returning home from work and turning on the latest episode of a popular series that just came out that day. Before containers, this meant that a lot of virtual machines would be created and/or started simultaneously to meet the predictable demand of a hungry user base. Starting that many VMs at one time can really put some strain on infrastructure, especially the underlying storage systems. Containers have really curved that strain widely since there is far less required to be retrieved from an underlying storage system during process start-up. This means creating 1,000 copies of a process is far less demanding on physical hardware than creating 1,000 VMs to run that single process.

Networking

The advent of containerization also provided the ability to take new, more unique approaches to solving a number of networking challenges that many large scale applications like Facebook and others have encountered inside of data centers. You can run all the fiber and switches you want, but depending on how your application is designed, you can still create massive traffic jams across data center networks. Since container orchestration quickly became much more advanced to meet the growing engineering demands of containerized applications, this paved the road for some new approaches to mitigating these types of traffic jams.

Traditionally, it wouldn’t be uncommon to see data center designs that kept all of the HTTP services on one cluster of servers and all of the database services on a totally different set of servers. Naturally, this meant there often had to be a very hefty fiber optic and switch network to interconnect the clusters. This can quickly create other scalability issues where throwing money at the problem simply won’t resolve the issues.

Enter a more modern approach where now a company might use advanced container orchestration to ensure that anytime a copy of an HTTP service is deployed to a physical host, that a copy of the supporting database server also gets deployed on the same physical machine along side the HTTP service. This means that requests served by the HTTP service no longer require the network communication to the database service to leave the physical machine and traverse an expensive and hard to maintain fiber optic network.

In Closing

I could go on with many other use case examples and detailed explanations but I imagine you get the general idea. Just as when the advent of virtual machines happened and took the computing world by storm, containers have done the same thing thereby taking us all on another huge leap into the future of large scale computing. I realize that many of my examples were based on large scale deployments of huge applications but it’s important to remember that many of these same benefits translate to smaller organizations as well. After all, automation is automation no matter what the scale. If a human doesn’t have to handle the task or a task is less error prone, then there is still a quantifiable value proposition.

Enter the Nginx Ingress Controller for Kubernetes. This is not to be confused for the Kubernetes Ingress Controller that also uses Nginx. The outcomes provided by the two are very similar but project goals different. For this tutorial and many others on this blog, I will focus on the former of the two as that is what I run in my production environments. It is also worth mentioning that this option paves the road for a very useful integration that solves another need that is just as common as the need for simple virtual hosts. That integration would be automatic SSL certificate registration and deployment using ACME providers like LetsEncrypt. I have also provided a tutorial on this blog to cover the implementation of that solution as well.

This tutorial is pretty simple so let’s jump right into it. Working from a kubectl enabled environment for your Kubernetes cluster, execute the following commands:

git clone -b v2.1.1 https://github.com/nginxinc/kubernetes-ingress/
cd kubernetes-ingress/deployments

The first change you should make is to apply the role based access control (RBAC) configurations to the Kubernetes environment by executing the following command:

kubectl apply -f common/ns-and-sa.yaml -f rbac/rbac.yaml

Now you can setup all of the definitions and dependencies required to run the Nginx Ingress Controller by executing the following commands:

kubectl apply -f common/default-server-secret.yaml
kubectl apply -f common/nginx-config.yaml
kubectl apply -f common/ingress-class.yaml
kubectl apply -f common/crds/k8s.nginx.org_virtualservers.yaml
kubectl apply -f common/crds/k8s.nginx.org_virtualserverroutes.yaml
kubectl apply -f common/crds/k8s.nginx.org_transportservers.yaml
kubectl apply -f common/crds/k8s.nginx.org_policies.yaml
kubectl apply -f common/crds/k8s.nginx.org_globalconfigurations.yaml

Now you’re ready to deploy the actual ingress controller. There are two general approaches for deployment which are DaemonSets and Deployments. In this tutorial, the latter will be used as it is likely most sufficient for your environment. Execute the following command:L

kubectl apply -f deployment/nginx-ingress.yaml

Now for the final step which is to expose access to the ingress controller via a Kubernetes service. Let’s take a look at the default service configuration which can be found in the service/loadbalancer.yaml file:

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  - port: 443
    targetPort: 443
    protocol: TCP
    name: https
  selector:
    app: nginx-ingress

This file will work as is to get you started but I will assume you probably want to expose some of the services connected to the ingress controller via a public IP address in your network. If you do not need to expose the ingress controller via a public IP address then you may skip the next section and jump to the last part of this step.

This approach assumes that you have connected your Kubernetes environment into your network via BGP with Calico CNI. Take a look at the modified service/loadbalancer.yaml file below to see how easy it is to add one or more additional IP addresses to the controller which will be automatically announced as a /32 route via BGP. Keep in mind, the IP addresses used here must be defined in your Calico CNI BGP configuration under the “serviceExternalIPs” section or the address assignments will not be announced via BGP.

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  externalIPs:
  - 123.45.54.321
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  - port: 443
    targetPort: 443
    protocol: TCP
    name: https
  selector:
    app: nginx-ingress

Once you have the service/loadbalancer.yaml file modified to your requirements, execute the following command to create the service:

kubectl apply -f service/loadbalancer.yaml

Now that this has been setup, you can easily connect the Kubernetes services you deploy to the ingress controller by defining a configuration object in your application namespace similar to the following:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: http-lb
  namespace: website-namespace
spec:
  rules:
  - host: your.website.domain
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: http-svc
            port:
              number: 80

That’s it! Hopefully that was as simple as I made it out to be!

I recommend you check out this post next to take your Kubernetes deployment to the next level.

Enter Cert-Manager for Kubernetes. This is one of a number of projects that have provided a simple and reliable solution for bridging the gap in automation left by this transition. You should find that this tutorial will be rather short and sweet to achieving this goal. You do however need to make sure you have already implemented a proper ingress controller as this is a key dependency of making all of this work. If you haven’t done so yet, check out my tutorial on deploying the Nginx Ingress Controller for Kubernetes as that will get you where you need to be to make use of this.

Let’s jump right into it as this will go very quickly. Execute the following commands in a kubectl enabled environment to install Cert-Manager to your Kubernetes environment:

cm_version=$(curl --silent "https://api.github.com/repos/cert-manager/cert-manager/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/$cm_version/cert-manager.yaml

Now you need to create some ACME providers for Cert-Manager to use for SSL registration. For this tutorial, I will be focusing on the use of LetsEncrypt given it’s popularity. Execute the following commands after updating the “cm_email” value to an email address that you would like to receive LetsEncrypt expiration notifications at:

cm_email=your-alias@your-company.domain

echo 'apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    email: '"$cm_email"'
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-production-account-key
    solvers:
    - http01:
        ingress:
          class: nginx' | kubectl apply -f -

echo 'apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: '"$cm_email"'
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    solvers:
    - http01:
        ingress:
          class: nginx' | kubectl apply -f -

That’s it! Now you’re ready to start deploying services and ingress controllers that receive automatic SSL registration. The previous commands will setup configuration for both the LetsEncrypt production and staging environments so you can test configurations before pounding the production instance with bad requests.

Let’s take a look at what a simple scenario using the Cert-Manager looks like to obtain a temporary SSL certificate from the LetsEncrypt staging environment. If you want to setup an ingress object to use the LetsEncrypt production environment, you just need to change the “letsencrypt-staging” string below to “letsencrypt” and update the object. I recommend you start out always using the staging environment first until you get the hang of the whole process through repetition.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: http-lb
  namespace: website-namespace
  annotations:
    kubernetes.io/tls-acme: "true"
    cert-manager.io/cluster-issuer: "letsencrypt-staging"
    cert-manager.io/issue-temporary-certificate: "true"
    acme.cert-manager.io/http01-edit-in-place: "true"
spec:
  rules:
  - host: your.website.domain
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: http-svc
            port:
              number: 80
  tls:
  - hosts:
    - your.website.domain
    secretName: http-lb-cert

It’s very important to remember that you need to have the DNS configuration complete and propagated before pushing one of these configurations to the Kubernetes environment. If you don’t then there is a good chance you will cause a negative query hit for the domain you’re trying to register SSL certificates for and that will take longer to clear out of the internal DNS cache. Thankfully, Cert-Manager has proper local verification features that will prevent the request from going to LetsEncrypt before all pieces are in place but that will not mitigate the aforementioned delay should you get ahead of yourself.

Ceph offers three types of storage that can all be used in Kubernetes which are object storage (rados), block storage (RBD), and file systems (Ceph FS). This tutorial will focus on the latter two as that is all that is needed to hit the ground running with a number of common scenarios. It’s also worth mentioning that setting up the Ceph object storage system rados is a fairly intensive process that would at least justify it’s own blog post. You are probably already familiar with Ceph’s object storage system rados though as it’s a cross-compatible equivalent of Amazon Web Service’s S3 product. Anywhere you can use AWS’s S3 for storage, you can alternatively use Ceph’s rados gateway to achieve the same outcome.

There are a number of different ways to connect a Kubernetes cluster to Ceph to achieve network based data storage but I will be focusing on a method that uses Rook. This method seems to be the most popular choice currently for ease of use and efficiency. Before you get started, you will need a command shell in an environment that has kubectl available with the appropriate configuration to allow administrative access to the cluster you wish to connect to Ceph. You will also need administrative access to the Ceph cluster which you intend to connect to the Kubernetes cluster. This tutorial will focus on using CLI based access to Ceph in order to complete the steps but if you know what you’re doing, you can achieve some of the same goals using a GUI based approach such as the controls provided in Proxmox Virtualization Environment.

I have not widely tested this implementation with various combinations of Ceph and Rook but you should be aware there is a history of specific releases having known issues that require some manual intervention. My test case is based off of a Ceph 16.2.7 deployment running on top of Proxmox Virtualization Environment 7.1-7 with Rook version 1.8.1.

Let’s start by gathering some data from the Ceph environment that will be needed to connect Kubernetes to the Ceph environment. Execute the following commands in the Ceph environment with a user that has the appropriate permissions to access sensitive Ceph configuration data (such as root) and then copy the output so that it can be executed in your Kubernetes environment:

ROOK_EXTERNAL_FSID=$(ceph fsid)

ROOK_EXTERNAL_CEPH_MON_DATA=$(ceph mon dump -f json 2>/dev/null|jq --raw-output .mons[0].name)=$(ceph mon dump -f json 2>/dev/null|jq --raw-output .mons[0].public_addrs.addrvec[0].addr)

ROOK_EXTERNAL_ADMIN_SECRET=$(ceph auth get-key client.admin)

clear
echo 'export NAMESPACE=rook-ceph-external'
echo 'export ROOK_EXTERNAL_FSID='"$ROOK_EXTERNAL_FSID"
echo 'export ROOK_EXTERNAL_CEPH_MON_DATA='"$ROOK_EXTERNAL_CEPH_MON_DATA"
echo 'export ROOK_EXTERNAL_ADMIN_SECRET='"$ROOK_EXTERNAL_ADMIN_SECRET"

This should produce a result similar to the following:

export NAMESPACE=rook-ceph-external
export ROOK_EXTERNAL_FSID=XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
export ROOK_EXTERNAL_CEPH_MON_DATA=NODE_HOSTNAME=000.000.000.000:3300
export ROOK_EXTERNAL_ADMIN_SECRET=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Copy all four of those lines and stash them in a text file for use later in this tutorial. Now you need to grab a copy of the Rook project files so execute the following commands in your kubectl enabled environment:

git clone -b v1.8.1 https://github.com/rook/rook.git
cd rook/deploy/examples

Now it’s time to load the Kubernetes cluster with all of the Rook / Ceph dependencies needed to connect the two environments. Execute the following command:

kubectl create -f crds.yaml -f common.yaml -f operator.yaml -f common-external.yaml

You can check that all is well and completed by running the following command:

kubectl get pods -n rook-ceph

This command should yield an output similar to the following:

NAME                                 READY   STATUS    RESTARTS   AGE
rook-ceph-operator-b89545b4f-psh7j   1/1     Running   0          2m56s

Now it’s time to setup some security and configuration related objects in Kubernetes to support the connection. Make a copy of the command output from the earlier step that you stashed away and execute those four commands in your Kubernetes environment.

The rest of this step is pretty easy as Rook has provided a bash script to do the work for you by using the values set in the environment variables from the previous step. Execute the following command:

bash import-external-cluster.sh

This next step shouldn’t be necessary but there is still a bit of a lingering oversight in the Rook’s implementation design that results in an issue where a blank Ceph username and secret value are stored into the associated Kubernetes configuration. Execute the following command:

kubectl edit secret rook-ceph-mon -n rook-ceph-external

This should launch a VIM style editor in the CLI with some YAML formatted data visible. You need to delete the following two lines from this view and then save changes and exit:

  ceph-secret: ""
  ceph-username: ""

Now you’re ready to actually connect the Ceph environment with Kubernetes by applying a YAML configuration that contains the CephCluster object. Execute the following command:

kubectl apply -f cluster-external.yaml

This step may take some time to fully complete depending on your environment so you can run the following command to watch for the status:

watch -n 2 kubectl get cephcluster -n rook-ceph-external

When everything is ready and all went well, you should see the “Phase” column value as “Connected”.

Now let’s move on to getting Ceph configured with a new Ceph file system to support the Kubernetes cluster. Of course, you can re-use an existing one if you like to conserve growth capacity given that each Ceph file system requires it’s own metadata server (MDS) and only one MDS can be ran on a Ceph node. Be aware though, if you re-use the same Ceph file system you may have setup for use in a PVE cluster, you need to be extra cautious with actions you take in the future as to not accidentally damage or destroy a mission critical storage system for your PVE cluster.

Update the following list of commands to reflect the name of the CephFS you would like to create or use the appropriate names of an existing CephFS that you would like to re-use for your Kubernetes environment. Once you have updated the list of commands, execute them in your kubectl environment. If you are creating a new CephFS instead of re-using an existing one, execute these commands in your Ceph environment as well.

cephfs_name=k8s-cephfs
cephfs_metadata_pool=k8s-cephfs_metadata
cephfs_data_pool=k8s-cephfs_data

If you are creating a new CephFS instead of re-using an existing one, execute the following commands in your Ceph environment:

ceph osd pool create $cephfs_metadata_pool
ceph osd pool create $cephfs_data_pool
ceph fs new $cephfs_name $cephfs_metadata_pool $cephfs_data_pool

Now that you have a Ceph file system ready for use, it’s time to setup two storage classes in your Kubernetes environment to provide containers with both block and file system storage options. You need to update the storage class templates before you apply them to the Kubernetes environment by executing the following commands:

sed -i 's/clusterID\:\srook-ceph/clusterID\: rook-ceph-external/g' csi/rbd/storageclass.yaml
sed -i 's/clusterID\:\srook-ceph/clusterID\: rook-ceph-external/g' csi/cephfs/storageclass.yaml
sed -i 's/fsName\:\smyfs/fsName\: '"$cephfs_name"'/g' csi/cephfs/storageclass.yaml
sed -i 's/pool\:\smyfs\-replicated/pool\: '"$cephfs_data_pool"'/g' csi/cephfs/storageclass.yaml
sed -i 's/csi.storage.k8s.io\/provisioner-secret-namespace\:\srook-ceph/csi.storage.k8s.io\/provisioner-secret-namespace\: rook-ceph-external/g' csi/cephfs/storageclass.yaml
sed -i 's/csi.storage.k8s.io\/controller-expand-secret-namespace\:\srook-ceph/csi.storage.k8s.io\/controller-expand-secret-namespace\: rook-ceph-external/g' csi/cephfs/storageclass.yaml
sed -i 's/csi.storage.k8s.io\/node-stage-secret-namespace\:\srook-ceph/csi.storage.k8s.io\/node-stage-secret-namespace\: rook-ceph-external/g' csi/cephfs/storageclass.yaml

Now you can apply the storage class templates to the Kubernetes environment by executing the following command:

kubectl create -f csi/rbd/storageclass.yaml -f csi/cephfs/storageclass.yaml

The final step to this process is to set a default storage class for scenarios where a storage class is not specifically defined. I typically choose to use the RBD storage class as I most commonly use that. I use the CephFS storage class for scenarios where data needs to be consumed by more than one container simultaneously such as web application files in a high availability configuration. Execute the following command to set the default storage class to the new Ceph RBD option:

kubectl patch storageclass rook-ceph-block -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

That’s it! You’re now ready to start deploying containers that consume storage directly from your Ceph cluster.

I recommend you check out this post next to take your Kubernetes deployment to the next level.

Enter BGP peering for Kubernetes using Calico CNI. Using this method, you can have all of your Kubernetes nodes communicate directly with an external network router via BGP in order to distribute routes to your cluster services and pods. Once this has been setup, any services you configure (properly) in your cluster will become immediately accessible to your network as routes are injected immediately upon activation.

In order to apply the YAML formatted configuration files needed to setup BGP peering, you will need to have the Calico CNI control script installed in an environment that has proper permissions to manage the cluster. If you used the tutorial I provided in this blog for deploying Kubernetes, then this control script will have already been installed to each of your control plane nodes with the alias “kubectl-calico” and “calicoctl” so you’re already good to get started.

If you setup your Kubernetes cluster some other way and do not currently have the Calico CNI control script installed, execute the following commands from an environment you have setup with kubectl for the cluster:

export calico_version=$(curl --silent "https://api.github.com/repos/projectcalico/calico/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')

curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" -o kubectl-calico -L https://github.com/projectcalico/calico/releases/download/$calico_version/calicoctl-linux-amd64

sudo chown root:root kubectl-calico
sudo chmod +x kubectl-calico
sudo mv kubectl-calico /usr/local/bin/
sudo ln -s /usr/local/bin/kubectl-calico /usr/local/bin/calicoctl

At this point, you should be ready to start building the basic configuration file required to update the Calico CNI in order to establish BGP peering both with your network router and between nodes (route reflection). The example I provide in this tutorial will assume a very basic setup that assumes only one availability zone with a single top-of-rack (ToR) router. If you are building something with even better redundancy, it’s not complicated to pivot from this configuration to one that supports multiple availability zones and/or multiple ToR routers.

Let’s start by applying some additional labels to all of your cluster nodes. These labels will be used to selectively enable route reflection via BGP. Execute the following command for each node in your cluster by updating the “HOSTNAME-HERE” string to match the hostname configured on each node. If you’re unsure of a node’s hostname, just execute the “hostname” command on that node.

kubectl label node HOSTNAME-HERE route-reflector=true

After this has been completed, you can verify that everything looks correct by executing the following command:

kubectl get nodes --show-labels

Now let’s create a YAML formatted configuration file using the following command:

echo 'apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false
  serviceClusterIPs:
  - cidr: 10.10.10.0/23
  serviceExternalIPs:
  - cidr: 10.10.0.0/23
  - cidr: 172.21.0.0/24
  listenPort: 179
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: as-rtr1
spec:
  peerIP: 172.22.100.254
  asNumber: 64512
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: as-node
spec:
  nodeSelector: all()
  peerSelector: route-reflector == \'true\'' | tee /tmp/calico-bgp-configuration.yaml

Before you apply this configuration file to the cluster, some of the values will probably need modified to reflect appropriate settings that match your environment.

The first item to update is the “serviceClusterIPs” entry. This should contain the CIDR that you configured for your service networks CIDR when initializing the cluster. If you followed my tutorial for deploying the Kubernetes cluster initially, all of this should have some familiarity already.

  serviceClusterIPs:
  - cidr: 10.10.10.0/23

The next item to update is the “serviceExternalIPs” entry. This should contain one or more CIDR entries to reflect any IP ranges that you intend to assign external service IP addresses from. For example, if you were going to deploy a DNS resolver for your network with an IP address of “10.10.0.1” then you would add a CIDR entry that includes that particular IP address such as “- cidr: 10.10.0.0/24”. IP addresses assigned to services from the ranges configured in this entry will be announced as individual /32 routes via BGP. This can be very useful if you need to cherry pick addresses in-between existing network allocations or if you need to temporarily “hijack” the traffic of another service on your network. A good example of this would be doing brief testing of new services that will replace existing deployments elsewhere on your network.

  serviceExternalIPs:
  - cidr: 10.10.0.0/23
  - cidr: 172.21.0.0/24

Lastly, you may need to update the “peerIP” and “asNumber” entries to reflect the appropriate IP address and ASN of your ToR router that the cluster nodes should peer to.

spec:
  peerIP: 172.22.100.254
  asNumber: 64512

Now that the configuration file is ready, it’s time to apply it to the cluster. Don’t forget that you will need to update your ToR router with the appropriate configuration to support BGP peering from each of your cluster nodes. The remote ASN used by the cluster nodes in this configuration will be the same as the ASN assigned in the previous section. To apply the configuration file to the cluster, execute the following command:

calicoctl apply -f /tmp/calico-bgp-configuration.yaml

At this point, if your ToR router has already been properly configured for BGP peering to your cluster nodes, you should see the peering relationships begin to establish after no more than one to two minutes typically, often quicker depending on your ToR router’s BGP configuration.

That’s it! Wow, wasn’t that simple? Your Kubernetes cluster is now ready to begin announcing routes for any services and pods that you deploy. Once the relationships have been established, you should see some announcements to cover what is already deployed in the cluster. You will only see /32 announcements for services which have been configured with one or more external IP addresses. None of the default services deployed for a fresh cluster define this setting so don’t expect to see any out of the gate if you have not already configured something this way.

There are some important details to note about this particular configuration (which can be alternated to work slightly differently). With this configuration, the node-to-node mesh networking is disabled which means that traffic for a given service IP will be routed only to the cluster nodes which are actively hosted an instance of the service. This is really handy if any of your services have a need to see the source IP of ingress traffic (such as for security purposes). Depending on your use case, this might not be the most ideal configuration though since additional external configuration would be required on your router to properly balance ingress traffic to each node hosting a service instance.

When you use the node-to-node mesh networking feature, you can direct ingress traffic for your services to any mesh node in the cluster even if the service doesn’t have an instance running on a particular node. If the service isn’t running locally, the node will forward the traffic to a different node that is running the service. This approach is great if you want ingress traffic to be automatically balanced across all available nodes running a particular service. The caveat here is that when using mesh networking, your service may not always see the source IP address of the ingress traffic since the packets will be rewritten with a source IP address that reflects the cluster node that forwarded the traffic via the mesh network.

I recommend you check out this post next to take your Kubernetes deployment to the next level.

To get started, you need at least one clean Ubuntu 20.04 environment, ideally at least two Ubuntu 20.04 environments to separate your control plane and worker nodes. For high availability environments, you should have at least three control plane nodes and at least two worker nodes. You don’t have to create all the additional nodes at once as adding them later isn’t much of a chore. The instructions that follow will assume you’re working with at least two nodes, one for your control plane, and one for your workload. It is very important that every node you intend to join to the same cluster has a unique hostname configured. It does not matter if two nodes are located in different network search domains, their hostname must specifically be unique to the cluster.

Planning Cluster Configuration

Before you start executing commands, you need to get all your ducks in a row with a few pieces of configuration information. The following is an example of the configuration used in the Azorian Solutions production Kubernetes cluster:

             Cluster Domain: k8s.azorian.solutions
     Control Plane Endpoint: cp.k8s.azorian.solutions
          Pod Networks CIDR: 10.255.0.0/16
      Service Networks CIDR: 10.10.10.0/23
            Kubelet Version: 1.22.0-00
            Kubeadm Version: 1.22.0-00
            Kubectl Version: 1.22.0-00

The first configuration item is the cluster domain. Think of this much like you would a network search domain although it isn’t necessarily used in the same way.

The next configuration item is the control plane endpoint. This is typically a domain name containing an A record corresponding to the IP address of each control plane node that will operate in the cluster. Initially, this domain should only contain a single A record that points to the IP address of the first control plane node that you intend to initialize the Kubernetes cluster on. If you miss this detail during setup, it could lean to painful results forcing you to start the entire process all over again (assuming you don’t have great proficiency in Kubernetes clusters to fix it).

Moving on, the pod networks CIDR should be a fairly large private network allocation as this will be carved up into many smaller networks for each pod that gets deployed to the cluster. Remember, this could easily equate to one network per container deployed depending on how you deploy your services so don’t get stingy on address space. This is not something that is easily changed after the fact. The allocation chosen here should not be your easiest to remember as you likely won’t spend much if any time directly interacting with the IP addresses that are assigned from these networks. It’s also important to remember that the addresses automatically assigned from this network are ephemeral in nature in accordance with the destruction and recreation of pods that happen automatically from time to time.

Next up is the service networks CIDR. Until you get familiar with exactly how Kubernetes works, I recommend starting with a /23 private network as this will ensure you don’t likely end up in a situation where you run out of space from deploying services to the cluster. Services in Kubernetes are like a public facing gateway to your various pods (which contain one or more containers). You can think of a service similar to how you would think of a load balancer. It will receive communication requests to specific IP addresses and ports which are then automatically routed to any pods that have been linked to the service.

Finally, you have your Kubelet, Kubeadm, and Kubectl versions. You can select any of the available Kubernetes release versions here. It is considered best practice for compatibility reasons to keep all three of these packages in sync with at least major and minor versions. If you know what you’re doing, you can install different versions of each but you may experience problems by doing so if there is a breaking change between two selected versions. Version 1.23 hasn’t been out long and I haven’t tested it so for this tutorial, we’ll stick with version 1.22.0-00 which is what I am running today.

Preparing The Nodes

Let’s start by running a number of commands to get each node of the Kubernetes cluster prepared for deployment. Begin by opening up a command shell to each of the nodes with a non-root user that has super user (sudo) privileges.

First, you need to disable the use of swap partitions. Execute the following commands on each node:

sudo sed -i 's/^\/swap/#\/swap/g' /etc/fstab
sudo swapoff -a

Next, you need to update the OS to the latest packages for the current distribution. Execute the following commands on each node:

sudo apt update
sudo apt upgrade -y

Now it’s time to install some basic dependencies to support the remainder of the installation process. Execute the following command on each node:

sudo apt install -y apt-transport-https ca-certificates curl gnupg lsb-release net-tools containerd

Now you need to load the “overlay” and “br_netfilter” Linux kernel modules. Execute the following commands on each node:

sudo modprobe overlay
sudo modprobe br_netfilter

The previous commands only load kernel modules temporarily. You need to add a configuration file to ensure that these modules are loaded after each machine reboot. Execute the following command on each node:

echo 'overlay
br_netfilter' | sudo tee /etc/modules-load.d/k8s.conf > /dev/null

Now you need to add a kernel configuration file to tweak three networking settings. Execute the following command on each node:

echo 'net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1' | sudo tee /etc/sysctl.d/99-k8s.conf > /dev/null

The first of the three aforementioned settings instruct the kernel to accept and forward IP packets not intended for the host machine, but instead one or more of the guests running on the host. The second and third settings control whether or not packets passing through a Linux networking bridge are passed to iptables for processing.

Now you need to instruct the kernel to load the new kernel configuration file to apply the changes. Execute the following command on each node:

sudo sysctl --system

Now you need to install the GPG key for the Google APT repository that contains the Kubernetes packages you’ll need. Execute the following command on each node:

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

Now you need to add a configuration file for the APT package manager to make the system aware of the Kubernetes package repository hosted by Google. Execute the following command on each node:

echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list > /dev/null

Now you need to update APT’s package cache to download the latest package information from the Kubernetes repository. Execute the following command on each node:

sudo apt update

Now you need to set some temporary environment variables that will be used during the Kubernetes package installation process. Execute the following commands on each node, taking care to update each variable with the appropriate versions that you chose for Kubelet, Kubeadm, and Kubectl at the beginning of this tutorial:

kubelet_version=1.22.0-00
kubeadm_version=1.22.0-00
kubectl_version=1.22.0-00

Now you are finally ready to install the Kubernetes packages. Execute the following commands on each node:

KUBELET_EXTRA_ARGS='--feature-gates="AllAlpha=false,RunAsGroup=true"
  --container-runtime=remote --cgroup-driver=systemd
  --runtime-request-timeout=5m
  --container-runtime-endpoint="unix:///var/run/containerd/containerd.sock"'

sudo apt install -y --allow-change-held-packages \
  kubelet=$kubelet_version kubeadm=$kubeadm_version \
  kubectl=$kubectl_version kubernetes-cni

Now there is one final dependency to install which is the control script for the Calico CNI since this tutorial will be making use of Calico as the container networking interface (CNI). Execute the following commands on each node:

export calico_version=$(curl --silent "https://api.github.com/repos/projectcalico/calico/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')

curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" -o kubectl-calico -L https://github.com/projectcalico/calico/releases/download/$calico_version/calicoctl-linux-amd64

sudo chown root:root kubectl-calico
sudo chmod +x kubectl-calico
sudo mv kubectl-calico /usr/local/bin/
sudo ln -s /usr/local/bin/kubectl-calico /usr/local/bin/calicoctl

Alright, that completes the node preparation phase of the deployment! I promise the next phase will be much quicker.

Initializing The Cluster

It’s finally time to initialize the Kubernetes cluster now that you have all of the nodes in a prepared state. If you are running the nodes in virtualization environment like Proxmox or ESXI, you may want to consider doing a quick snapshot of each node just in case something doesn’t go right with the initialization process. This will give you a relatively simple way to revert the coming changes and try again without having to completely rebuild the cluster nodes all over again.

Let’s start by setting some additional temporary environment variables to assist with building a cluster configuration file. Execute the following commands on the first control plane node only! Take care to update each variable to the corresponding configuration value chosen at the beginning of this tutorial.

cluster_domain=k8s.azorian.solutions
control_plane_endpoint=cp.k8s.azorian.solutions
pod_networks_cidr=10.255.0.0/16
service_networks_cidr=10.10.10.0/23

Now you need to create a cluster initialization file in YAML format. Execute the following command on the first control plane node only!

echo 'kind: InitConfiguration
apiVersion: kubeadm.k8s.io/v1beta2
nodeRegistration:
  criSocket: /var/run/containerd/containerd.sock
---
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta2
controlPlaneEndpoint: '"$control_plane_endpoint"':6443
networking:
  dnsDomain: '"$cluster_domain"'
  podSubnet: '"$pod_networks_cidr"'
  serviceSubnet: '"$service_networks_cidr"'
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
---
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1' | tee /tmp/kubeadm-config.yaml

Now you can finally initialize the first node of the cluster! Execute the following command on the first control plane node only!

sudo kubeadm init --config /tmp/kubeadm-config.yaml --upload-certs

If all goes well with the cluster initialization, then the aforementioned command should produce some output that provides two different commands beginning with “kubeadm join” that you need to capture for use a little later in this tutorial.

Now you need to copy the admin security configuration file of the cluster to the appropriate location in the current user’s home directory so that you can execute various cluster administration commands as the current user without the user of super user (sudo) privileges. Execute the following command on the first control plane node only!

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You now have a bare bones Kubernetes cluster! You aren’t done quiet yet though as this cluster doesn’t have any container networking interface. Read on.

Installing Calico CNI

This step should prove to be the easiest. Execute the following command on the first control plane node only!

curl https://docs.projectcalico.org/manifests/calico.yaml | kubectl apply -f -

Alright, now the cluster has a container network interface installed and is ready for use if you are only working with a single node. Otherwise, read on.

Initializing The Remaining Nodes

If you are setting up more than one node for your Kubernetes environment, then you need to execute the “kubeadm join” commands that you captured during cluster initialization in an earlier step. For each of the worker nodes you are going to join to the cluster, execute the shorter of the two commands (the one that does not contain the –control-plane parameter) on the node.

If you are setting up more than one control plane nodes for your Kubernetes environment, then you need to execute the longer of the two commands (the one that does contain the –control-plane parameter) on each remaining control plane nodes. Additionally, for ease of operation, also execute the following commands on each of the additional control plane nodes:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

There is one final step to ensure that the Kubernetes cluster is ready to start deploying components. Each node that will be handling workloads needs to have a Kubernetes worker role label attached. Run the following command from a control plane node for each worker node in the cluster taking care to substitute the “HOSTNAME-HERE” string with the node’s configured hostname. If you aren’t sure of the node’s hostname, you can determine this by executing the “hostname” command on the node in question.

kubectl label node HOSTNAME-HERE node-role.kubernetes.io/worker=worker

That’s it! You now have an operational Kubernetes cluster environment. I don’t want to toot your horn, but you’re kind of a big deal now.

I recommend you check out this post next to take your Kubernetes deployment to the next level.