Given that the general focus of many of my blog posts will be containerization using OCI compliant images, I am not going to go into great detail about the LXC system but some of the benefits I outline about OCI containers will also be true for LXC.

So why bother with containers anyways when you have perfectly great environments for running virtual machines? Well, it seems like the list of reasons is endless but I will try to cover many of the reasons I know and find relevant to most of what I do. So let’s jump right in to it and I will explain many of the high level areas of benefits.

Resource Efficiency

If you have ever created a number of virtual machines and then taken notice to their idle resource utilization before you have configured them for their particular roles, you may have noticed that the utilization is not exactly tiny for a machine that isn’t really doing anything. This is because the Linux kernel while very efficient compared to that POS Windows, is still a bit heavy once loaded into memory. This isn’t a bad thing since after all, it offers some pretty amazing abilities. This is however often wasteful in a number of environments that don’t really need to have a large number of Linux kernel copies running just to service one or two primary processes per virtual machine.

There are some scenarios that justify the use of separate kernels still for truly mission critical security guarantees such as those maintained by nation states. This isn’t to say that the Linux kernel is insecure as it provides all of the appropriate isolation facilities for containers to run fully isolated from one another. However, humans make mistakes and every once in awhile, a security bug may come along that could put this isolation at risk. This is exactly why every application should be considered on a case by case basis. If you’re just running some everyday applications like database servers, DNS servers, and others alike that don’t hold truly sensitive data (like the kind that could contribute to the loss of millions of dollars or human life) then you really don’t need to achieve paranoid level security policies.

This is where containers really shine in a noticeable way as the overhead for a container is more or less just the overhead of the process or processes that it runs as opposed to creating entire copies of the Linux kernel. Instead, containers on a container host simply share the resources of the host machine through the user of kernel cgroups and namespaces. Namespaces allow for the creation of virtual hardware in the container and act as a sort of gateway to the underlying host hardware that is to be shared between containers. Kernel control groups (or cgroups for short) are used to apply limits, prioritization, accounting and control of various system resources such as memory, CPU, and file systems. You can can read more about Kernel cgroups here.

Portability

Containers provide a really effective means of packaging applications for distribution and hosting. If you have ever managed traditional bare metal or virtual machine based servers back in the day, you have almost certainly ran into scenarios where you were trying to deploy multiple applications to the same machine only to find out the hard way that the applications depend on different versions of a shared package. A good example of this would be a web application that requires a newer version of a language interpreter like PHP but the other application might be fairly outdated and require a much older version to work. This can be a real PITA to make work on the same machine in a clean way that doesn’t make maintaining ongoing updates a real nightmare.

Containers solve this problem by creating an isolated environment for each process as desired which means you can now run an arbitrary amount of applications on the same host machine, all requiring a different version of a common dependency without any concern. Furthermore, this design also isolates the package updates of one packaged application from another that might be loosely coupled together. An example of this could be a web application that also has a separate application to provide an API.

Deployment

Another great thing that containers solve is the ability to quickly roll out additional copies of an application for testing or updates. Since every container can maintain it’s own entire set of dependencies, this means you can create multiple copies of the same application with different versions of dependencies installed. This is incredibly useful for mission critical processes that demand zero downtime. With this model, you can easily roll out a copy of an application with the latest updates for dependencies to test.

I know what some of you are thinking at this point, “I could do that before with virtual machines through the use of snapshots and cloning.” You are correct about that but the speed and efficiency at which that can be done has been rivaled by the equivalent process in containers. Depending on the operational requirements of the application, you may not be able to keep both versions actively running side by side thus forcing you to do a form of fast swapping. This process becomes much slower if you’re dealing in the speed of VM snapshot restoration or rebooting.

With containers, since you don’t have the time it takes to load the entire kernel and all supporting startup processes, you can move immediately to starting the actual primary process from the moment you say go. This is great if you have a scenario where a database is too big to run multiple copies for testing but an application that relies on that database would be destructive if two copies were using the same database instance (such as network monitoring for example).

Scalability

Since the general design intention of a container is to one a single primary process such as a database server, an HTTP server, or maybe an SSH server as examples, this by design lends itself very well to the modern micro-service architecture that has taken hold in the application development world. A micro-service architecture is seemingly the most ideal design choice for most environments today because not only does it better isolate application component failures thereby limiting the effects of an outage, it also makes it much easier to scale applications more efficiently.

Consider the design of the original eBay architecture. It followed the old school principal of monolithic design where you had one ginormous application that couldn’t be ran on multiple machines very well. This meant that as user demand grew, the underlying hardware running the application had to grow as well. At different points in history, this quickly became a problem because hardware has only ever been so powerful at any given time. If the best CPU at the time only offered 8 cores and the best motherboards could only handle 4 CPUs, then you have quickly found a limit to just how far that application can scale. This scenario illustrated one of the many great examples why micro-service architecture had to become the new approach moving forward. It was the only realistic way to eventually to to the scale that some websites today run at such as eBay, Facebook, Twitter, Instagram, etc…

Possibly to your surprise, just about every major tech company today is making use of not only micro-service architecture but containerization as well. Examples include all of the major video streaming companies like Netflix, Hulu, Amazon Prime, etc.. as well as some of the biggest telecommunications providers in the world like Comcast, Cox, Charter, Verizon, AT&T, etc… This new model of application design has really changed the landscape for efficient scalability.

Even before containers, large organizations would use automation to grow and shrink the underlying resources of applications to handle spikes and valleys in user demand. Imagine having an application that would see a 5000% traffic growth at roughly the same time every day on a daily basis. Think about people returning home from work and turning on the latest episode of a popular series that just came out that day. Before containers, this meant that a lot of virtual machines would be created and/or started simultaneously to meet the predictable demand of a hungry user base. Starting that many VMs at one time can really put some strain on infrastructure, especially the underlying storage systems. Containers have really curved that strain widely since there is far less required to be retrieved from an underlying storage system during process start-up. This means creating 1,000 copies of a process is far less demanding on physical hardware than creating 1,000 VMs to run that single process.

Networking

The advent of containerization also provided the ability to take new, more unique approaches to solving a number of networking challenges that many large scale applications like Facebook and others have encountered inside of data centers. You can run all the fiber and switches you want, but depending on how your application is designed, you can still create massive traffic jams across data center networks. Since container orchestration quickly became much more advanced to meet the growing engineering demands of containerized applications, this paved the road for some new approaches to mitigating these types of traffic jams.

Traditionally, it wouldn’t be uncommon to see data center designs that kept all of the HTTP services on one cluster of servers and all of the database services on a totally different set of servers. Naturally, this meant there often had to be a very hefty fiber optic and switch network to interconnect the clusters. This can quickly create other scalability issues where throwing money at the problem simply won’t resolve the issues.

Enter a more modern approach where now a company might use advanced container orchestration to ensure that anytime a copy of an HTTP service is deployed to a physical host, that a copy of the supporting database server also gets deployed on the same physical machine along side the HTTP service. This means that requests served by the HTTP service no longer require the network communication to the database service to leave the physical machine and traverse an expensive and hard to maintain fiber optic network.

In Closing

I could go on with many other use case examples and detailed explanations but I imagine you get the general idea. Just as when the advent of virtual machines happened and took the computing world by storm, containers have done the same thing thereby taking us all on another huge leap into the future of large scale computing. I realize that many of my examples were based on large scale deployments of huge applications but it’s important to remember that many of these same benefits translate to smaller organizations as well. After all, automation is automation no matter what the scale. If a human doesn’t have to handle the task or a task is less error prone, then there is still a quantifiable value proposition.

There are many different container images available for various DNS recursors in the two primary repositories Docker Hub and RedHat Quay. The problem is that many are poorly implemented, not well maintained, and often lack pseudo-standard deployment features that really put that shine on the final product. Often times, even when a product vendor provides their own container images directly, they still are quite lackluster to say the least. For this reason, I will be using container images that I have built myself and are being actively maintained for my own production use.

To get started, be sure to check out the specific image for this tutorial in my GitHub repository which can be found here. This tutorial will assume that you have both a functioning Docker environment as well as the docker-compose tool installed. There are a few ways you can deploy images to a Docker environment but I am going to only cover the docker-compose route as it’s most practical for ongoing maintenance and also translates very well to the Portainer GUI app designed for managing container environments.

Let’s start by creating a YAML formatted file for docker-compose. I will go over some of the various settings in this file following the example so you may want to review those first and tweak the following example accordingly. From a command shell in your Docker environment. execute the following command:

echo 'version: '3.3'
services:
  recursor:
    image: azoriansolutions/powerdns-recursor:latest
    restart: unless-stopped
    environment:
      - PDNS_allow_from=10.0.0.0/8,123.45.67.0/23
      - PDNS_local_port=53
      - PDNS_local_address=0.0.0.0
      - PDNS_any_to_tcp=yes
      - PDNS_api_key=SECURE-API-KEY
      - PDNS_dnssec=validate
      - PDNS_dnssec_log_bogus=yes
      - PDNS_loglevel=3
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=0.0.0.0/0
      - PDNS_webserver_password=SECURE-HTTP-AUTH-PASSWORD
    ports:
      - "8053:53/udp"
      - "8053:53"
      - "8080:8082"' | tee /tmp/dc-powerdns-recursor.yaml

One of the first and most important settings to take notice of is the “PDNS_allow_from” setting. This defines one or more networks in CIDR format that the recursor will accept DNS queries from. The default value of the setting only allows access from RFC 1918 private IP addresses so you may need to update this if you will be receiving traffic from some public IP networks. You should never set this value to “0.0.0.0/0” unless you know what you’re doing. Otherwise, in many scenarios this can lead to an open resolver which can and will eventually be turned into an attack vector by bad actors. If you are implementing proper firewall filters somewhere along the communication chain, then it would be okay to change this to allow all traffic for ease of configuration maintenance.

Another important setting is the “PDNS_api_key” which controls access to the built-in API server which can be used for issuing various commands to the recursor at runtime. This setting isn’t important if you end up disabling the built-in web-server.

You should definitely take note to the “PDNS_dnssec” setting as this example has configured the most strict form of DNSSEC which is full blown validation. This may have adverse effects depending on how you intend to use the recursor. For more information on this setting, check out the recursor settings page here.

Next up is the “PDNS_webserver” setting. This controls whether or not the built-in web server is enabled. This web server provides a basic page to see real-time performance statistics of the recursor. The web server is also responsible for providing the API server so you must enable it to use the API features.

Also to go with the last setting, if you do enable the web server you may want to consider adjusting the “PDNS_webserver_allow_from” setting. This defines one or more networks in CIDR format which HTTP requests to the web server should be allowed from. If you are performing packet filtering upstream then you are okay to leave this to allow all traffic for ease of configuration maintenance.

Lastly, take notice to the “PDNS_webserver_password” setting which configures a basic password used for HTTP authentication to the real-time statistics page provided by the web server.

Depending on your environment and what you have deployed to the Docker server you’ll be using, you may want to consider changing the port settings:

    ports:
      - "8053:53/udp"
      - "8053:53"
      - "8080:8082"

The value “8053” defines the host port on the Docker machine to bind the internal container port of 53 to. The other value of “8080” defines the host port on the Docker machine to bind the container’s internal web server port of 8082 to. If you aren’t using any form of reverse proxy to manage ingress traffic to your containerized services and just want to have the Docker machine handle requests directly, then you may consider changing the “8053” values to the standard port of “53”.

You need to be aware that in many environments such as Ubuntu, you can not just change to port 53 alone as the container will fail to start. This is because newer Ubuntu releases have a local resolver running by default on port 53 but it is only bound to the loopback address. Docker has a great feature to work around this challenge though. You can also add a specific IP address before the host port number to keep from binding to all available interfaces on the machine. For example, if your Docker host has a public IP address of “123.45.67.76” then you could change the port configuration to “123.45.67.76:53:53/udp” and “123.45.67.76:53:53” respectively. This will prevent conflicts with the local resolver that is running by default on Ubuntu.

At this point, you should have tweaked the example configuration either before or after creating the YAML file so presumably you’re ready to deploy the recursor now. Execute the following command:

docker-compose -u /tmp/dc-powerdns-recursor.yaml

Once the deployment process has been completed, you should now have a working recursor ready to serve DNS requests. There are many more settings that you can change to tweak the recursor behavior. For more information, check out the recursor settings page here.

Have you ever been in a position where it was time to upgrade a production service to a new major version but are reluctant to make such a potentially impacting change to your production environment? I would say many have been here a time or two. Since machine virtualization came along, this challenge has somewhat been mitigated through the availability of great virtualization features like snapshots with timely rollbacks and other similar backup solutions. While this approach has proven to be very valuable, it can still yield some undesired down time during the upgrade and/or rollback process depending on your deployment configurations.

Enter the containerization approach. Wouldn’t it be great if testing a new version of your chosen software was essentially nothing more than stopping one machine process and starting another? This is one of the many outcomes you can achieve when using containerized services. No longer is there a need to spin up a second (potentially resource hungry) virtual machine (VM) just to find out that you have more work to do before upgrading to a new package release for your software. This also applies to scenarios where a new software release goes south sometime after deployment and you need to quickly find your way back to a known working version.

Not all containers are created equally though! The term container is a bit ambiguous since there are actually different types of containers. For example, if you’re a Proxmox Virtualization Environment (PVE) user, you may have noticed the containers feature. This refers to what are known as Linux Containers (LXC). These containers are not the same as the containers you might run with Docker or Podman. LXC is a great alternative to using more resource hungry VMs when you don’t necessarily need the features only found with traditional VMs. A Docker or Podman container is more akin to a convenient development and deployment solution for applications. Think of these container types as a great way to package an application with all of it’s environmental dependencies to make execution in different hosting environments much smoother. Since it is outside the scope of this article, I will not be going into the vast details regarding the differences between LXC and Docker/Podman containers.

Since this article assumes you’re a container novice, I will be focusing on the simplest use cases to get you started with containers. This means I will focus on the use of Docker vs Podman even though both have many things in common including a nearly identical command line interface. There are many reasons why one might choose to use one container solution over the other but again, that is outside the scope of this article.

Installing Docker Engine

To get started with running Docker containers, you need to install the Docker Engine. This software package is readily available on many major Linux distributions including both Debian and RHEL based systems. I will be focusing on deployment in a Debian based environment (Ubuntu) in this article. Before beginning, I will assume you have a fresh, unmodified Ubuntu 18.04+ environment with an active shell session using a non-root user that has super user (sudo) privileges.

The first step is to update your environment with all the latest updates for your current distribution release. Execute the following command:

sudo apt update && sudo apt dist-upgrade -y

The next step is to install a few dependencies that will simplify the remaining steps of the process by condensing some commands into more abstract, non-release specific commands. Execute the following command:

sudo apt install -y ca-certificates curl gnupg lsb-release

Now you need to download and install the Docker repository GPG key to the local key ring. Execute the following command:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Next you will need to add the appropriate APT package manager configuration so that your system is aware of the Docker repository. Execute the following command:

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Now that the operating system is aware of the Docker repository, you need to update the local APT package cache to include all of the packages available from the Docker repository. Execute the following command:

sudo apt update

Now it’s time to install the Docker Engine as well as containerd. Containerd is what ultimately runs the containers in this environment while the Docker Engine manages the setup and maintenance of those running containers. Execute the following command:

sudo apt install -y docker-ce docker-ce-cli containerd.io

Now that the Docker Engine is installed, you should override the default network configuration to help ensure the internal networks used for the containers don’t overlap with your network infrastructure. This step isn’t fool-proof but will likely be sufficient for the typical production environment that doesn’t make use of the IANA 192.168.0.0/16 subnet. Execute the following command:

echo '{
  "bip": "192.168.228.1/23",
  "default-address-pools": [{"base":"192.168.230.0/23","size":27}]
}
' | sudo tee /etc/docker/daemon.json > /dev/null

Now the Docker Engine needs restarted in order for the networking changes to take effect. Execute the following command:

sudo systemctl restart docker

Boom! You’re ready to start running Docker containers already! It really is that simple! Since you’re here reading this article, I will assume you’re not yet a CLI ninja though so you’ll probably want some sort of graphical user interface (GUI) for managing the containers on this new Docker host. There are some options out there but Portainer is by far the most heavily developed product on the market at the time of writing this article.

Installing Portainer GUI

If you would like to install the latest Portainer version to the server to manage it, execute the following commands:

export portainer_version=$(curl --silent "https://api.github.com/repos/portainer/portainer/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')

sudo docker volume create portainer_data

sudo docker run -d -p 8000:8000 -p 9443:9443 --name portainer \
    --restart=always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v portainer_data:/data \
    portainer/portainer-ce:$portainer_version

If everything worked, this should result in a new GUI ready to use by navigating to https://YOUR-MACHINE-IP-OR-HOSTNAME:9443./ On your first visit, you will be prompted to create a new administrator account to use with the app. Now you’re really ready to hit the ground running with ninja speed as you start deploying containers!

To get started, you need at least one clean Ubuntu 20.04 environment, ideally at least two Ubuntu 20.04 environments to separate your control plane and worker nodes. For high availability environments, you should have at least three control plane nodes and at least two worker nodes. You don’t have to create all the additional nodes at once as adding them later isn’t much of a chore. The instructions that follow will assume you’re working with at least two nodes, one for your control plane, and one for your workload. It is very important that every node you intend to join to the same cluster has a unique hostname configured. It does not matter if two nodes are located in different network search domains, their hostname must specifically be unique to the cluster.

Planning Cluster Configuration

Before you start executing commands, you need to get all your ducks in a row with a few pieces of configuration information. The following is an example of the configuration used in the Azorian Solutions production Kubernetes cluster:

             Cluster Domain: k8s.azorian.solutions
     Control Plane Endpoint: cp.k8s.azorian.solutions
          Pod Networks CIDR: 10.255.0.0/16
      Service Networks CIDR: 10.10.10.0/23
            Kubelet Version: 1.22.0-00
            Kubeadm Version: 1.22.0-00
            Kubectl Version: 1.22.0-00

The first configuration item is the cluster domain. Think of this much like you would a network search domain although it isn’t necessarily used in the same way.

The next configuration item is the control plane endpoint. This is typically a domain name containing an A record corresponding to the IP address of each control plane node that will operate in the cluster. Initially, this domain should only contain a single A record that points to the IP address of the first control plane node that you intend to initialize the Kubernetes cluster on. If you miss this detail during setup, it could lean to painful results forcing you to start the entire process all over again (assuming you don’t have great proficiency in Kubernetes clusters to fix it).

Moving on, the pod networks CIDR should be a fairly large private network allocation as this will be carved up into many smaller networks for each pod that gets deployed to the cluster. Remember, this could easily equate to one network per container deployed depending on how you deploy your services so don’t get stingy on address space. This is not something that is easily changed after the fact. The allocation chosen here should not be your easiest to remember as you likely won’t spend much if any time directly interacting with the IP addresses that are assigned from these networks. It’s also important to remember that the addresses automatically assigned from this network are ephemeral in nature in accordance with the destruction and recreation of pods that happen automatically from time to time.

Next up is the service networks CIDR. Until you get familiar with exactly how Kubernetes works, I recommend starting with a /23 private network as this will ensure you don’t likely end up in a situation where you run out of space from deploying services to the cluster. Services in Kubernetes are like a public facing gateway to your various pods (which contain one or more containers). You can think of a service similar to how you would think of a load balancer. It will receive communication requests to specific IP addresses and ports which are then automatically routed to any pods that have been linked to the service.

Finally, you have your Kubelet, Kubeadm, and Kubectl versions. You can select any of the available Kubernetes release versions here. It is considered best practice for compatibility reasons to keep all three of these packages in sync with at least major and minor versions. If you know what you’re doing, you can install different versions of each but you may experience problems by doing so if there is a breaking change between two selected versions. Version 1.23 hasn’t been out long and I haven’t tested it so for this tutorial, we’ll stick with version 1.22.0-00 which is what I am running today.

Preparing The Nodes

Let’s start by running a number of commands to get each node of the Kubernetes cluster prepared for deployment. Begin by opening up a command shell to each of the nodes with a non-root user that has super user (sudo) privileges.

First, you need to disable the use of swap partitions. Execute the following commands on each node:

sudo sed -i 's/^\/swap/#\/swap/g' /etc/fstab
sudo swapoff -a

Next, you need to update the OS to the latest packages for the current distribution. Execute the following commands on each node:

sudo apt update
sudo apt upgrade -y

Now it’s time to install some basic dependencies to support the remainder of the installation process. Execute the following command on each node:

sudo apt install -y apt-transport-https ca-certificates curl gnupg lsb-release net-tools containerd

Now you need to load the “overlay” and “br_netfilter” Linux kernel modules. Execute the following commands on each node:

sudo modprobe overlay
sudo modprobe br_netfilter

The previous commands only load kernel modules temporarily. You need to add a configuration file to ensure that these modules are loaded after each machine reboot. Execute the following command on each node:

echo 'overlay
br_netfilter' | sudo tee /etc/modules-load.d/k8s.conf > /dev/null

Now you need to add a kernel configuration file to tweak three networking settings. Execute the following command on each node:

echo 'net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1' | sudo tee /etc/sysctl.d/99-k8s.conf > /dev/null

The first of the three aforementioned settings instruct the kernel to accept and forward IP packets not intended for the host machine, but instead one or more of the guests running on the host. The second and third settings control whether or not packets passing through a Linux networking bridge are passed to iptables for processing.

Now you need to instruct the kernel to load the new kernel configuration file to apply the changes. Execute the following command on each node:

sudo sysctl --system

Now you need to install the GPG key for the Google APT repository that contains the Kubernetes packages you’ll need. Execute the following command on each node:

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

Now you need to add a configuration file for the APT package manager to make the system aware of the Kubernetes package repository hosted by Google. Execute the following command on each node:

echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list > /dev/null

Now you need to update APT’s package cache to download the latest package information from the Kubernetes repository. Execute the following command on each node:

sudo apt update

Now you need to set some temporary environment variables that will be used during the Kubernetes package installation process. Execute the following commands on each node, taking care to update each variable with the appropriate versions that you chose for Kubelet, Kubeadm, and Kubectl at the beginning of this tutorial:

kubelet_version=1.22.0-00
kubeadm_version=1.22.0-00
kubectl_version=1.22.0-00

Now you are finally ready to install the Kubernetes packages. Execute the following commands on each node:

KUBELET_EXTRA_ARGS='--feature-gates="AllAlpha=false,RunAsGroup=true"
  --container-runtime=remote --cgroup-driver=systemd
  --runtime-request-timeout=5m
  --container-runtime-endpoint="unix:///var/run/containerd/containerd.sock"'

sudo apt install -y --allow-change-held-packages \
  kubelet=$kubelet_version kubeadm=$kubeadm_version \
  kubectl=$kubectl_version kubernetes-cni

Now there is one final dependency to install which is the control script for the Calico CNI since this tutorial will be making use of Calico as the container networking interface (CNI). Execute the following commands on each node:

export calico_version=$(curl --silent "https://api.github.com/repos/projectcalico/calico/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')

curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" -o kubectl-calico -L https://github.com/projectcalico/calico/releases/download/$calico_version/calicoctl-linux-amd64

sudo chown root:root kubectl-calico
sudo chmod +x kubectl-calico
sudo mv kubectl-calico /usr/local/bin/
sudo ln -s /usr/local/bin/kubectl-calico /usr/local/bin/calicoctl

Alright, that completes the node preparation phase of the deployment! I promise the next phase will be much quicker.

Initializing The Cluster

It’s finally time to initialize the Kubernetes cluster now that you have all of the nodes in a prepared state. If you are running the nodes in virtualization environment like Proxmox or ESXI, you may want to consider doing a quick snapshot of each node just in case something doesn’t go right with the initialization process. This will give you a relatively simple way to revert the coming changes and try again without having to completely rebuild the cluster nodes all over again.

Let’s start by setting some additional temporary environment variables to assist with building a cluster configuration file. Execute the following commands on the first control plane node only! Take care to update each variable to the corresponding configuration value chosen at the beginning of this tutorial.

cluster_domain=k8s.azorian.solutions
control_plane_endpoint=cp.k8s.azorian.solutions
pod_networks_cidr=10.255.0.0/16
service_networks_cidr=10.10.10.0/23

Now you need to create a cluster initialization file in YAML format. Execute the following command on the first control plane node only!

echo 'kind: InitConfiguration
apiVersion: kubeadm.k8s.io/v1beta2
nodeRegistration:
  criSocket: /var/run/containerd/containerd.sock
---
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta2
controlPlaneEndpoint: '"$control_plane_endpoint"':6443
networking:
  dnsDomain: '"$cluster_domain"'
  podSubnet: '"$pod_networks_cidr"'
  serviceSubnet: '"$service_networks_cidr"'
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
---
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1' | tee /tmp/kubeadm-config.yaml

Now you can finally initialize the first node of the cluster! Execute the following command on the first control plane node only!

sudo kubeadm init --config /tmp/kubeadm-config.yaml --upload-certs

If all goes well with the cluster initialization, then the aforementioned command should produce some output that provides two different commands beginning with “kubeadm join” that you need to capture for use a little later in this tutorial.

Now you need to copy the admin security configuration file of the cluster to the appropriate location in the current user’s home directory so that you can execute various cluster administration commands as the current user without the user of super user (sudo) privileges. Execute the following command on the first control plane node only!

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You now have a bare bones Kubernetes cluster! You aren’t done quiet yet though as this cluster doesn’t have any container networking interface. Read on.

Installing Calico CNI

This step should prove to be the easiest. Execute the following command on the first control plane node only!

curl https://docs.projectcalico.org/manifests/calico.yaml | kubectl apply -f -

Alright, now the cluster has a container network interface installed and is ready for use if you are only working with a single node. Otherwise, read on.

Initializing The Remaining Nodes

If you are setting up more than one node for your Kubernetes environment, then you need to execute the “kubeadm join” commands that you captured during cluster initialization in an earlier step. For each of the worker nodes you are going to join to the cluster, execute the shorter of the two commands (the one that does not contain the –control-plane parameter) on the node.

If you are setting up more than one control plane nodes for your Kubernetes environment, then you need to execute the longer of the two commands (the one that does contain the –control-plane parameter) on each remaining control plane nodes. Additionally, for ease of operation, also execute the following commands on each of the additional control plane nodes:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

There is one final step to ensure that the Kubernetes cluster is ready to start deploying components. Each node that will be handling workloads needs to have a Kubernetes worker role label attached. Run the following command from a control plane node for each worker node in the cluster taking care to substitute the “HOSTNAME-HERE” string with the node’s configured hostname. If you aren’t sure of the node’s hostname, you can determine this by executing the “hostname” command on the node in question.

kubectl label node HOSTNAME-HERE node-role.kubernetes.io/worker=worker

That’s it! You now have an operational Kubernetes cluster environment. I don’t want to toot your horn, but you’re kind of a big deal now.

I recommend you check out this post next to take your Kubernetes deployment to the next level.