Given that the general focus of many of my blog posts will be containerization using OCI compliant images, I am not going to go into great detail about the LXC system but some of the benefits I outline about OCI containers will also be true for LXC.

So why bother with containers anyways when you have perfectly great environments for running virtual machines? Well, it seems like the list of reasons is endless but I will try to cover many of the reasons I know and find relevant to most of what I do. So let’s jump right in to it and I will explain many of the high level areas of benefits.

Resource Efficiency

If you have ever created a number of virtual machines and then taken notice to their idle resource utilization before you have configured them for their particular roles, you may have noticed that the utilization is not exactly tiny for a machine that isn’t really doing anything. This is because the Linux kernel while very efficient compared to that POS Windows, is still a bit heavy once loaded into memory. This isn’t a bad thing since after all, it offers some pretty amazing abilities. This is however often wasteful in a number of environments that don’t really need to have a large number of Linux kernel copies running just to service one or two primary processes per virtual machine.

There are some scenarios that justify the use of separate kernels still for truly mission critical security guarantees such as those maintained by nation states. This isn’t to say that the Linux kernel is insecure as it provides all of the appropriate isolation facilities for containers to run fully isolated from one another. However, humans make mistakes and every once in awhile, a security bug may come along that could put this isolation at risk. This is exactly why every application should be considered on a case by case basis. If you’re just running some everyday applications like database servers, DNS servers, and others alike that don’t hold truly sensitive data (like the kind that could contribute to the loss of millions of dollars or human life) then you really don’t need to achieve paranoid level security policies.

This is where containers really shine in a noticeable way as the overhead for a container is more or less just the overhead of the process or processes that it runs as opposed to creating entire copies of the Linux kernel. Instead, containers on a container host simply share the resources of the host machine through the user of kernel cgroups and namespaces. Namespaces allow for the creation of virtual hardware in the container and act as a sort of gateway to the underlying host hardware that is to be shared between containers. Kernel control groups (or cgroups for short) are used to apply limits, prioritization, accounting and control of various system resources such as memory, CPU, and file systems. You can can read more about Kernel cgroups here.

Portability

Containers provide a really effective means of packaging applications for distribution and hosting. If you have ever managed traditional bare metal or virtual machine based servers back in the day, you have almost certainly ran into scenarios where you were trying to deploy multiple applications to the same machine only to find out the hard way that the applications depend on different versions of a shared package. A good example of this would be a web application that requires a newer version of a language interpreter like PHP but the other application might be fairly outdated and require a much older version to work. This can be a real PITA to make work on the same machine in a clean way that doesn’t make maintaining ongoing updates a real nightmare.

Containers solve this problem by creating an isolated environment for each process as desired which means you can now run an arbitrary amount of applications on the same host machine, all requiring a different version of a common dependency without any concern. Furthermore, this design also isolates the package updates of one packaged application from another that might be loosely coupled together. An example of this could be a web application that also has a separate application to provide an API.

Deployment

Another great thing that containers solve is the ability to quickly roll out additional copies of an application for testing or updates. Since every container can maintain it’s own entire set of dependencies, this means you can create multiple copies of the same application with different versions of dependencies installed. This is incredibly useful for mission critical processes that demand zero downtime. With this model, you can easily roll out a copy of an application with the latest updates for dependencies to test.

I know what some of you are thinking at this point, “I could do that before with virtual machines through the use of snapshots and cloning.” You are correct about that but the speed and efficiency at which that can be done has been rivaled by the equivalent process in containers. Depending on the operational requirements of the application, you may not be able to keep both versions actively running side by side thus forcing you to do a form of fast swapping. This process becomes much slower if you’re dealing in the speed of VM snapshot restoration or rebooting.

With containers, since you don’t have the time it takes to load the entire kernel and all supporting startup processes, you can move immediately to starting the actual primary process from the moment you say go. This is great if you have a scenario where a database is too big to run multiple copies for testing but an application that relies on that database would be destructive if two copies were using the same database instance (such as network monitoring for example).

Scalability

Since the general design intention of a container is to one a single primary process such as a database server, an HTTP server, or maybe an SSH server as examples, this by design lends itself very well to the modern micro-service architecture that has taken hold in the application development world. A micro-service architecture is seemingly the most ideal design choice for most environments today because not only does it better isolate application component failures thereby limiting the effects of an outage, it also makes it much easier to scale applications more efficiently.

Consider the design of the original eBay architecture. It followed the old school principal of monolithic design where you had one ginormous application that couldn’t be ran on multiple machines very well. This meant that as user demand grew, the underlying hardware running the application had to grow as well. At different points in history, this quickly became a problem because hardware has only ever been so powerful at any given time. If the best CPU at the time only offered 8 cores and the best motherboards could only handle 4 CPUs, then you have quickly found a limit to just how far that application can scale. This scenario illustrated one of the many great examples why micro-service architecture had to become the new approach moving forward. It was the only realistic way to eventually to to the scale that some websites today run at such as eBay, Facebook, Twitter, Instagram, etc…

Possibly to your surprise, just about every major tech company today is making use of not only micro-service architecture but containerization as well. Examples include all of the major video streaming companies like Netflix, Hulu, Amazon Prime, etc.. as well as some of the biggest telecommunications providers in the world like Comcast, Cox, Charter, Verizon, AT&T, etc… This new model of application design has really changed the landscape for efficient scalability.

Even before containers, large organizations would use automation to grow and shrink the underlying resources of applications to handle spikes and valleys in user demand. Imagine having an application that would see a 5000% traffic growth at roughly the same time every day on a daily basis. Think about people returning home from work and turning on the latest episode of a popular series that just came out that day. Before containers, this meant that a lot of virtual machines would be created and/or started simultaneously to meet the predictable demand of a hungry user base. Starting that many VMs at one time can really put some strain on infrastructure, especially the underlying storage systems. Containers have really curved that strain widely since there is far less required to be retrieved from an underlying storage system during process start-up. This means creating 1,000 copies of a process is far less demanding on physical hardware than creating 1,000 VMs to run that single process.

Networking

The advent of containerization also provided the ability to take new, more unique approaches to solving a number of networking challenges that many large scale applications like Facebook and others have encountered inside of data centers. You can run all the fiber and switches you want, but depending on how your application is designed, you can still create massive traffic jams across data center networks. Since container orchestration quickly became much more advanced to meet the growing engineering demands of containerized applications, this paved the road for some new approaches to mitigating these types of traffic jams.

Traditionally, it wouldn’t be uncommon to see data center designs that kept all of the HTTP services on one cluster of servers and all of the database services on a totally different set of servers. Naturally, this meant there often had to be a very hefty fiber optic and switch network to interconnect the clusters. This can quickly create other scalability issues where throwing money at the problem simply won’t resolve the issues.

Enter a more modern approach where now a company might use advanced container orchestration to ensure that anytime a copy of an HTTP service is deployed to a physical host, that a copy of the supporting database server also gets deployed on the same physical machine along side the HTTP service. This means that requests served by the HTTP service no longer require the network communication to the database service to leave the physical machine and traverse an expensive and hard to maintain fiber optic network.

In Closing

I could go on with many other use case examples and detailed explanations but I imagine you get the general idea. Just as when the advent of virtual machines happened and took the computing world by storm, containers have done the same thing thereby taking us all on another huge leap into the future of large scale computing. I realize that many of my examples were based on large scale deployments of huge applications but it’s important to remember that many of these same benefits translate to smaller organizations as well. After all, automation is automation no matter what the scale. If a human doesn’t have to handle the task or a task is less error prone, then there is still a quantifiable value proposition.

Enter BGP peering for Kubernetes using Calico CNI. Using this method, you can have all of your Kubernetes nodes communicate directly with an external network router via BGP in order to distribute routes to your cluster services and pods. Once this has been setup, any services you configure (properly) in your cluster will become immediately accessible to your network as routes are injected immediately upon activation.

In order to apply the YAML formatted configuration files needed to setup BGP peering, you will need to have the Calico CNI control script installed in an environment that has proper permissions to manage the cluster. If you used the tutorial I provided in this blog for deploying Kubernetes, then this control script will have already been installed to each of your control plane nodes with the alias “kubectl-calico” and “calicoctl” so you’re already good to get started.

If you setup your Kubernetes cluster some other way and do not currently have the Calico CNI control script installed, execute the following commands from an environment you have setup with kubectl for the cluster:

export calico_version=$(curl --silent "https://api.github.com/repos/projectcalico/calico/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')

curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" -o kubectl-calico -L https://github.com/projectcalico/calico/releases/download/$calico_version/calicoctl-linux-amd64

sudo chown root:root kubectl-calico
sudo chmod +x kubectl-calico
sudo mv kubectl-calico /usr/local/bin/
sudo ln -s /usr/local/bin/kubectl-calico /usr/local/bin/calicoctl

At this point, you should be ready to start building the basic configuration file required to update the Calico CNI in order to establish BGP peering both with your network router and between nodes (route reflection). The example I provide in this tutorial will assume a very basic setup that assumes only one availability zone with a single top-of-rack (ToR) router. If you are building something with even better redundancy, it’s not complicated to pivot from this configuration to one that supports multiple availability zones and/or multiple ToR routers.

Let’s start by applying some additional labels to all of your cluster nodes. These labels will be used to selectively enable route reflection via BGP. Execute the following command for each node in your cluster by updating the “HOSTNAME-HERE” string to match the hostname configured on each node. If you’re unsure of a node’s hostname, just execute the “hostname” command on that node.

kubectl label node HOSTNAME-HERE route-reflector=true

After this has been completed, you can verify that everything looks correct by executing the following command:

kubectl get nodes --show-labels

Now let’s create a YAML formatted configuration file using the following command:

echo 'apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false
  serviceClusterIPs:
  - cidr: 10.10.10.0/23
  serviceExternalIPs:
  - cidr: 10.10.0.0/23
  - cidr: 172.21.0.0/24
  listenPort: 179
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: as-rtr1
spec:
  peerIP: 172.22.100.254
  asNumber: 64512
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: as-node
spec:
  nodeSelector: all()
  peerSelector: route-reflector == \'true\'' | tee /tmp/calico-bgp-configuration.yaml

Before you apply this configuration file to the cluster, some of the values will probably need modified to reflect appropriate settings that match your environment.

The first item to update is the “serviceClusterIPs” entry. This should contain the CIDR that you configured for your service networks CIDR when initializing the cluster. If you followed my tutorial for deploying the Kubernetes cluster initially, all of this should have some familiarity already.

  serviceClusterIPs:
  - cidr: 10.10.10.0/23

The next item to update is the “serviceExternalIPs” entry. This should contain one or more CIDR entries to reflect any IP ranges that you intend to assign external service IP addresses from. For example, if you were going to deploy a DNS resolver for your network with an IP address of “10.10.0.1” then you would add a CIDR entry that includes that particular IP address such as “- cidr: 10.10.0.0/24”. IP addresses assigned to services from the ranges configured in this entry will be announced as individual /32 routes via BGP. This can be very useful if you need to cherry pick addresses in-between existing network allocations or if you need to temporarily “hijack” the traffic of another service on your network. A good example of this would be doing brief testing of new services that will replace existing deployments elsewhere on your network.

  serviceExternalIPs:
  - cidr: 10.10.0.0/23
  - cidr: 172.21.0.0/24

Lastly, you may need to update the “peerIP” and “asNumber” entries to reflect the appropriate IP address and ASN of your ToR router that the cluster nodes should peer to.

spec:
  peerIP: 172.22.100.254
  asNumber: 64512

Now that the configuration file is ready, it’s time to apply it to the cluster. Don’t forget that you will need to update your ToR router with the appropriate configuration to support BGP peering from each of your cluster nodes. The remote ASN used by the cluster nodes in this configuration will be the same as the ASN assigned in the previous section. To apply the configuration file to the cluster, execute the following command:

calicoctl apply -f /tmp/calico-bgp-configuration.yaml

At this point, if your ToR router has already been properly configured for BGP peering to your cluster nodes, you should see the peering relationships begin to establish after no more than one to two minutes typically, often quicker depending on your ToR router’s BGP configuration.

That’s it! Wow, wasn’t that simple? Your Kubernetes cluster is now ready to begin announcing routes for any services and pods that you deploy. Once the relationships have been established, you should see some announcements to cover what is already deployed in the cluster. You will only see /32 announcements for services which have been configured with one or more external IP addresses. None of the default services deployed for a fresh cluster define this setting so don’t expect to see any out of the gate if you have not already configured something this way.

There are some important details to note about this particular configuration (which can be alternated to work slightly differently). With this configuration, the node-to-node mesh networking is disabled which means that traffic for a given service IP will be routed only to the cluster nodes which are actively hosted an instance of the service. This is really handy if any of your services have a need to see the source IP of ingress traffic (such as for security purposes). Depending on your use case, this might not be the most ideal configuration though since additional external configuration would be required on your router to properly balance ingress traffic to each node hosting a service instance.

When you use the node-to-node mesh networking feature, you can direct ingress traffic for your services to any mesh node in the cluster even if the service doesn’t have an instance running on a particular node. If the service isn’t running locally, the node will forward the traffic to a different node that is running the service. This approach is great if you want ingress traffic to be automatically balanced across all available nodes running a particular service. The caveat here is that when using mesh networking, your service may not always see the source IP address of the ingress traffic since the packets will be rewritten with a source IP address that reflects the cluster node that forwarded the traffic via the mesh network.

I recommend you check out this post next to take your Kubernetes deployment to the next level.