Given that the general focus of many of my blog posts will be containerization using OCI compliant images, I am not going to go into great detail about the LXC system but some of the benefits I outline about OCI containers will also be true for LXC.

So why bother with containers anyways when you have perfectly great environments for running virtual machines? Well, it seems like the list of reasons is endless but I will try to cover many of the reasons I know and find relevant to most of what I do. So let’s jump right in to it and I will explain many of the high level areas of benefits.

Resource Efficiency

If you have ever created a number of virtual machines and then taken notice to their idle resource utilization before you have configured them for their particular roles, you may have noticed that the utilization is not exactly tiny for a machine that isn’t really doing anything. This is because the Linux kernel while very efficient compared to that POS Windows, is still a bit heavy once loaded into memory. This isn’t a bad thing since after all, it offers some pretty amazing abilities. This is however often wasteful in a number of environments that don’t really need to have a large number of Linux kernel copies running just to service one or two primary processes per virtual machine.

There are some scenarios that justify the use of separate kernels still for truly mission critical security guarantees such as those maintained by nation states. This isn’t to say that the Linux kernel is insecure as it provides all of the appropriate isolation facilities for containers to run fully isolated from one another. However, humans make mistakes and every once in awhile, a security bug may come along that could put this isolation at risk. This is exactly why every application should be considered on a case by case basis. If you’re just running some everyday applications like database servers, DNS servers, and others alike that don’t hold truly sensitive data (like the kind that could contribute to the loss of millions of dollars or human life) then you really don’t need to achieve paranoid level security policies.

This is where containers really shine in a noticeable way as the overhead for a container is more or less just the overhead of the process or processes that it runs as opposed to creating entire copies of the Linux kernel. Instead, containers on a container host simply share the resources of the host machine through the user of kernel cgroups and namespaces. Namespaces allow for the creation of virtual hardware in the container and act as a sort of gateway to the underlying host hardware that is to be shared between containers. Kernel control groups (or cgroups for short) are used to apply limits, prioritization, accounting and control of various system resources such as memory, CPU, and file systems. You can can read more about Kernel cgroups here.

Portability

Containers provide a really effective means of packaging applications for distribution and hosting. If you have ever managed traditional bare metal or virtual machine based servers back in the day, you have almost certainly ran into scenarios where you were trying to deploy multiple applications to the same machine only to find out the hard way that the applications depend on different versions of a shared package. A good example of this would be a web application that requires a newer version of a language interpreter like PHP but the other application might be fairly outdated and require a much older version to work. This can be a real PITA to make work on the same machine in a clean way that doesn’t make maintaining ongoing updates a real nightmare.

Containers solve this problem by creating an isolated environment for each process as desired which means you can now run an arbitrary amount of applications on the same host machine, all requiring a different version of a common dependency without any concern. Furthermore, this design also isolates the package updates of one packaged application from another that might be loosely coupled together. An example of this could be a web application that also has a separate application to provide an API.

Deployment

Another great thing that containers solve is the ability to quickly roll out additional copies of an application for testing or updates. Since every container can maintain it’s own entire set of dependencies, this means you can create multiple copies of the same application with different versions of dependencies installed. This is incredibly useful for mission critical processes that demand zero downtime. With this model, you can easily roll out a copy of an application with the latest updates for dependencies to test.

I know what some of you are thinking at this point, “I could do that before with virtual machines through the use of snapshots and cloning.” You are correct about that but the speed and efficiency at which that can be done has been rivaled by the equivalent process in containers. Depending on the operational requirements of the application, you may not be able to keep both versions actively running side by side thus forcing you to do a form of fast swapping. This process becomes much slower if you’re dealing in the speed of VM snapshot restoration or rebooting.

With containers, since you don’t have the time it takes to load the entire kernel and all supporting startup processes, you can move immediately to starting the actual primary process from the moment you say go. This is great if you have a scenario where a database is too big to run multiple copies for testing but an application that relies on that database would be destructive if two copies were using the same database instance (such as network monitoring for example).

Scalability

Since the general design intention of a container is to one a single primary process such as a database server, an HTTP server, or maybe an SSH server as examples, this by design lends itself very well to the modern micro-service architecture that has taken hold in the application development world. A micro-service architecture is seemingly the most ideal design choice for most environments today because not only does it better isolate application component failures thereby limiting the effects of an outage, it also makes it much easier to scale applications more efficiently.

Consider the design of the original eBay architecture. It followed the old school principal of monolithic design where you had one ginormous application that couldn’t be ran on multiple machines very well. This meant that as user demand grew, the underlying hardware running the application had to grow as well. At different points in history, this quickly became a problem because hardware has only ever been so powerful at any given time. If the best CPU at the time only offered 8 cores and the best motherboards could only handle 4 CPUs, then you have quickly found a limit to just how far that application can scale. This scenario illustrated one of the many great examples why micro-service architecture had to become the new approach moving forward. It was the only realistic way to eventually to to the scale that some websites today run at such as eBay, Facebook, Twitter, Instagram, etc…

Possibly to your surprise, just about every major tech company today is making use of not only micro-service architecture but containerization as well. Examples include all of the major video streaming companies like Netflix, Hulu, Amazon Prime, etc.. as well as some of the biggest telecommunications providers in the world like Comcast, Cox, Charter, Verizon, AT&T, etc… This new model of application design has really changed the landscape for efficient scalability.

Even before containers, large organizations would use automation to grow and shrink the underlying resources of applications to handle spikes and valleys in user demand. Imagine having an application that would see a 5000% traffic growth at roughly the same time every day on a daily basis. Think about people returning home from work and turning on the latest episode of a popular series that just came out that day. Before containers, this meant that a lot of virtual machines would be created and/or started simultaneously to meet the predictable demand of a hungry user base. Starting that many VMs at one time can really put some strain on infrastructure, especially the underlying storage systems. Containers have really curved that strain widely since there is far less required to be retrieved from an underlying storage system during process start-up. This means creating 1,000 copies of a process is far less demanding on physical hardware than creating 1,000 VMs to run that single process.

Networking

The advent of containerization also provided the ability to take new, more unique approaches to solving a number of networking challenges that many large scale applications like Facebook and others have encountered inside of data centers. You can run all the fiber and switches you want, but depending on how your application is designed, you can still create massive traffic jams across data center networks. Since container orchestration quickly became much more advanced to meet the growing engineering demands of containerized applications, this paved the road for some new approaches to mitigating these types of traffic jams.

Traditionally, it wouldn’t be uncommon to see data center designs that kept all of the HTTP services on one cluster of servers and all of the database services on a totally different set of servers. Naturally, this meant there often had to be a very hefty fiber optic and switch network to interconnect the clusters. This can quickly create other scalability issues where throwing money at the problem simply won’t resolve the issues.

Enter a more modern approach where now a company might use advanced container orchestration to ensure that anytime a copy of an HTTP service is deployed to a physical host, that a copy of the supporting database server also gets deployed on the same physical machine along side the HTTP service. This means that requests served by the HTTP service no longer require the network communication to the database service to leave the physical machine and traverse an expensive and hard to maintain fiber optic network.

In Closing

I could go on with many other use case examples and detailed explanations but I imagine you get the general idea. Just as when the advent of virtual machines happened and took the computing world by storm, containers have done the same thing thereby taking us all on another huge leap into the future of large scale computing. I realize that many of my examples were based on large scale deployments of huge applications but it’s important to remember that many of these same benefits translate to smaller organizations as well. After all, automation is automation no matter what the scale. If a human doesn’t have to handle the task or a task is less error prone, then there is still a quantifiable value proposition.

There are many different container images available for various DNS recursors in the two primary repositories Docker Hub and RedHat Quay. The problem is that many are poorly implemented, not well maintained, and often lack pseudo-standard deployment features that really put that shine on the final product. Often times, even when a product vendor provides their own container images directly, they still are quite lackluster to say the least. For this reason, I will be using container images that I have built myself and are being actively maintained for my own production use.

To get started, be sure to check out the specific image for this tutorial in my GitHub repository which can be found here. This tutorial will assume that you have both a functioning Docker environment as well as the docker-compose tool installed. There are a few ways you can deploy images to a Docker environment but I am going to only cover the docker-compose route as it’s most practical for ongoing maintenance and also translates very well to the Portainer GUI app designed for managing container environments.

Let’s start by creating a YAML formatted file for docker-compose. I will go over some of the various settings in this file following the example so you may want to review those first and tweak the following example accordingly. From a command shell in your Docker environment. execute the following command:

echo 'version: '3.3'
services:
  recursor:
    image: azoriansolutions/powerdns-recursor:latest
    restart: unless-stopped
    environment:
      - PDNS_allow_from=10.0.0.0/8,123.45.67.0/23
      - PDNS_local_port=53
      - PDNS_local_address=0.0.0.0
      - PDNS_any_to_tcp=yes
      - PDNS_api_key=SECURE-API-KEY
      - PDNS_dnssec=validate
      - PDNS_dnssec_log_bogus=yes
      - PDNS_loglevel=3
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=0.0.0.0/0
      - PDNS_webserver_password=SECURE-HTTP-AUTH-PASSWORD
    ports:
      - "8053:53/udp"
      - "8053:53"
      - "8080:8082"' | tee /tmp/dc-powerdns-recursor.yaml

One of the first and most important settings to take notice of is the “PDNS_allow_from” setting. This defines one or more networks in CIDR format that the recursor will accept DNS queries from. The default value of the setting only allows access from RFC 1918 private IP addresses so you may need to update this if you will be receiving traffic from some public IP networks. You should never set this value to “0.0.0.0/0” unless you know what you’re doing. Otherwise, in many scenarios this can lead to an open resolver which can and will eventually be turned into an attack vector by bad actors. If you are implementing proper firewall filters somewhere along the communication chain, then it would be okay to change this to allow all traffic for ease of configuration maintenance.

Another important setting is the “PDNS_api_key” which controls access to the built-in API server which can be used for issuing various commands to the recursor at runtime. This setting isn’t important if you end up disabling the built-in web-server.

You should definitely take note to the “PDNS_dnssec” setting as this example has configured the most strict form of DNSSEC which is full blown validation. This may have adverse effects depending on how you intend to use the recursor. For more information on this setting, check out the recursor settings page here.

Next up is the “PDNS_webserver” setting. This controls whether or not the built-in web server is enabled. This web server provides a basic page to see real-time performance statistics of the recursor. The web server is also responsible for providing the API server so you must enable it to use the API features.

Also to go with the last setting, if you do enable the web server you may want to consider adjusting the “PDNS_webserver_allow_from” setting. This defines one or more networks in CIDR format which HTTP requests to the web server should be allowed from. If you are performing packet filtering upstream then you are okay to leave this to allow all traffic for ease of configuration maintenance.

Lastly, take notice to the “PDNS_webserver_password” setting which configures a basic password used for HTTP authentication to the real-time statistics page provided by the web server.

Depending on your environment and what you have deployed to the Docker server you’ll be using, you may want to consider changing the port settings:

    ports:
      - "8053:53/udp"
      - "8053:53"
      - "8080:8082"

The value “8053” defines the host port on the Docker machine to bind the internal container port of 53 to. The other value of “8080” defines the host port on the Docker machine to bind the container’s internal web server port of 8082 to. If you aren’t using any form of reverse proxy to manage ingress traffic to your containerized services and just want to have the Docker machine handle requests directly, then you may consider changing the “8053” values to the standard port of “53”.

You need to be aware that in many environments such as Ubuntu, you can not just change to port 53 alone as the container will fail to start. This is because newer Ubuntu releases have a local resolver running by default on port 53 but it is only bound to the loopback address. Docker has a great feature to work around this challenge though. You can also add a specific IP address before the host port number to keep from binding to all available interfaces on the machine. For example, if your Docker host has a public IP address of “123.45.67.76” then you could change the port configuration to “123.45.67.76:53:53/udp” and “123.45.67.76:53:53” respectively. This will prevent conflicts with the local resolver that is running by default on Ubuntu.

At this point, you should have tweaked the example configuration either before or after creating the YAML file so presumably you’re ready to deploy the recursor now. Execute the following command:

docker-compose -u /tmp/dc-powerdns-recursor.yaml

Once the deployment process has been completed, you should now have a working recursor ready to serve DNS requests. There are many more settings that you can change to tweak the recursor behavior. For more information, check out the recursor settings page here.