Help, I Have Moved to Containers and Now I’m Blind

Containers and microservices are becoming a very popular option for deploying applications. There are many benefits of containers, faster deployments, reproducibility of environments, cost optimizations, isolation, and flexibility in general.

There is one glaring problem that is seen right after initial deployment, monitoring and troubleshooting is exponentially more complex when it comes to containers. Containers are designed to run programs in an isolated context, and that means that they tend to be opaque environments. Because of this, the same visibility tools we’ve all been using for years are now failing to perform as expected. Now, you suddenly realize you are flying blind.

Container Environment

The first thing you realize when running in a containerized environment is that containers are not like traditional virtual machines. When a container is run, the actual processes inside the container run isolated inside the operating system environment, but they still run in a shared resource context using the same kernel in the host operating system. Containers are not virtual machines, where every machine runs in complete isolation from a resources point of view. All elements in a container can actually be seen running in the kernel, which means that they are visible to standard monitoring tools and operating system tools (ps, top, etc.). In a normal cloud virtual machine, or in a bare-metal server, you can monitor its standard metrics (like cpu, disk i/o, ram usage, network usage) in order to have a quick view of the general machine state. The problem with a container environment is, on a single server (VM or bare metal) you can have hundreds or thousands of “containerized” applications. Each container using its own part of the machine resources in a completely different way. Thankfully, the actual container metrics are registered in the host pseudo file system “sys”, inside the “/sys/fs/cgroup” structure. Also, if running Docker, it contains its own subcommand “docker stats” which shows you, per container, important metrics related to actual hardware resource usage (cpu, ram, network bandwidth and disk I/O).

Monitoring the Environment

Now that we have some insight into the containers, you realize that you still need to see how the overall hosting infrastructure is doing, the VM or bare metal server’s stats. These need to somehow be intertwined with the individual container stats in a meaningful way. Also, in your monitoring solution, find a way to relate the actual host with the containers running inside. It’s not the same to say, “containers X, Y, and Z failed, and I don’t know why” as it is to say, “host A failed, affecting containers X, Y, and Z.” Relate events in a smart way, and you’ll have a better overall vision of your infrastructure when disaster strikes.

Log Handling

Now that we have all the stats monitoring under control, how are we going to handle logs? If your containerized infrastructure uses applications that produce logs in specific locations inside the container “virtualized” file system, then you really need those logs in a place where you can analyze them, or, rotate them outside the host operating system. If you are using Docker, there is already a way to map directories inside the virtualized file system to directories in the host file system. This means, for example, your web server logs can be outside the container, where you can locate them very easily and do whatever you want with them. Also, think about a centralized logs file system where all the logs can be sent and processed.

Author Bio: Joe Goldberg is the Senior Cloud Program manager at CCSI. Over the past 15+ years, Joe has helped companies to design, build out, and optimize their network and data center infrastructure. As a result of his efforts, major gains in ROI have been realized through virtualization, WAN implementation, core network redesigns, and the adoption of cloud services. Joe is also ITIL certified.