Follow Best Practices
Logging on Kubernetes is split into two main components the first is your Container monitoring and the second is your host monitoring. The techniques for logging on containers are not fundamentally different from logging best practices outside of containers. The problem is most people when switching to a container environment don’t follow best practices and are forced to retrofit. If logging is properly externalized within an application than it should be simple to pull logs within a containerized environment.
The larger problem comes from the increased amount of moving parts in a container environment. This is why centralized logging and monitoring become even more paramount. This phenomenon only increases with the transition to microservices. There is a fairly standard architecture when it comes to how to get logs out of containers and Kubenernetes, but there is no standard for where to put those logs. This becomes an organizational choice. Most of the time it is best to just use the same logging system you are already familiar with.
Logging Vs. Monitoring
These become two parts of the same coin. Logging is important but is only one part of the equation. What you do with the logs is much more important, otherwise, they just create noise. Monitoring is where the value of logs come from. Monitoring is where this conversation can switch from a science to an art. Yes, there are some generic things you should look for such as CPU, Memory, errors, etc. Knowing when to create an alert though is the art portion of this. After all the alerting, and being able to figure out what went wrong is where these systems shine. They should be able to effectively give you a better signal to noise ratio on your infrastructure.
So what are the tools?
Sending the Logs
The second part is where the logs get sent to and aggregated. This one is dependent on your current setup, and is a matter of preference.
Open Source Solutions
The last step is what to do with the data. These tools enable monitoring and alerting for a container environment.