29

We have an application that writes three types of logs into three separate files: access logs, generic application logs and system logs. The format (and purpose) of these logs are very different. And we have separate logforwarders that send them separately to our centralised logging system.

Based on the treat logs as event streams principle, we are thinking about moving from using files to stdout. While we know some of the benefit of this approach, this would also mean that we'd get a merged stream of the differently formatted logs, which we would need to split again either before we can send them to our central system (Kibana / Splunk / etc.), or inside there.

We're wondering whether there are any tools or recommendations on how we should approach this situation.

SztupY
  • 1,597
  • 1
  • 16
  • 18

2 Answers2

17

I'm still looking for a merging/splitting approach myself, but meanwhile this approach recommended by Kubernetes documentation seems like a sound solution: Use a sidecar container for each of your separate logs.

A "sidecar" is any docker container that you use alongside another docker container to work with it in some way. In this case for each of your three logs you would have a separate container that scans or tails the logs and outputs to stdout.

This way each of your log-sidecar-containers has its own docker-log from its own stdout. Being separate like this, you can use standard docker (and kubernetes, etc) practices for separating or aggregating. Here's what the Kubernetes page has to say:

This approach allows you to separate several log streams from different parts of your application, some of which can lack support for writing to stdout or stderr. The logic behind redirecting logs is minimal, so it’s hardly a significant overhead. Additionally, because stdout and stderr are handled by the kubelet, you can use built-in tools like kubectl logs.

The "separate log streams" stem from the built-in tagging that docker applies to logs from different containers, described in the docker documentation here:

The tag log option specifies how to format a tag that identifies the container’s log messages. By default, the system uses the first 12 characters of the container id. To override this behavior, specify a tag option

Naliba
  • 103
  • 2
Rhubarb
  • 286
  • 2
  • 2
8

The idea of merging them into one stream just to split them later sounds painful. I've not had a reason to do this myself, but here's where I'd start:

  • When you run the docker container, create a volume such that it will be easy to view/ship logs from the host.
  • Use something like remote_syslog2 to ship the logs to your log collector

It feels a little less-than-elegant to have to do some setup on the host as well, but if you use something like ansible where you can run a playbook and set that up during your deploy to the box it shouldn't be too bad.

bradym
  • 556
  • 2
  • 8