1

Brief
We are running our workloads in kubernetes on AWS EKS. All our applications write the logs in /var/log/app_logs directory using hostPath. (yes it's not a best practice, this is due to legacy codebase, eventually we want to move to STDOUT).

Problems we face:

  1. We rely on the hourly logrotate to rotate the files under /var/log/app_logs
  2. During the hourly boundary, we tend to lose the loglines
  3. Upon investigating seems like the copytruncate is creating this datalos
  4. Looks like the application is not able to write to the logfile during the logrotation and we are losing those logs during the rotation.
  5. Since these services are running in kubernetes containers/pods, we won't be able to use the stop service, rotate and start the service approach.

The EKS worker nodes are running in Amazon Linux(centos).

The logrotate.conf we use:

"/var/log/app_logs/*.log" {
  create 664 foo foo
  daily
  missingok
  copytruncate
  rotate 7
  compress
  delaycompress
  notifempty
  sharedscripts
  postrotate
    /bin/bash /opt/scripts/upload-logs.sh
  endscript
  }

Is there anything that we could do to avoid this dataloss by tweaking the logrotate config? or is there any similar tools that solves this problem in kubernetes based environments.

dm90
  • 111

0 Answers0