49

I'm particularly interested in this for looking at the output of oneshot services that run on a timer. The --unit flag is close, but it concatenates all the runs of the service together. The most obvious way I can think of would be to filter on PID, but that makes me worry about PID reuse / services that fork, and getting the last PID is pretty inconvenient. Is there some other identifier that corresponds to a single run of a service, that I could use to filter the logs?

EDIT: I would happily accept an authoritative "no" if that's the real answer.

Jack O'Connor
  • 941

7 Answers7

40

Since systemd version 232, we have the concept of invocation ID. Each time a unit is ran, it has a unique 128 bit invocation ID. Unlike MainPID which can be recycled, or ActiveEnterTimestamp which can have resolution troubles, it is a failsafe way to get all the log of a particular systemd unit invocation.

To obtain the latest invocation ID of a unit

$ systemctl show --value -p InvocationID openipmi
bd3eb84c3aa74169a3dcad2af183885b

To obtain the journal of the latest invocation of, say, openipmi, whether it failed or not, you can use the one liner

$ journalctl _SYSTEMD_INVOCATION_ID=`systemctl show -p InvocationID --value openipmi.service`
-- Logs begin at Thu 2018-07-26 12:09:57 IDT, end at Mon 2019-07-08 01:32:50 IDT. --
Jun 21 13:03:13 build03.lbits openipmi[1552]:  * Starting ipmi drivers
Jun 21 13:03:13 build03.lbits openipmi[1552]:    ...fail!
Jun 21 13:03:13 build03.lbits openipmi[1552]:    ...done.

(Note that the --value is available since systemd 230, older than InvocationID)

Elazar Leibovich
  • 606
16

I'm not sure which timestamp makes the most sense but this works for me. Hopefully there is a better way of working with the timestamps from systemctl show than awk - could not figure out how to control the format of timestamps.

unit=foo.service

ts=$(systemctl show -p ActiveEnterTimestamp $unit)

echo $ts
ActiveEnterTimestamp=Fri 2016-11-11 12:30:01 MST

journalctl -u $unit --since "$(echo $ts | awk '{print $2 $3}')"
Erik Stephens
  • 161
5

You can use the boot flag to fetch only the logs from that boot. for instance 

journalctl _SYSTEMD_UNIT=avahi-daemon.service -b 5
Nikolaidis Fotis
  • 2,052
5

These might help you:

  • journalctl -u foo.service | tail -n 2

    or replace 2 with expected number of lines

  • journalctl -u foo.service --since='2016-04-11 13:00:00'

You can as well combine them to get firstly the last run time timestamp, and then use that timestamp with the --since switch.

Breign
  • 116
  • 8
5

You can use field filters with Journalctl. E.g.

journalctl _PID=1234

Get a list of all the available fields using:

journalctl --fields --unit kubelet

One available field is _PID.

You can get the PID of a running process using pidof or systemctl show --property MainPID <SERVICE_NAME>

So here's how I get the logs from the current Kubernetes kubelet process:

# journalctl --unit kubelet _PID=$(systemctl show --property MainPID kubelet 2>/dev/null | cut -d= -f2) | head

Now tell me why I Kubernetes is so hard to install :-(

richardw
  • 151
1

Addition to the answer of @Elazar:

I had the problem, that for one specific service, the systemctl show command consistently did not output an invocation ID. Another, very similar unit, worked without problems. I could not find out why.

I solved it the following way:

# Get invocation ID from last log entry
invocation_id=$(journalctl -q -o cat --no-pager -n 1 --output-fields INVOCATION_ID -u NAME_OF_UNIT)

Now get log entries (as already described)


journalctl _SYSTEMD_INVOCATION_ID=$invocation_id
Florian Kusche
  • 11
0

journalctl -r | grep -m1 foo.service

nieman
  • 1