Can systemd manage a pipeline?

Question

Can systemd manage a pipeline similar to how the daemontools family does it? And if so, what's the best way to achieve this?

I want to run the equivalent of service1 | service2 where both service1 and service2 are (separate or not) services managed by systemd.

I would like to be able to restart the service2 process without interrupting service1. In other words, the file descriptor to which service1 is writing must not be closed when service2 exits. When a new instance of service2 starts, it should inherit the existing file descriptor so that stdout from service1 will flow into the new service2. (Much like daemontools maintains a pipe between run and log/run, though the pipeline need not be a service and a logger.)

Perhaps something with a systemd-managed FIFO in between?

Patrick · Accepted Answer · 2017-01-05T16:44:34.447

Finally had the opportunity and need to work through this myself. My solution requires support for the fd option to StandardOutput=, which is available in (at least) systemd version 232 but not in version 215.

There are three services and two FIFOs. Together they create the pipeline input | filter | output, and any part of the pipeline can be individually restarted without data loss.

The input process writes to a FIFO from which filter reads, which in turn writes to a FIFO that output reads.

input.service

[Unit]
Description=The input process
Requires=filter.socket
After=filter.socket

Wants=filter.service output.service

[Service]
TimeoutStartSec=infinity

Sockets=filter.socket

StandardInput=null
StandardOutput=fd:filter.socket
StandardError=journal
ExecStart=/path/to/input

Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

filter.service

[Unit]
Description=The filter process
Requires=filter.socket output.socket
After=filter.socket output.socket

[Service]
TimeoutStartSec=infinity

Sockets=filter.socket
Sockets=output.socket

StandardInput=fd:filter.socket
StandardOutput=fd:output.socket
StandardError=journal
ExecStart=/path/to/filter

Restart=always
RestartSec=5s

filter.socket

[Unit]
Description=Filter process reads from this

[Socket]
ListenFIFO=/run/filter
SocketMode=0600
RemoveOnStop=false

output.service

[Unit]
Description=The output process
Requires=output.socket
After=output.socket

[Service]
TimeoutStartSec=infinity

Sockets=output.socket

StandardInput=fd:output.socket
StandardOutput=journal
StandardError=journal
ExecStart=output

Restart=always
RestartSec=5s

output.socket

[Unit]
Description=Output process reads from this

[Socket]
ListenFIFO=/run/output
SocketMode=0600
RemoveOnStop=false

score 1 · Answer 2 · answered Aug 27 '16 at 11:58

Have the service write to stdout, and configure StandardOutput in the systemd unit file for the service to write to the journal:

http://0pointer.de/public/systemd-man/systemd.exec.html

This makes logs available to the journald service, which offers other options for log consumption.

http://0pointer.de/public/systemd-man/journald.conf.html

A custom "logger" can be a journald client, and can directly pull from the journal, and if not available the upstream service is of course not impacted. The logger can also be configured with its own unit file so it is managed by systemd.

Can systemd manage a pipeline?

2 Answers2