How to remove missing systemd units?

Question

I'm having trouble figuring out how to remove systemd units that no longer have files. They still seem to linger in the system somehow.

The old broken units I am trying to remove:

core@ip-172-16-32-83 ~ $ systemctl list-units --all firehose-router*
  UNIT                       LOAD      ACTIVE SUB    DESCRIPTION
<E2><97><8F> firehose-router@02.service not-found failed failed firehose-router@02.service
<E2><97><8F> firehose-router@03.service not-found failed failed firehose-router@03.service

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

2 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.

The files do not exist, yet a reload still has these units lingering:

core@ip-172-16-32-83 ~ $ systemctl list-unit-files firehose-router@02.service
core@ip-172-16-32-83 ~ $ sudo systemctl daemon-reload
core@ip-172-16-32-83 ~ $ systemctl list-units --all firehose-router*
  UNIT                       LOAD      ACTIVE SUB    DESCRIPTION
<E2><97><8F> firehose-router@02.service not-found failed failed firehose-router@02.service
<E2><97><8F> firehose-router@03.service not-found failed failed firehose-router@03.service

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

2 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.

There are no files related to them that I can find:

core@ip-172-16-32-83 ~ $ sudo find /var/run/systemd -name "*firehose-router*"
core@ip-172-16-32-83 ~ $ find /etc/systemd/ -name "*firehose-router*"
core@ip-172-16-32-83 ~ $ find /usr/lib/systemd/ -name "*firehose-router*"
core@ip-172-16-32-83 ~ $

So how do I get rid of these?

Answer 1

The command you're after is systemctl reset-failed

Answer 2

• failed - happens when a unit has entered a failed state and can be reset with the systemctl reset-failed command
• not-found - happens when you've removed a unit but systemd still has a reference to it, like when a unit gets enabled and a symlink gets placed in /etc/systemd/system, this can be fixed by removing references to the unit in /etc/systemd/system/*.wants/ then running systemctl daemon-reload

For example, assume the following bash script:

#!/bin/bash
# script.sh
while true
do
    sleep 1
done

And three systemd units: example-foo.service, example-bar.service, and example-baz.service

$ sudo systemctl cat example-{foo,bar,baz}.service
# /etc/systemd/system/example-foo.service
[Service]
ExecStart=/home/vagrant/script.sh
[Install]
WantedBy=multi-user.target
/etc/systemd/system/example-bar.service
[Service]
ExecStart=/home/vagrant/script.sh
[Install]
WantedBy=multi-user.target
/etc/systemd/system/example-baz.service
[Service]
ExecStart=/home/vagrant/script.sh
[Install]
WantedBy=multi-user.target

Now, let's start and enable the units. Observe how symlinks get created.

$ sudo systemctl start example-{foo,bar,baz}.service
$ sudo systemctl enable example-{foo,bar,baz}.service
Created symlink from /etc/systemd/system/multi-user.target.wants/example-foo.service to /etc/systemd/system/example-foo.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/example-bar.service to /etc/systemd/system/example-bar.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/example-baz.service to /etc/systemd/system/example-baz.service.

Confirm there are actually six files for our three units.

$ find /etc/systemd/system -name 'example*.service'
/etc/systemd/system/multi-user.target.wants/example-bar.service
/etc/systemd/system/multi-user.target.wants/example-foo.service
/etc/systemd/system/multi-user.target.wants/example-baz.service
/etc/systemd/system/example-bar.service
/etc/systemd/system/example-foo.service
/etc/systemd/system/example-baz.service

Now, check the state of all three units, they're running.

$ systemctl list-units example*
UNIT                LOAD   ACTIVE SUB     DESCRIPTION
example-bar.service loaded active running example-bar.service
example-baz.service loaded active running example-baz.service
example-foo.service loaded active running example-foo.service

Now, simulate a failure by sending a SIGKILL to example-foo.service. Observe how the unit is in a failed state.

$ sudo systemctl kill -s KILL example-foo.service
$ systemctl list-units example*
  UNIT                LOAD   ACTIVE SUB     DESCRIPTION
  example-bar.service loaded active running example-bar.service
  example-baz.service loaded active running example-baz.service
● example-foo.service loaded failed failed  example-foo.service

To reset a unit in a failed state use the systemctl reset-failed command. Observe how the unit is now in an inactive state.

$ sudo systemctl reset-failed
$ systemctl list-units example*
UNIT                LOAD   ACTIVE SUB     DESCRIPTION
example-bar.service loaded active running example-bar.service
example-baz.service loaded active running example-baz.service
...
$ systemctl list-units --all example*
UNIT                LOAD   ACTIVE   SUB     DESCRIPTION
example-bar.service loaded active   running example-bar.service
example-baz.service loaded active   running example-baz.service
example-foo.service loaded inactive dead    example-foo.service

Okay, now let's remove the example-bar.service unit. Observe how the unit is in a not-found state; however, the example-bar.service broken symlink is still in /etc/systemd/system/multi-user.target.wants

$ sudo rm /etc/systemd/system/example-bar.service
$ sudo systemctl daemon-reload
$ sudo systemctl stop example-bar.service
Failed to stop example-bar.service: Unit example-bar.service not loaded.
$ systemctl list-units --all example*
  UNIT                LOAD      ACTIVE   SUB     DESCRIPTION
● example-bar.service not-found inactive dead    example-bar.service
  example-baz.service loaded    active   running example-baz.service
  example-foo.service loaded    inactive dead    example-foo.service
$ find /etc/systemd/system -name 'example*.service'
/etc/systemd/system/multi-user.target.wants/example-bar.service
/etc/systemd/system/multi-user.target.wants/example-foo.service
/etc/systemd/system/multi-user.target.wants/example-baz.service
/etc/systemd/system/example-foo.service
/etc/systemd/system/example-baz.service

Remove the broken symlink and confirm the example-bar.service unit is gone.

$ sudo rm /etc/systemd/system/multi-user.target.wants/example-bar.service
$ sudo systemctl daemon-reload
$ systemctl list-units --all example*
UNIT                LOAD   ACTIVE   SUB     DESCRIPTION
example-baz.service loaded active   running example-baz.service
example-foo.service loaded inactive dead    example-foo.service

Answer 3

When systemd analyzes unit definition files, it takes note of any other related units called out in the file - whether those other units exist or not.

$ systemctl --state=not-found --all
> ( ...prints list of 'not-found' units )

$ grep -r "<missing-unit>" /usr/lib/systemd/system
> ( returns files with references to <missing-unit> )

When a unit shows up as "not-found", it's not necessarily an error - all we know is, a local unit definition claims to have some relationship with it. This relationship might not be one we care about. For example, it could be "Before:" some other unit, but we don't use that other unit.

Answer 4

It seems that systemd maintains links but does not know what to do with them when you delete the unit file.

You could try to remove them manually in /etc/systemd/system/suspend.target.wants/ and such but of course systemctl reset-failed from a previous answer sounds like a better option.

$ cd /etc/systemd/system
$ sudo mv lock.service /tmp 
$ sudo systemctl disable lock.service
Failed to disable unit: No such file or directory
$ sudo mv /tmp/lock.service .
$ sudo systemctl disable lock.service
Removed /etc/systemd/system/suspend.target.wants/lock.service.