3

I am running Linux Mint 20.3 with ZFS (including for /). Up until very recently, whenever I installed or removed packages, it would also create a snapshot so that I could select it when booting to revert a change that might have broken the system.

But it seems there is a limitation of this system, where it can't cope with a large (1200+) number of snapshots. I can't find the issue anymore, but it suggested running for dataset in $(zfs list -t snapshot -o name | grep @autozsys); do sudo zfs destroy "$dataset"; done to delete most snapshots.

Since I didn't need these older snapshots, I gave it a go. I couldn't delete some because they were clones of others and not knowing whether this would make me lose live data, I left it at that.

I later learned about the zsysctl command to manage snapshots... And it seems I broke my system by not using that command but zfs destroy directly.

Now, whenever I apt install or apt remove, I'm getting this error (after a 30s pause): ERROR Service took too long to respond. Disconnecting client.

I think zsysctl is partially broken because running zsysctl show results in the last entry in the lists for bpool and rpool having a date of 0001-01-01 00:00:00 which is obviously wrong:

$ zsysctl show
Name:           rpool/ROOT/ubuntu_uyv8gh
ZSys:           true
Last Used:      current
History:        
  - Name:       rpool/ROOT/ubuntu_uyv8gh@autozsys_nma9lm
    Created on: 2022-11-17 17:43:38
  - Name:       rpool/ROOT/ubuntu_uyv8gh@autozsys_1ayzvs
    Created on: 2022-11-14 13:22:37
  - Name:       rpool/ROOT/ubuntu_uyv8gh@autozsys_yv8nrd
    Created on: 2022-11-11 17:13:10
  - Name:       rpool/ROOT/ubuntu_uyv8gh@autozsys_yu0btn
    Created on: 2022-02-15 15:58:44
  - Name:       rpool/ROOT/ubuntu_6xf31g
    Created on: 2021-10-22 17:13:50
Users:
  - Name:    pierre
    History: 
     - rpool/USERDATA/pierre_gg46xm@autozsys_nma9lm (2022-11-17 17:43:40)
     - rpool/USERDATA/pierre_gg46xm@autozsys_m6wubi (2022-11-17 14:29:58)
     - rpool/USERDATA/pierre_gg46xm@autozsys_17okng (2022-11-17 11:29:23)
     - rpool/USERDATA/pierre_gg46xm@autozsys_dgvgfz (2022-11-17 10:29:19)
     - rpool/USERDATA/pierre_gg46xm@autozsys_5mk4hd (2022-11-16 15:55:03)
     - rpool/USERDATA/pierre_gg46xm@autozsys_z90d2e (2022-11-16 14:54:03)
     - rpool/USERDATA/pierre_gg46xm@autozsys_pfutl5 (2022-11-16 13:54:03)
     - rpool/USERDATA/pierre_gg46xm@autozsys_5dfwde (2022-11-16 12:42:58)
     - rpool/USERDATA/pierre_gg46xm@autozsys_iuusno (2022-11-16 11:42:54)
     - rpool/USERDATA/pierre_gg46xm@autozsys_zzrd7w (2022-11-16 10:41:57)
     - rpool/USERDATA/pierre_gg46xm@autozsys_d4kkrw (2022-11-16 09:41:55)
     - rpool/USERDATA/pierre_gg46xm@autozsys_2ws9fj (2022-11-15 16:06:23)
     - rpool/USERDATA/pierre_gg46xm@autozsys_cfb7iu (2022-11-15 15:06:23)
     - rpool/USERDATA/pierre_gg46xm@autozsys_3rcqfi (2022-11-15 13:35:17)
     - rpool/USERDATA/pierre_gg46xm@autozsys_56mb7s (2022-11-15 12:07:32)
     - rpool/USERDATA/pierre_gg46xm@autozsys_9b068o (2022-11-15 11:06:38)
     - rpool/USERDATA/pierre_gg46xm@autozsys_o610qz (2022-11-15 10:06:35)
     - rpool/USERDATA/pierre_gg46xm@autozsys_fw4jbj (2022-11-15 08:38:21)
     - rpool/USERDATA/pierre_gg46xm@autozsys_j9ow55 (2022-11-14 17:00:43)
     - rpool/USERDATA/pierre_gg46xm@autozsys_mfc8hk (2022-11-14 15:59:43)
     - rpool/USERDATA/pierre_gg46xm@autozsys_1ayzvs (2022-11-14 13:22:39)
     - rpool/USERDATA/pierre_gg46xm@autozsys_y6xypr (2022-11-14 12:57:47)
     - rpool/USERDATA/pierre_gg46xm@autozsys_o0aj49 (2022-11-14 11:57:46)
     - rpool/USERDATA/pierre_gg46xm@autozsys_iljt6z (2022-11-14 10:57:43)
     - rpool/USERDATA/pierre_gg46xm@autozsys_2fmmg5 (2022-11-14 09:56:43)
     - rpool/USERDATA/pierre_gg46xm@autozsys_lxkohx (2022-11-14 08:55:51)
     - rpool/USERDATA/pierre_gg46xm@autozsys_czv8zx (2022-11-13 16:58:50)
     - rpool/USERDATA/pierre_gg46xm@autozsys_8kl96y (2022-11-13 15:57:50)
     - rpool/USERDATA/pierre_gg46xm@autozsys_5ks5h8 (2022-11-13 14:56:50)
     - rpool/USERDATA/pierre_gg46xm@autozsys_5m7wwk (2022-11-13 13:56:50)
     - rpool/USERDATA/pierre_gg46xm@autozsys_40udn4 (2022-11-13 12:28:40)
     - rpool/USERDATA/pierre_gg46xm@autozsys_uagprp (2022-11-13 11:28:39)
     - rpool/USERDATA/pierre_gg46xm@autozsys_4vbncy (2022-11-13 10:28:39)
     - rpool/USERDATA/pierre_gg46xm@autozsys_9buhaa (2022-11-13 09:26:59)
     - rpool/USERDATA/pierre_gg46xm@autozsys_92de9m (2022-11-12 14:48:45)
     - rpool/USERDATA/pierre_gg46xm@autozsys_x5vgcm (2022-11-12 12:00:12)
     - rpool/USERDATA/pierre_gg46xm@autozsys_pm05qo (2022-11-12 10:43:35)
     - rpool/USERDATA/pierre_gg46xm@autozsys_kxd0z1 (2022-11-12 09:43:32)
     - rpool/USERDATA/pierre_gg46xm@autozsys_welapl (2022-11-11 17:18:27)
     - rpool/USERDATA/pierre_gg46xm@autozsys_yv8nrd (2022-11-11 17:13:11)
     - rpool/USERDATA/pierre_gg46xm-rpool.ROOT.ubuntu-6xf31g (2022-11-09 07:17:16)
     - rpool/USERDATA/pierre_gg46xm@autozsys_yu0btn (2022-02-15 15:58:45)
     - rpool/USERDATA/pierre_hntkn5 (0001-01-01 00:00:00)
  - Name:    root
    History: 
     - rpool/USERDATA/root_gg46xm@autozsys_nma9lm (2022-11-17 17:43:40)
     - rpool/USERDATA/root_gg46xm@autozsys_1ayzvs (2022-11-14 13:22:39)
     - rpool/USERDATA/root_gg46xm@autozsys_yv8nrd (2022-11-11 17:13:11)
     - rpool/USERDATA/root_gg46xm-rpool.ROOT.ubuntu-6xf31g (2022-11-09 07:17:16)
     - rpool/USERDATA/root_gg46xm@autozsys_yu0btn (2022-02-15 15:58:45)
     - rpool/USERDATA/root_hntkn5 (0001-01-01 00:00:00)

Running zsysctl save -vv doesn't give much more information:

$ zsysctl save -vv
DEBUG /zsys.Zsys/SaveUserState() call logged as [e560ff96:3c3c6ace] 
DEBUG Check if grpc request peer is authorized     
DEBUG Polkit call result, authorized: true         
DEBUG Didn't receive any information from service in 30s 
ERROR Service took too long to respond. Disconnecting client.

How do I recover from this partially broken state?

Pierre
  • 81
  • 3

1 Answers1

1

You can remove multiple snapshots by using %.

Example:

zfs destroy rpool/USERDATA/pierre_gg46xm@autozsys_nma9lm%autozsys_yv8nrd

This removes everything from autozsys_nma9lm through autozsys_yv8nrd.

If you have clones intermixed amongst the 1200+ snapshots, then it may be worth reevaluating the setup or separating your data into data-specific pools.

I see you've used the zsysctl. I'm not familiar with it, but ZFS is already streamlined with two main commands; zfs and zpool.

There doesn't seem to be much utility to using zsysctl here because it's definitely complicated your setup. It may be an unnecessary abstraction and doesn't have a future on the platform. Consider removing it.

zsys being removed from ubuntu #235

zsys cannot handle large amounts of snapshots #224

(I'd say the same for ZFS root and the high number of snapshots)

ewwhite
  • 201,205