Recover from "FAULTED" zfs pool. "-F" option seems to be ignored

Question

On Ubuntu, I recently started trying out ZFS on an external USB hard drive. I now seem to have a corrupted zfs pool, which probably happened simply as a result of me shutting down my computer (with poweroff).

$ zpool import
   pool: zfs
     id: 1234512345123451234
  state: FAULTED
 status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
    The pool may be active on another system, but can be imported using
    the '-f' flag.
   see: http://zfsonlinux.org/msg/ZFS-8000-72
 config:

        zfs                      FAULTED  corrupted data
          usb-Disk_Name_etc-0:0  ONLINE

When trying to import it I get this helpful message explaining that I'll lose about 1 second of data (which would be fine) and that I can attempt recovery with the "-F" option:

$ zpool import zfs
cannot import 'zfs': I/O error
    Recovery is possible, but will result in some data loss.
    Returning the pool to its state as of Thu 12 Mar 2020 00:58:03 GMT
    should correct the problem.  Approximately 1 seconds of data
    must be discarded, irreversibly.  Recovery can be attempted
    by executing 'zpool import -F zfs'.  A scrub of the pool
    is strongly recommended after recovery.

Using the dry-run flag, I get confirmation that -F should work:

$ zpool import -Fn zfs
Would be able to return zfs to its state as of Thu 12 Mar 2020 00:58:03 GMT.
Would discard approximately 1 seconds of transactions.

But, when I try the real thing, it seems to completely ignore the "-F" option and repeats the original message:

$ zpool import -F zfs
cannot import 'zfs': I/O error
    Recovery is possible, but will result in some data loss.
    Returning the pool to its state as of Thu 12 Mar 2020 00:58:03 GMT
    should correct the problem.  Approximately 1 seconds of data
    must be discarded, irreversibly.  Recovery can be attempted
    by executing 'zpool import -F zfs'.  A scrub of the pool
    is strongly recommended after recovery.

I found mention of the "-X" option from this answer: https://serverfault.com/a/645866/74394 but I continue to get exactly the same message. I've tried all of these option combinations, both with the pool name (zfs) and the numerical ID number (like 1234512345123451234), and I always get exactly the same message as above suggesting to use the -F option.

-F -FX -f -fF -fFX

Am I doing something obviously wrong here? ZFS seems incredibly flaky if I've irretrievably lost the whole disk just because of 1 second of data due to a system shutdown! I don't have a backup copy of the pool since I'd only just started trying out ZFS. Thanks for any help.

Gordan Bobić · Accepted Answer · 2020-03-14T21:21:25.543

If that USB device is lying about commits, all bets are off because the writes could be re-ordered, and between write re-ordering and lying about barriers, all bets are off on whether the metadata is trashed.

Try: zdb -e zfs -ul

This will hopefully list some txgs.

Then try: zpool import -f -FX -N -T [txg_number]

where you got the txg_number from zdb above.

Other things to try: Update ZFS to latest 0.8.3. What ships with Ubuntu is quite ancient and there have been a lot of fixes and improvements since that version.

Recover from "FAULTED" zfs pool. "-F" option seems to be ignored

1 Answers1