2

On a Xen 4.8 hypervisor, running on Debian Stretch, I am trying to remove the logical volume of a domU that is no longer used.

I stopped the domU and successfully removed the logical volume for the swap file.

However, when trying to remove the logical volume of the main disk, I get the error:

Logical volume xen3-vg/vmXX-disk in use.

As I need the disk space on the hypervisor for other domUs, I successfully resized the logical volume to 4 MB.

To make it obvious which logical volume needs to be deleted, I renamed the logical volume to "xen3-vg/deleteme".

Nevertheless:

> lvremove -vf /dev/xen3-vg/deleteme

Returns:

Logical volume xen3-vg/deleteme in use.

On searching, I have tried all the following, but to no avail.

> lvdisplay

--- Logical volume ---
LV Path                /dev/xen3-vg/deleteme
LV Name                deleteme
VG Name                xen3-vg
LV UUID                iL2tB4-4tjc-0dlj-ibBX-nQup-oZsX-g81XXO
LV Write Access        read/write
LV Creation host, time xen3, 2017-01-04 07:16:26 +0100
LV Status              available
# open                 1
LV Size                4.00 MiB
Current LE             1
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     256
Block device           254:7

> dmsetup info -c | grep deleteme

xen3--vg-deleteme   254   7 L--w    1    1      3 LVM-aAW4aSeLjqJPPWlF1s1WxAgzeXAjWmXiiL2tB44tjc0dljibBXnQupoZsXg81XXO

> lvchange -an -v /dev/xen3-vg/deleteme

Logical volume xen3-vg/deleteme in use.

> lvremove -vf /dev/xen3-vg/deleteme

Logical volume xen3-vg/deleteme in use.

> umount /dev/xen3-vg/deleteme

umount: /dev/xen3-vg/deleteme: not mounted

> lsof | grep "254,7"

[no output]

> lsof | grep deleteme

[no output]

> fuser /dev/xen3-vg/deleteme

[no output]

If someone could offer advice on how to remove xen3-vg/deleteme I would be very happy.

Rebooting the hypervisor is the final option (to be executed out of office hours), but I would prefer not rebooting.

Thank you in advance.

Thank you, Brandon for your suggestion.

Following the instructions at http://naveen161089.blogspot.com/2014/03/forcefully-remove-lvm.html I tried the following:

> dmsetup ls

[..]
xen3--vg-deleteme   (254:7)
[..]

> dmsetup info -c xen3--vg-deleteme

Name              Maj Min Stat Open Targ Event  UUID                                                                
xen3--vg-deleteme 254   7 L--w    1    1      3 LVM-aAW4aSeLjqJPPWlF1s1WxAgzeXAjWmXiiL2tB44tjc0dljibBXnQupoZsXg81XXO

> dmsetup remove xen3--vg-deleteme

device-mapper: remove ioctl on xen3--vg-deleteme failed: Device or resource busy
Command failed

> lvremove -f /dev/xen3-vg/deleteme

Logical volume xen3-vg/deleteme in use.

I already followed the instructions at: Can't remove open logical volume as stated in my original post.

Neither approach solved the problem.

Does anyone have any further suggestions?

MacMartin
  • 388
Jonathan Maron
  • 121

5 Answers5

2

I think the main mistake was to execute lsof/fuser on the device-file "/dev/xen3-vg/deleteme" and not on the mount-point (e.g.: "/mnt/myDataMount").

(Thats misleading and I also tripped into it: but it's the same, as if you wanted to run lsof on "/dev/sda3" - that does not work. But you can run lsof on /mount/somedrive)

The steps to do in short:

  1. search the "LV path" in the mounts with mount |grep deleteme # where "deleteme" stands for the lv-name. this exemplary outputs something like "/my/mountpoint"
  2. lsof /my/mountpoint to find out which process is using it
  3. kill the process (kill -9 naggingprocess) or stop the service (service something stop)
  4. check (2.) again
  5. umount /dev/xen3-vg/deleteme
  6. lvremove /dev/xen3-vg/deleteme (or lvremove -f)
MacMartin
  • 388
2

I had managed to forget I had associated a loop device with the LV:

lrwxrwxrwx 1 root root       7 Feb  1 14:17 /dev/mapper/vg--main-test -> ../dm-8
#losetup -a
<snip>
/dev/loop7: [0005]:3088 (/dev/dm-8)

by doing:

#losetup -d /dev/loop7

I could then remove the LV

root@virtual3:/dev# lvremove vg-main/test
Do you really want to remove and DISCARD active logical volume vg-main/test? [y/n]: y
  Logical volume "test" successfully removed

Stupid I know.. but it took me half a day to realise. lsof doesn't discover this problem.

David
  • 111
  • 1
  • 2
  • 7
1

I had the same problem and I have used this command and fixed it. Give it a try.

kpartx -dv /dev/mapper/VG-XXX_rootimg
Then lvremove -f /dev/mapper/VG-XXX_rootimg
Hannan
  • 49
0

I had a similar problem, there was an rsync process stuck on using a snapshot, that was already lazily unmounted. So:

  • Because being unmounted, it did not show up on a mount command.
  • Because the umount was lazy, the process that had an open file could continue to have it open. However, it was not connected to the mount point (/mnt/snapshot) anymore, so I did not wind it with lsof | grep snapshot.
  • It seems that because the snapshot used up all allocated space, I/O was totally blocked and rsync was stuck waiting, thus keeping the file descriptor open, thus not letting the lazy unmount finish.

Of course I did not know this back then, but I happened to try to get the device ID using lvdisplay:

lvdisplay big/db_snapshot_20231004.0505 | grep Block.device
   Block device           253:3

And then search for the culprit with lsof:

lsof |grep 253,3
rsync     30508                             root  cwd       DIR              253,3       4096      45059 /postgresql/14/main
rsync     30508                             root    3r      REG              253,3 1073741824      46124 /postgresql/14/main/base/16466/735757683.4

(Those are not valid paths at the end, by the way.)

So after killing the rsync process, lvremove worked!

P.Péter
  • 649
0

In the recent Christmas holidays, I had the opportunity of rebooting the Xen 4.8 hypervisor.

Following a successful reboot, I was able to delete the problematic logical volume without issue.

I can only speculate as to what the problem was, but I think that the hypervisor – despite apparently having deleted the domU – still had a reference to the logical volume in memory, hence, it could not be deleted.

Jonathan Maron
  • 121