1

After upgrading a rhel 5 server today I rebooted to the new kernel: curr=2.6.18-371.el5PAE prev=2.6.18-348.18.1.el5PAE.

In the boot sequence I saw a message indicating Logical Volume Management was starting and then almost immediately I saw this and was offered a rescue shell:

Found duplicate PV BPF...ayV: using /dev/sdc1 not /dev/md3.

Note: /dev/sdc1 and /dev/sdb1 are members of the raid1 array /dev/md3.

From this, I assumed that the lvm2 software thinks that /dev/sdc1 and /dev/md3 are pv's with the same UUID and that the lvm2 software was choosing to ignore /dev/md3 and using /dev/sdc1.

I powered down and unplugged the drive for sdc and restarted. Unexpectedly, the system booted without me noticing any problem. Of course, md3 was degraded.

I powered down, plugged in the drive that I unplugged, rebooted, and the system started again without me noticing any problem. Of course, md3 was still degraded but something unexpected happened.

The filesystem within the troubled logical volume was mounted.

I executed pvdisplay and saw the same above error. Of course, when I tried to add sdc1 back into md3 it wouldn't let me because it was in use by the lvm2 software.

I unmounted the filesystem and ran e2fsck on the lv device path. No problems there (but there should have been problems).

There are really four related questions (sorry). Assuming 3's answer is 'yes or sorta', then the answer to 4 is what I need. I asked the first two because I'm assuming that I need to understand their answers to make sense of any answer to the last two.

  1. Why is the filesystem ok if the logical volume was originally made up of a pv of /dev/md3 instead of /dev/sdc1?

  2. Shouldn't the /dev/sdc1 be different from /dev/md3 so as to prevent the logical volume from being consistent with respect to the physical volumes within in? This might be answered by question 1.

  3. Can I fix my problem by removing the pv information from /dev/sdc1 and adding /dev/sdc1 back into /dev/md3?

  4. If the answer to #3 is yes, then how do I go about it without trashing the logical volume and its filesystem?

Some history:

I've never executed "pvcreate /dev/sdc1" so I have no idea why this should be happening. It is true, however, that /dev/sdc has been troubling me lately in that smartmon (sp?) will tell me it can't read the smart data or it can't even see the device. I'll fix the problem either by (a) rebooting, (b) reboot+bios hang+power down+reset sata cable+power on, or sequence b but replace the sata cable instead of just reseating it.

Jeff Holt
  • 11

2 Answers2

2

  1. I'm not sure you asked the question you think you asked, but /dev/md3 is the same as /dev/sdb1 and /dev/sdc1 since it's a mirror set.

  2. No, it shouldn't.

  3. No, that will create data loss for you.

  4. N/A

You can probably get rid of this error message by modifying your /etc/lvm.conf file to change the filter to reject sdb* and scd* devices, regenrate your initrd, then reboot.

John
  • 9,208
  • 1
  • 32
  • 34
1

The fundamental problem is that the array was created with an MD superblock at the end, which means the superblocks at the start are still recognisable at their expected offsets. The only thing that prevents the PV superblock from being parsed is that the MD subsystem grabs the devices first; usually. Sometimes upper layers take care to yield when another superblock is also detectable, but that can be fragile.

There are two ways to avoid this.

  • Create the array with --metadata=1.2, which is the default since 2010. The PV superblock will be shifted by 512k, and won't be recognisable on unassembled devices
  • Use LVM's MD integration. Specify --type=raidXX to lvcreate or lvconvert. LVM doesn't expose unassembled devices.

Normally these precautions are taken at creation time, but in your case (raid1 with metadata at the end, containing a PV), you can convert to LVM-integrated MD without too much trouble.

Once you've made sure the array is synced and the filesystem mostly sane, you can disassemble it, nuke the raid superblocks on both disks (read the wipefs manpage carefully, you don't want to nuke PV superblocks by mistake), nuke the PV superblock on one member only, extend your VG onto that, and lvconvert your logical volumes to --type=raid1 --mirrors=1. Finally, re-run grub-install onto both disks.

Gabriel
  • 271