2

Background:

I have a 6x 4tb MDADM array on Box 1. Box 1 developed MB/CPU issues that cause a power-down halt about 15-25 minutes into operation, but during the uptime the array is accessible just fine. I need the data off of that array.

I hotwired the PSU to power up the HDDs without the MB/CPU (paperclip, pin 4+5), and wired the HDD's to a controller in Box 2 which is booted with Live Ubuntu. This lets me use Box 2 to see all of the drives and they show up in lsblk just fine.

Problem:

The array is not reassembling. It's saying all devices are missing their MBRs, despite gdisk saying they have MBRs. I've reassembled this array before and it didn't have any issues... why now?

root@ubuntu:~# mdadm --assemble /dev/md0 /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh
mdadm: Cannot assemble mbr metadata on /dev/sdc
mdadm: /dev/sdc has no superblock - assembly aborted

root@ubuntu:~# mdadm --assemble --scan
mdadm: No arrays found in config file or automatically


root@ubuntu:~# mdadm --examine /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh
/dev/sdc:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdd:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sde:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdf:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdg:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdh:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)

Gdisk on each HDD

root@ubuntu:~# gdisk /dev/sdc
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present


Found valid GPT with protective MBR; using GPT.


Command (? for help): q
root@ubuntu:~# gdisk /dev/sdd
GPT fdisk (gdisk) version 1.0.8


Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present


Found valid GPT with protective MBR; using GPT.


Command (? for help): q
root@ubuntu:~# gdisk /dev/sde
GPT fdisk (gdisk) version 1.0.8


Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present


Found valid GPT with protective MBR; using GPT.


Command (? for help): q
root@ubuntu:~# gdisk /dev/sdf
GPT fdisk (gdisk) version 1.0.8


Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present


Found valid GPT with protective MBR; using GPT.


Command (? for help): q
root@ubuntu:~# gdisk /dev/sdg
GPT fdisk (gdisk) version 1.0.8


Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present


Found valid GPT with protective MBR; using GPT.


Command (? for help): q
root@ubuntu:~# gdisk /dev/sdh
GPT fdisk (gdisk) version 1.0.8


Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present


Found valid GPT with protective MBR; using GPT.


Command (? for help): q

SMART metrics for all drives are basically the same as /dev/sdc (they have all been in the array together since day 1):

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   184   175   021    Pre-fail  Always       -       7766
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       489
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   008   008   000    Old_age   Always       -       67555
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       489
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       439
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       2155317
194 Temperature_Celsius     0x0022   124   098   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

What am I missing here? Why can I not reassemble this array?! I just need to put it together and have the machine run for about 2 days to copy off the contents. I have moved this array twice before when the server was upgraded and it was no issue...

Adding Gdisk -l for all drives:

root@ubuntu:/nfs/spruce# gdisk -l /dev/sdc
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present


Found valid GPT with protective MBR; using GPT.
Disk /dev/sdc: 7814037168 sectors, 3.6 TiB
Model: WDC WD40EZRX-00S
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 4BE9E7EF-6811-4225-B6A1-36DA90DE9CA4
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 7814037134
Partitions will be aligned on 2048-sector boundaries
Total free space is 7814037101 sectors (3.6 TiB)


Number  Start (sector)    End (sector)  Size       Code  Name


root@ubuntu:/nfs/spruce# gdisk -l /dev/sdd
GPT fdisk (gdisk) version 1.0.8


Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present


Found valid GPT with protective MBR; using GPT.
Disk /dev/sdd: 7814037168 sectors, 3.6 TiB
Model: WDC WD40EZRX-00S
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 504EAA64-CB71-409A-8409-17C2CAE09FEB
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 7814037134
Partitions will be aligned on 2048-sector boundaries
Total free space is 7814037101 sectors (3.6 TiB)


Number  Start (sector)    End (sector)  Size       Code  Name


root@ubuntu:/nfs/spruce# gdisk -l /dev/sde
GPT fdisk (gdisk) version 1.0.8


Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present


Found valid GPT with protective MBR; using GPT.
Disk /dev/sde: 7814037168 sectors, 3.6 TiB
Model: WDC WD40EZRX-00S
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): F516176B-1DA8-4870-B765-F1D5D1C82C28
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 7814037134
Partitions will be aligned on 2048-sector boundaries
Total free space is 7814037101 sectors (3.6 TiB)


Number  Start (sector)    End (sector)  Size       Code  Name


root@ubuntu:/nfs/spruce# gdisk -l /dev/sdf
GPT fdisk (gdisk) version 1.0.8


Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present


Found valid GPT with protective MBR; using GPT.
Disk /dev/sdf: 7814037168 sectors, 3.6 TiB
Model: WDC WD40EZRX-00S
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): EE127477-19DD-40B6-9012-D75A78914456
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 7814037134
Partitions will be aligned on 2048-sector boundaries
Total free space is 7814037101 sectors (3.6 TiB)


Number  Start (sector)    End (sector)  Size       Code  Name


root@ubuntu:/nfs/spruce# gdisk -l /dev/sdg
GPT fdisk (gdisk) version 1.0.8


Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present


Found valid GPT with protective MBR; using GPT.
Disk /dev/sdg: 7814037168 sectors, 3.6 TiB
Model: WDC WD40EZRX-00S
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): D10CA92C-66B9-47D5-989C-D8A46740A2D0
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 7814037134
Partitions will be aligned on 2048-sector boundaries
Total free space is 7814037101 sectors (3.6 TiB)


Number  Start (sector)    End (sector)  Size       Code  Name


root@ubuntu:/nfs/spruce# gdisk -l /dev/sdh
GPT fdisk (gdisk) version 1.0.8


Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present


Found valid GPT with protective MBR; using GPT.
Disk /dev/sdh: 7814037168 sectors, 3.6 TiB
Model: WDC WD40EZRX-00S
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 4439E365-A85A-428A-8A18-1056662A8195
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 7814037134
Partitions will be aligned on 2048-sector boundaries
Total free space is 7814037101 sectors (3.6 TiB)


Number  Start (sector)    End (sector)  Size       Code  Name
Russell Anthony
  • 243

1 Answers1

2

You disks have each a single partition (probably) covering the entire LBA span. Try assembling the array with this command:

mdadm --assemble /dev/md0 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1

If not working, show the ouput of --examine (rather than --assemble) with the same partitions as above.

EDIT: well, something seems strange here. gdisk -l does not show any partition, but mdadm --examine /dev/sdc does not recognize any RAID disks. If you are sure these disks were really used for a RAID array, you probably have a metadata-less array - ie: an array which needs to be manually put online via mdadm --build. However, for this command to work, you need a complete understanding of the array geometry (level, chunk size, etc), which is missing here.

Another possibility is that the old motherboard used some custom RAID metadata (did you use the BIOS fake-RAID feature?). This should write something to the disks, though.

I strongly suggest reconnecting the disks to the old motherboard and gather as much information you can about your RAID array (ie: cat /proc/mdstat; mdadm --examine; etc).

shodanshok
  • 52,255