How to resize RAID1 array with mdadm?

Question

I've running Ubuntu 11.04 (kernel 2.6.38-11). I replaced 2x160GB with 2x500GB drives. They are configured as RAID1.

The partition tables show the right sizes. Here's sfdisk:

# sfdisk -d /dev/sdb
# partition table of /dev/sdb
unit: sectors

/dev/sdb1 : start=       63, size=   192717, Id=fd, bootable
/dev/sdb2 : start=   192780, size=  7807590, Id=fd
/dev/sdb3 : start=  8000370, size=968767695, Id=fd
/dev/sdb4 : start=        0, size=        0, Id= 0

And fdisk:

# fdisk -l /dev/sdb

Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006c78f

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          12       96358+  fd  Linux raid autodetect
/dev/sdb2              13         498     3903795   fd  Linux raid autodetect
/dev/sdb3             499       60801   484383847+  fd  Linux raid autodetect

But I'm not seeing the new space:

root@green:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md2              143G  134G  8.3G  95% /


root@green:~# mdadm --examine /dev/sdb3
/dev/sdb3:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : b8f83980:f60d820c:74c46fbf:0baa68bc
  Creation Time : Sun Mar 29 18:48:46 2009
     Raid Level : raid1
  Used Dev Size : 152247936 (145.19 GiB 155.90 GB)
     Array Size : 152247936 (145.19 GiB 155.90 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 2

    Update Time : Mon Oct 10 19:22:36 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 7b5debb7 - correct
         Events : 10729526


      Number   Major   Minor   RaidDevice State
this     0       8       19        0      active sync   /dev/sdb3
   0     0       8       19        0      active sync   /dev/sdb3
   1     1       8        3        1      active sync   /dev/sda3

I tried mdadm and resize2fs:

# mdadm --grow /dev/md2 --size=max
mdadm: component size of /dev/md2 has been set to 152247936K

# resize2fs /dev/md2 
resize2fs 1.41.14 (22-Dec-2010)
The filesystem is already 38061984 blocks long.  Nothing to do!

Any ideas?

Added per request

# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md2 : active raid1 sdb3[0] sda3[1]
      152247936 blocks [2/2] [UU]

md0 : active raid1 sdb1[0] sda1[1]
      96256 blocks [2/2] [UU]

md1 : active raid1 sdb2[0] sda2[1]
      3903680 blocks [2/2] [UU]

unused devices: <none>

partitions

# cat /proc/partitions 
major minor  #blocks  name

   8        0  488386584 sda
   8        1      96358 sda1
   8        2    3903795 sda2
   8        3  152248005 sda3
   8       16  488386584 sdb
   8       17      96358 sdb1
   8       18    3903795 sdb2
   8       19  152248005 sdb3
   9        1    3903680 md1
   9        0      96256 md0
   9        2  152247936 md2

parted:

# parted
GNU Parted 2.3
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print all                                                        
Model: ATA WDC WD5000AAKX-0 (scsi)
Disk /dev/sda: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system     Flags
 1      32.3kB  98.7MB  98.7MB  primary  ext3            boot, raid
 2      98.7MB  4096MB  3997MB  primary  linux-swap(v1)  raid
 3      4096MB  500GB   496GB   primary  ext3            raid


Model: ATA WDC WD5000AAKS-4 (scsi)
Disk /dev/sdb: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system     Flags
 1      32.3kB  98.7MB  98.7MB  primary  ext3            boot, raid
 2      98.7MB  4096MB  3997MB  primary  linux-swap(v1)  raid
 3      4096MB  500GB   496GB   primary  ext3            raid


Model: Linux Software RAID Array (md)
Disk /dev/md1: 3997MB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system     Flags
 1      0.00B  3997MB  3997MB  linux-swap(v1)


Model: Linux Software RAID Array (md)
Disk /dev/md0: 98.6MB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system  Flags
 1      0.00B  98.6MB  98.6MB  ext3


Model: Linux Software RAID Array (md)
Disk /dev/md2: 156GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End    Size   File system  Flags
 1      0.00B  156GB  156GB  ext3

Comment via email:

the problem is in metadata, you just need to assemble raid array with parameter --update devicesize

and after that -G /dev/md? -z max will done job :)

Answer 1

Just use

mdadm --grow --size max /dev/md2

Then you'll be able to use

resize2fs /dev/md2

To let file system match the raid size. All of that is done online without even unmounting the md2.

Answer 2

From looking at /proc/partitions, it's apparent that linux thinks sda3 and sdb3 are smaller than they are.

Sum the sizes of the partitions

8       17      96358 sdb1
8       18    3903795 sdb2
8       19  152248005 sdb3

and you'll get a number much lower than the size of the disk.

8       16  488386584 sdb

152248005 blocks of 1024 bytes is consistent with the size mdadm --grow and resize2fs are reporting for md2.

Did you initially create these partitions with a smaller size, then later recreate them to utilize the rest of the disks? If so, rebooting should allow the kernel to re-read the partition table. After that, growing the RAID device and resizing the filesystem should work.

Answer 3

I regularly use mdadm and consider it one of the most dangerous Linux utilities. However, if you employ the correct safety precautions you can avoid most cases of potential data loss Backup all your data!!! i have been bitten twice by mdadm in the past, lost over 700GB of data and very very little of it was able to be recovered, you have been warned.

There is a very good chance you will need to create the RAID array again as mdadm does not expect or compensate for drives suddenly increasing in size. It will use the size stated in the raid superblock and not the drive itself. Provided that the drives are already synced, you shouldn't have many problems.

Remember if you want to boot of it use superblock version 0.9.

Edit

This is how i would do it, untested!

Create a RAID1 with a missing dive just so we can quickly test that the data remains while still having another drive with a copy of the data, your old metadata was 0.90 so we will keep the same version here.

mdadm --create /dev/md2 --level=mirror --metadata=0.90 --raid-devices=2 missing /dev/sdb3

Mount it to test that everything works

mkdir /mnt/test
mount /dev/md2 /mnt/test

check your data

   ls -l /mnt/test

If it all looks ok then unmount the drive and resize.

unmount /mnt/md2
resize2fs /dev/md2

Once that is all ok you can add the other drive to the array.

mdadm --add /dev/md2 /dev/sdb3

and wait for the drives to resync

cat /proc/mdstat

Answer 4

Let me add my own log, for the record. I found myself in a situation where, after replacing both drives in an ancient MD mirror during the years, I ended up having both drives larger than what the mirror originally consisted of :-) I already had larger physical drives, but the Linux RAID partitions were still the original (=smaller) size. And, space on the MD mirror was running out. Fortunately, I had only one 0xFD partition per drive, which made the expansion relatively uncomplicated.

Here goes a copy of my "internal scrapbook entry":

First, let's get an overview of the initial state of affairs:

df
cat /proc/partitions
cat /proc/mdstat
cat /etc/mdadm/mdadm.conf

Are the partition tables the same?

sfdisk -d /dev/sdh
sfdisk -d /dev/sdi

Just in case, we should "park ourselves" someplace safe:

cd ~

You may want to stop any services that depend on the mountpoint. Such as, if the volume is served by Samba:

systemctl stop nmbd
systemctl stop smbd

If you go for an online resize of the FS, which should be the faster option, apparently unmounting is not even necessary:

umount /array/mountpoint

Now, whether you go for an online of offline resize of the FS, I have confirmed the hard way that, in order to resize the MD device, you actually must not stop it!

#mdadm --stop /dev/md3

You'd better make a backup of the partition table - just to make sure. Obviously this is no replacement for a full backup of all your data, but it gives you a retreat path if for some reason you goof up while modding the partition tables.

If both drives are equal, you can just pick one drive.

sfdisk -d /dev/sdh > sdh.bak

Your first and only partition probably starts at sector 63. Or about 2048 in the modern days. Mind the number 63 in the command below. Whatever it is, just keep that.

The partitions labeled 0xFD (= Linux RAID Autodetect) are in use by the active MD array. This is deliberate, you just need to force fdisk to make the change. I'm using the "scriptable" sfdisk, but you could probably just use the interactive version if this makes you uneasy:

echo -en "unit: sectors\n63,,fd,\n" | sfdisk --force /dev/sdh
echo -en "unit: sectors\n63,,fd,\n" | sfdisk --force /dev/sdi

Note that after specifying just the starting sector (at the start of the second track), we leave the partition size unspecified - that indicates to sfdisk to use all available space. The "fd" is the partition label (Linux RAID Autodetect). The trailing comma indicates another field left at default - an asterisk placed here would indicate an active=bootable partition. The \n is an escape sequence for a newline character. Because the input to sfdisk is actually a multi-line file. See the output of sfdisk -d. Note that I previously used this fact to back up the partition table, and store the backup in a file by simple redirection. That backup file can be just piped into the standard input of another sfisk, to re-create that partition table. You can also edit the file to alter the partitioning layout, and again pipe it into sfdisk to effect the changes...

In the output trace of fdisk, you should see that fdisk has increased the partition size, reports that it has written the new partition table to the disk (because we have forced it), but still has not told the kernel to re-read the partition table from the disk. I.e., the kernel did not re-interpret / update its idea of partition sizes, see also:

cat /proc/partitions

Yet, for the subsequent mdadm --grow --size max to succeed, we need the parttable re-read operation to have happened. Without stopping the MD array! There are several tools that can invoke the needed ioctl() for us, such as:

partprobe /dev/sdh
partprobe /dev/sdi

The same could probably be achieved using:

blockdev --rereadpt /dev/sdh
blockdev --rereadpt /dev/sdi

Re-check the result:

cat /proc/partitions

Then finally, resize the MD device:

mdadm --grow --size max /dev/md3

You'd better wait for the array to finish syncing the grown space:

watch -n 5 cat /proc/mdstat

Once the resync is finished, you may want to check if the array can be stopped and re-started:

mdadm --stop /dev/md3
mdadm --assemble scan --uuid=<check out the mdadm.conf>

or

mdadm --assemble /dev/md3 /dev/sdh1 /dev/sdi1

This step = array stop/start is not necessary - but if it succeeds, you know that you probably (possibly?) haven't lost your data yet :-)

The example below is tailored to the EXT2/3/4 FS.

You may try mounting the MD volume right away, and do the FS resize online. Or you can keep the MD volume unmounted, and resize the FS offline. The various behaviors of resize2fs depending on conditions have been debated here before.

If you try resize2fs offline, i.e. while the MD volume is not mounted, it may ask you to run e2fsck -f first.

e2fsck -f -v /dev/md3

This takes long! The "-v" should give you more clue about progress. And, do not run fsck before the MD sync is finished! If you do this, they'll compete for IOps and will likely take longer than if sequenced!

Then finally:

resize2fs -p /dev/md3

The "-p" option turns a progress indicator on. Then, it's time to mount the

mount /array/mountpoint

And check the free space:

df

And start your services...

You can actually resize while the FS is mounted - except that old versions of ext3 did not support this. I.e. this online resize runs in the background. Offline resize runs in the foreground and can take a couple dozen minutes (expansion. Shrinking can take many hours.) This is where the -p argument (progress indicator) comes really handy :-)

While doing all those resync / fsck / resize2fs, you may appreciate a monitor on your IOps. My favourite is iostat of the sysstat package. E.g. to auto-refresh the view every 2 seconds, use:

iostat 2

You can keep this running in another terminal window, beside the one where you conjure your black curses. I have found another useful "autorefreshing monitor" at the filesystem level, invoked in this way:

watch -n 10 df

I use this while copying files from disk to disk, e.g. when moving a volume to a different filesystem. If that's your case, mind also the noatime mount option.