RAID1 NAS drive still reporting No space left on device even after adding new disk

Question

I have an issue with my NAS drive reporting No space left on device.

On inspection, I found that mdadm reported the filesystem was degraded due to one of the two disks had failed. I replaced the disk and the array rebuilt, which it completed after 24 hours.

All the outputs from df, mdadm etc indicate the file system is healthy and there should be 1.6 TB of free space.

But trying to save a file, even a small one results in No space left on device.

Diagnostics outputs follow. The RAID1 filesystem is mounted on /mnt/pools/A/A0.

Space Check:

df -h
Filesystem            Size  Used Avail Use% Mounted on
rootfs                 50M  3.0M   48M   6% /
/dev/root.old         6.5M  2.1M  4.4M  33% /initrd
none                   50M  3.0M   48M   6% /
/dev/md0_vg/BFDlv     4.0G  624M  3.2G  17% /boot
/dev/loop0            592M  538M   54M  91% /mnt/apps
/dev/loop1            4.9M  2.2M  2.5M  47% /etc
/dev/loop2            260K  260K     0 100% /oem
tmpfs                 122M     0  122M   0% /mnt/apps/lib/init/rw
tmpfs                 122M     0  122M   0% /dev/shm
/dev/mapper/md0_vg-vol1
                       16G  1.5G   15G  10% /mnt/system
/dev/mapper/5244dd0f_vg-lv58141b0d
                      3.7T  2.0T  1.7T  55% /mnt/pools/A/A0

i-node check:

df -ih
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
rootfs                   31K     565     30K    2% /
/dev/root.old           1.7K     130    1.6K    8% /initrd
none                     31K     565     30K    2% /
/dev/md0_vg/BFDlv       256K      20    256K    1% /boot
/dev/loop0               25K     25K      11  100% /mnt/apps
/dev/loop1              1.3K    1.1K     152   88% /etc
/dev/loop2                21      21       0  100% /oem
tmpfs                    31K       4     31K    1% /mnt/apps/lib/init/rw
tmpfs                    31K       1     31K    1% /dev/shm
/dev/mapper/md0_vg-vol1
                         17M    9.7K     16M    1% /mnt/system
/dev/mapper/5244dd0f_vg-lv58141b0d
                        742M    2.6M    739M    1% /mnt/pools/A/A0

RAID Status:

mdadm --detail /dev/md1
/dev/md1:
        Version : 01.00
  Creation Time : Mon Mar  7 08:45:49 2011
     Raid Level : raid1
     Array Size : 3886037488 (3706.01 GiB 3979.30 GB)
  Used Dev Size : 7772074976 (7412.03 GiB 7958.60 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1
    Persistence : Superblock is persistent
Update Time : Thu Jan 30 03:16:36 2025
      State : clean

Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Name : ix2-200-DC386F:1
       UUID : 8a192f2c:9829df88:a6961d81:20478f62
     Events : 365739

Number   Major   Minor   RaidDevice State
   3       8       18        0      active sync   /dev/sdb2
   2       8        2        1      active sync   /dev/sda2

Partitions :

sfdisk -l /dev/sda
WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util sfdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sda: 486401 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1          0+      -       0- 2147483647+  ee  EFI GPT
        start: (c,h,s) expected (0,0,2) found (0,0,1)
/dev/sda2          0       -       0          0    0  Empty
/dev/sda3          0       -       0          0    0  Empty
/dev/sda4          0       -       0          0    0  Empty
sfdisk -l /dev/sdb
WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util sfdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sdb: 486401 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdb1          0+      -       0- 2147483647+  ee  EFI GPT
        start: (c,h,s) expected (0,0,2) found (0,0,1)
/dev/sdb2          0       -       0          0    0  Empty
/dev/sdb3          0       -       0          0    0  Empty
/dev/sdb4          0       -       0          0    0  Empty

pvs, lvs, vgs outputs:

pvs
  PV         VG          Fmt  Attr PSize  PFree
  /dev/md0   md0_vg      lvm2 a-   20.01G    0 
  /dev/md1   5244dd0f_vg lvm2 a-    3.62T    0
lvs
  LV         VG          Attr   LSize  Origin Snap%  Move Log Copy%  Convert
  lv58141b0d 5244dd0f_vg -wi-ao  3.62T

  BFDlv      md0_vg      -wi-ao  4.00G

  vol1       md0_vg      -wi-ao 16.01G
vgs
  VG          #PV #LV #SN Attr   VSize  VFree
  5244dd0f_vg   1   1   0 wz--n-  3.62T    0 
  md0_vg        1   2   0 wz--n- 20.01G    0

lvdisplay, pvdisplay and vgdispaly outputs:

lvdisplay 
  --- Logical volume ---
  LV Name                /dev/5244dd0f_vg/lv58141b0d
  VG Name                5244dd0f_vg
  LV UUID                hLUJyo-C8ge-1SRc-gvdg-dlLn-5A7e-D55XeO
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                3.62 TB
  Current LE             948739
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2
--- Logical volume ---
  LV Name                /dev/md0_vg/BFDlv
  VG Name                md0_vg
  LV UUID                N48AUD-nucp-gP18-wmQi-Ym12-3G1L-7wE1jd
  LV Write Access        read/write
  LV Status              available
open                 1
LV Size                4.00 GB
  Current LE             1024
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

currently set to     256

Block device           253:0
--- Logical volume ---
  LV Name                /dev/md0_vg/vol1
  VG Name                md0_vg
  LV UUID                73CtS1-b4KB-cLrG-hgUz-Qsli-Mlte-QhaUzy
  LV Write Access        read/write
  LV Status              available
open                 1
LV Size                16.01 GB
  Current LE             4098
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

currently set to     256

Block device           253:1
pvdisplay 
  --- Physical volume ---
  PV Name               /dev/md1
  VG Name               5244dd0f_vg
  PV Size               3.62 TB / not usable 2.30 MB
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              948739
  Free PE               0
  Allocated PE          948739
  PV UUID               mdm9UZ-dhcm-T26Z-LRAB-Pevo-4y7t-OHRDRr
--- Physical volume ---
  PV Name               /dev/md0
  VG Name               md0_vg
  PV Size               20.01 GB / not usable 1.06 MB
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              5122
  Free PE               0
  Allocated PE          5122
  PV UUID               AGH7Ci-jGbF-bLsB-pMKr-IttE-7GtR-e5Rf9k
vgdisplay 
  --- Volume group ---
  VG Name               5244dd0f_vg
  System ID

  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  8
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               3.62 TB
  PE Size               4.00 MB
  Total PE              948739
  Alloc PE / Size       948739 / 3.62 TB
  Free  PE / Size       0 / 0

  VG UUID               FB2tzp-8Gr2-6Dlj-9Dck-Tyc4-Gxx5-HHIsBD
--- Volume group ---
  VG Name               md0_vg
  System ID

  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               20.01 GB
  PE Size               4.00 MB
  Total PE              5122
  Alloc PE / Size       5122 / 20.01 GB
  Free  PE / Size       0 / 0

  VG UUID               EA3tJR-nVdm-0Dcf-YtBE-t1Qj-peHc-Sh0zXe

I'm at a loss as to what do next.

Update:

Number of directories under this filesystem is 280,286.

$ cd /mnt/pools/A/A0/
$ ls
Backups  Documents  Movies  Music  Pictures  QuikTransfer  stls_userdata  TimeMachine
$ find . -type d | wc -l
280286

Update2 - mount command output:

mount
rootfs on / type rootfs (rw)
/dev/root.old on /initrd type ext2 (rw,relatime,errors=continue)
none on / type tmpfs (rw,relatime,size=51200k,nr_inodes=31083)
/dev/md0_vg/BFDlv on /boot type ext2 (rw,noatime,errors=continue)
/dev/loop0 on /mnt/apps type ext2 (ro,relatime)
/dev/loop1 on /etc type ext2 (rw,sync,noatime)
/dev/loop2 on /oem type cramfs (ro,relatime)
proc on /proc type proc (rw,relatime)
none on /proc/bus/usb type usbfs (rw,relatime)
none on /proc/fs/nfsd type nfsd (rw,relatime)
none on /sys type sysfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620)
tmpfs on /mnt/apps/lib/init/rw type tmpfs (rw,nosuid,relatime,mode=755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,relatime)
/dev/mapper/md0_vg-vol1 on /mnt/system type xfs (rw,noatime,attr2,logbufs=8,noquota)
/dev/mapper/5244dd0f_vg-lv58141b0d on /mnt/pools/A/A0 type xfs (rw,noatime,attr2,nobarrier,logbufs=8,noquota)

Example command that fails:

mount > /mnt/pools/A/A0/Documents/mount.txt
-sh: /mnt/pools/A/A0/Documents/mount.txt: No space left on device

score 1 · Accepted Answer · answered Mar 06 '25 at 13:57

After much searching, I found the issue was indeed the i-nodes being exhausted, even though df -i reported lots of free i-nodes.

The cause was something to do with the XFS file systems storing i-node info in the first 1Tb of space. I do not understand fully how this works, but I was able to prove it was a i-node rather than 'space' issues as follows:

Try to create a new file:

$ cd /mnt/pools/A/A0
$ ls > newfile.txt
sh: newfile.txt: No space left on device

Delete an existing small file, then create a much larger one :

$ rm old.txt
$ dd if=/dev/urandom of=sample.txt bs=1M count=1 
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.91344 s, 1.1 MB/s
$ ls -l sample.txt 
-rw-r--r-- 1 root root 1048576 2025-03-06 01:37 sample.txt

It created the 1Mb file no problem.

To fix, I found this post:

Unable to create files on large XFS filesystem

In my case, I couldn't get the mount with inode64 option to work, so "Solution 2" worked, which was:

Plug in a sufficiently sized USB drive
Find the oldest folders
mv the old folders to the USB drive
mv the folders back

I have tried this with one folder, and it looks to have worked, so far.

RAID1 NAS drive still reporting No space left on device even after adding new disk

open 1

open 1

1 Answers1

Linked