3

This server has RAID-1 configured. It get readonly file system everyday. if i reboot it, it gets back with a read-write file system. but after a short while, readonly again...

Any idea please? Thanks. 

# dmesg |grep error
VFS: cannot write quota structure on device cciss/c0d0p8 (error -30). Quota may get out of sync!
VFS: cannot write quota structure on device cciss/c0d0p8 (error -30). Quota may get out of sync!
VFS: cannot write quota structure on device cciss/c0d0p8 (error -30). Quota may get out of sync!
VFS: cannot write quota structure on device cciss/c0d0p8 (error -30). Quota may get out of sync!
VFS: cannot write quota structure on device cciss/c0d0p8 (error -30). Quota may get out of sync!
VFS: cannot write quota structure on device cciss/c0d0p8 (error -30). Quota may get out of sync!
VFS: cannot write quota structure on device cciss/c0d0p8 (error -30). Quota may get out of sync!
VFS: cannot write quota structure on device cciss/c0d0p8 (error -30). Quota may get out of sync!
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71927230 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71927273 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71927333 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71927712 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71929238 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71929464 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71929704 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71929805 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71930367 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71931281 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71927230 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71927273 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71927333 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71927712 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71929238 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71929464 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71929704 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71929805 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71930367 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71931281 in dir #71927229
EXT3-fs error (device cciss/c0d0p8): ext3_lookup: unlinked inode 71927230 in dir #71927229
ewwhite
  • 201,205
Jasper
  • 63

2 Answers2

5

This is a cciss controller, so the server is probably an HP ProLiant system. I would suspect an issue with the drive array in the form of a failed or failing disk. In addition to the normal Linux-level disk check (fsck), try to see if you can get any information on the drive array's health.

Do you have physical access to the server? Can you see any error lights on the drives?

Which Linux distribution is this?

If you have root access give us the output of cat /proc/driver/cciss/cciss0. Check to see if the HP management agents are installed. Try hplog -v to print the system's IML log to check for error messages. If you have the hpacucli utility installed, you may be able to get the specifics of the array's health with hpacucli ctrl all show config detail.

If none of those HP utilities are installed, there are other ways to get basic array info. You could install the HP Management Agents for your distribution or check this utility to get quick array status.

Community
  • 1
ewwhite
  • 201,205
4

Looks like your disk needs a cleanup. You should force a fsck on it to clean up all these errors before it craps out on you totally.

There are a lot of switches available with fsck, but to get you started you can do one of the following:

This will check all mounts in your /etc/fstab file:

fsck -A

This will check the particular disk that is throwing those warnings:

fsck -t ext3 /dev/<device name>

You should be aware that an fsck can take a LONG time so this is not something you want to do in the middle of the day on a production server.

jdw
  • 3,955