File Server - Storage configuration: RAID vs LVM vs ZFS something else...?

Question

We are a small company that does video editing, among other things, and need a place to keep backup copies of large media files and make it easy to share them.

I've got a box set up with Ubuntu Server and 4 x 500 GB drives. They're currently set up with Samba as four shared folders that Mac/Windows workstations can see fine, but I want a better solution. There are two major reasons for this:

1. 500 GB is not really big enough (some projects are larger)
2. It is cumbersome to manage the current setup, because individual hard drives have different amounts of free space and duplicated data (for backup). It is confusing now and that will only get worse once there are multiple servers. ("the project is on sever2 in share4" etc)

So, I need a way to combine hard drives in such a way as to avoid complete data loss with the failure of a single drive, and so users see only a single share on each server. I've done linux software RAID5 and had a bad experience with it, but would try it again. LVM looks ok but it seems like no one uses it. ZFS seems interesting but it is relatively "new".

What is the most efficient and least risky way to to combine the hdd's that is convenient for my users?

---

Edit: The Goal here is basically to create servers that contain an arbitrary number of hard drives but limit complexity from an end-user perspective. (i.e. they see one "folder" per server) Backing up data is not an issue here, but how each solution responds to hardware failure is a serious concern. That is why I lump RAID, LVM, ZFS, and who-knows-what together.

My prior experience with RAID5 was also on an Ubuntu Server box and there was a tricky and unlikely set of circumstances that led to complete data loss. I could avoid that again but was left with a feeling that I was adding an unnecessary additional point of failure to the system.

I haven't used RAID10 but we are on commodity hardware and the most data drives per box is pretty much fixed at 6. We've got a lot of 500 GB drives and 1.5 TB is pretty small. (Still an option for at least one server, however)

I have no experience with LVM and have read conflicting reports on how it handles drive failure. If a (non-striped) LVM setup could handle a single drive failing and only loose whichever files had a portion stored on that drive (and stored most files on a single drive only) we could even live with that.

But as long as I have to learn something totally new, I may as well go all the way to ZFS. Unlike LVM, though, I would also have to change my operating system (?) so that increases the distance between where I am and where I want to be. I used a version of solaris at uni and wouldn't mind it terribly, though.

On the other end on the IT spectrum, I think I may also explore FreeNAS and/or Openfiler, but that doesn't really solve the how-to-combine-drives issue.

Answer 1

LVM is actually quite heavily used. Basically, LVM sits above the hardware (driver) layer. It doesn't add any redundancy or increased reliability (it relies on the underlying storage system to handle reliability). Instead, it provides a lot of added flexibility and additional features. LVM should never see a disk disappear or fail, because the disk failure should be handled by RAID (be it software or hardware). If you lose a disk and can't continue operating (rebuild the RAID, etc), then you should be going to backups. Trying to recover data from an incomplete array should never be needed (if it is, you need to reevaluate your entire design).

Among the things you get with LVM are the ability to easily grow and shrink partitions/filesystems, the ability to dynamically allocate new partitions, the ability to snapshot existing partitions, and mount the snapshots as read only or writable partitions. Snapshots can be incredibly useful, particularly for things like backups.

Personally, I use LVM for every partition (except /boot) on every box I build, and I've been doing so for the past 4 years. Dealing with non-LVM'ed boxes is a huge pain when you want to add or modify your disk layout. If you're using Linux, you definitely want use LVM. [Note: This above stuff on LVM has been updated to better explain what it is and how it fits into the storage equation.]

As for RAID, I don't do servers without raid. With disk prices as cheap as they are, I'd go with RAID1 or RAID10. Faster, simpler, and much more robust.

Honestly though, unless you're wedded to Ubuntu (which I would normally recommend), or if the box is performing other tasks, you might want to look into OpenFiler. It turns your box into a storage appliance with a web interface and will handle all of the RAID/LVM/etc for you, and allow you to export the storage as SMB, NFS, iSCSI, etc. Slick little setup.

Answer 2

ZFS is really reliable and it sure does make your storage management hell of a lot easier. As a bonus: smb is integrated with ZFS in OpenSolaris and it handles Raid very well. Wait a few days, download the by then released 2009.6 version and give it a go on a test machine. I am sure you will love ZFS.

And about your comment ZFS being new: not very new anymore!

Answer 3

The central question is: "How important is this data?"

If the answer is "I can recreate it easily" you want RAID5, possibly with LVM on top of it for simplicity of management.

If the answer is "I can recreate it but it would take a while and people would complain" you want RAID 6 or more likely RAID 1/10.

If the answer is "Nobody does any work while I recreate it and make sure it's bit-perfect" you want ZFS/Raid-Z

Note that you're always able to recreate it. RAID isn't a backup.

Answer 4

To connect a lot of drives in the same chassis, a hardware RAID controller is the best tool. It will provide lots of SATA connectors for your drives, redondancy via RAID-5 or preferably RAID-6, and may provide better performance too.

Software RAID performance is often better than hardware RAID in benchmarks, however file serving and software RAID are both CPU intensive and compete for your processors while working. My experience shows that unless you use dual quad-core systems, properly configured hardware RAID will beat software RAID hands down.

Good hardware controllers with good linux support :

• Areca
• 3Ware
• the new Adaptec series (the old are slooooow)
• LSI MegaRAID

Answer 5

RAID is NOT like LVM. You can use RAID to make fault-tolerance partitions, but LVM is used to easy disk partitioning and filesystem edition. You can use RAID over LVM or ZFS (ZFS is can work both RAID and LVM). In my opinion, ZFS works better than LVM, but:

• on Solaris 10/11/OpenSOlaris only, you can't use it from linux
• ZFS is disk management and filesystem, LVM allow to use any filesystem you need

On Ubuntu i prefer to use RAID5 MD with LVM.

Answer 6

Take a look at what Nexenta and OpenSolaris are offering and I think you'll be very pleased with that you can get for nothing. It's rumoured that the next releases of OpenFiler will use the FreeBSD ZFS port as well (though, they're quite behind from a feature perspective).

That being said, I try to avoid doing RAID5, RAID6 or RAID50 in software and prefer to use hardware controllers to offload all the XOR work. RAID1 and RAID10 in Linux software work pretty well, and from there I put LVM on top of them to allow more flexibility in what's done with the blocks I have after the redundancy in place. RAID+LVM+XFS is my favorite Linux config., but I'd take ZFS over it anyday.

Answer 7

RAID vs LVM isn't really a good comparison they perform separate roles, and are frequently used together. RAID is used for drive redundancy, LVM can be used to break up your RAID device into logical volumes, it is used for easy resizing, and for taking snapshots.

Answer 8

I ran the file server for a very similar company/situation. Basically a 3 person graphics department with 30TB of storage and the shoestring budget of a small company. Our typical projects ran from a 0.5TB to 6TB. And the file server was serving a sizable rendering farm which could really pound on it.

In my setup I ran a 3U server running Linux with external hardware RAID6 arrays attached to it. I managed the physical and logical volumes with LVM and ran the XFS file system. What I would do is create a logical volume for each project and then expand it as the project grew. When the project was done I could archive the job to tape and blow away the logical volume. This would return that space back to the volume group where it would get reallocated to the next project(s).

This was a very clean way to utilize our storage but there are two drawbacks to this approach. You end up having to micromanage the sizes of the logical volumes trying to balance the amount of space allocated to a logical volume so that you had enough space to do your job but not over allocate it and end up wasting space. Our rendering farm was capable of generating many TB's of data a day and if you didn't pay close attention to this you would run out of space in a hurry. I eventually setup some scripts that monitored the trends in the available space on the logical volumes and would auto grow them. Even with that in place with 80 or so Logical Volumes there was a lot of unused space tied up in all the logical volumes. I've already hinted at the 2nd problem....LVM doesn't really do thin provisioning and XFS only allows you to grow a file system. So over allocation of space to a logical volume can add up to a lot of unusable space.

This was all setup about 5 years ago and if I were setting it up today I would use OpenSolaris and ZFS. The main reason is ZFS pooled storage approach means less volume management. You can still separate each project into its own file system but without having to micromanage the sizes of the individual volumes. ZFS has some other very nice features that make it a better choice but there are other questions on serverfault that go into that.

In my opinion ZFS simply is the best free solution available today.

Answer 9

Some things to consider if you stay with Linux:

• Think about the filesystem, too. Your 4x 500GB example is about the largest capacity I would recommend with good conscience for ext3. I do not recommend to create much larger ext3 filesystems because e.g. fsck time can be enormous. Create several smaller instead of one large filesystem.
• As you mentioned video data: ext3 handles large files inefficiently because it'll have to create indirect, double-indirect and triple-indirect metadata blocks to store the data of large files and you'll pay the price. Nowadays, ext4 supports extents and handles this much better. But then, it is rather new and e.g. Red Hat Enterprise Linux 5 does not support it yet. (Some Enterprise distros will support alternatives like XFS).
• If there is data corruption on any data block you'll have a hard time to even notice this with Linux filesystems. ZFS on the other hand checksums all metadata and data and also verifies the checksum every time data is read from disk. (There is also background scrubbing)
• RAID rebuild time on Linux is proportional to disk size because the RAID layer does not know the contents of the filesystem (layer). ZFS's RAID-Z rebuild time depends on the amount of actual data on the failed disk because only used blocks will be copied/rebuilt on a replacement disk.
• Do you want to snapshot your filesystems? LVM snapshots don't even compare with ZFS's instantaneous snapshots. The latter can also easily exposed to the end-users e.g. for easy restores.
• Use RAID6 (RAID-Z2) and not only RAID5 (RAID-Z) with large disks (>500GB) because chances are that another disk will fail during rebuilt.

Answer 10

Use the "mdadm" utility to create a RAID-5 array out of your drives.

This provides the redundancy you need, so if a drive goes dead you can replace it with no data loss, and also gives you usage of 3 out of the 4 drives.

I'd also recommend you create an LVM volume on top of the RAID, so you can partition the space out as needed.

Answer 11

You may want to give a look at AFS. This will give you some measure of both availability (you can access these files both on and off your network) and redundancy (files can be replicated). For a workflow where you open a file, work on it for some time and then save it, this would be better (from a network standpoint) than NFS, at least older NFS.

Answer 12

As the other reply says, LVM is widely used, and can combine several disks into a single "volume group" that looks like one huge block device. It's worth remembering, though, that doing this increases your probability of failure -- it only takes one disk to fail in a volume group to take out the entire filesystem, and if you have four disks joined together, this is four times more likely. Best to use LVM on top of a RAID1 or similar to mitigate this. Linux software RAID is quite adequate for this, although hardware RAID may be more convenient.

Answer 13

Sorry this would be a comment but I don't have the rep...

How does RAID-Z or RAIDZ2 provide better redundancy than RAID6?

ZFS has checksumming everywhere

About the original question. Whatever the data I would use 2 active parity per 10 discs. I would use good quality RAID card the 3ware ones are excellent. Personally I use hardware RAID with battery backup. LVM just so you can migrate the data easily at the end of the hardware's life. XFS as the file system.

Answer 14

Why you don't use a disk or SSD card to have the system server to bootup and the 500 GB as only storage. Use one 500GB disk and when it becomes full exchange it manually. You can do the backup later in another computer with calm. Because while the harddisk is turning (spinning) it can be damaged. Anyway if you connect all disks at the same time they all are turning and can take damage if you use it or not. The probability of failure raise when you have more disks turned on. Use one per time and exchange it when are full or in a period of time anticipating a failure (use too the SMART capability to get rid of that). Use a carry disk or use some external SCSI/SATA disk adapter so you don't need to dissasemble the computer server every time you exchange a disk. That is the most secure and reliable form. Using RAID is more expensive and you just waste some disks (because you let it turned on in risk of become damaged for only letting it turned on? stupid, or not?). If you want more datatroughtput then using a RAID configuration for that I guess is a good choice but never trust in a computer configuration. The backup must be done always manually with a person (the network or server administrator). That is one of the works of an administrator. You can use tapes, DVD, BlueRay, or other disks to do the backup. But you always will need a reliable storage medium and a running disk is not one. A turned off disk and well saved (in a fresh and free of humidity place) is a reliable storage medium.