3

Currently, I'm tasked with progressively building a ZFS store. IOPS are not the priority, data security and capacity are.

In essence what we have is one brute of a machine, that contains an OS (FreeBSD most likely but OmniOS is a possible), 2 RAID-1 OS disks, and 4 SSDs, for caching and ZILs. RAM will start of at 16GB but I can go all the way up to 128GB if need be (and if accounting doesn't suffer a stroke). That in turn is connected to a JBOD chassis that can handle upto 45 drives. Each drive is 4TB.

I was considering incrementing by 9 drives, using a RAID-Z2 schema and then as needed, creating a new vDEV of another 9 drives until the 45 drive capacity is fulfilled, after which we would add another chassis that also handles another 45 drives. There is a limit of 3 or 4 connected chassis before a new similar configuration would need to be created.

Another possible option is to go up in 7s (RAID-Z2) and once we hit the 42 drive mark, we can slot in 3 drives to be used at any moment by any of the vdevs if a drive pushes up the daisies.

The Architecture is something as follows:

                                Storage Server
                                       |
                                       |
                                       |
               ------------------------------------------------
              |                       |                       | 
             JBOD 1       ...        JBOD 2                   JBOD 3
            -vdev 1.1 (9 drives)    -vdev 2.1 (9 drives)     -vdev 3.1 (9 drives)
            -vdev 1.2 (9 drives)    -vdev 2.2 (9 drives)     -vdev 3.2 (9 drives)
            -vdev 1.3 (9 drives)    -vdev 2.3 (9 drives)     -vdev 3.3 (9 drives)
            -vdev 1.4 (9 drives)    -vdev 2.4 (9 drives)     -vdev 3.4 (9 drives)
            -vdev 1.5 (9 drives)    -vdev 2.5 (9 drives)     -vdev 3.5 (9 drives)

On a side note, I'd also highly appreciate any input on FreeBSD vs Illumian vs OmniOS with respect to ZFS. I'm fairly fluent on FreeBSD and I've played a bit with OmniOS but no working knowledge on Illumian.

TIA

[edit]

Referencing Nex7's blog post point 9, we're building our VDEV's on 9 disks, 2 are for parity, so we're still in the clear, but we could I believe exchange that for a 15 vdev with 2 for parity since we're using RAIDZ2.

Point made on the SATA drives, switching to SAS will not be a problem. The SSD drives are Intel server drives, but I'll be checking. In all, because we have a few dozen boxes running 100% on the same hardware, I'm not worried about compatibility being an issue.

The reason we did not got with Linux is because we're not 100% certain of how decent and stable the implementation is. If it were stable...

What advantage does OpenIndiana have over FreeBSD for ZFS? What is the advantage of Napp-it?

How would you suggest we deal with the storage head HA issue?

Thanks

SteveMustafa
  • 429

1 Answers1

2

What's your question? I don't see a question mark in your post... :)

  • If you're asking if this will work... Yes, this will work under any ZFS-compatible OS. If you don't like one, you can move your ZFS data to another OS with ease.
  • If you're asking if it will perform well... It depends... But really, no, It probably won't. SATA is bad news for what you're proposing.
  • Are you considering your cabling design? A 45-disk JBOD is going to have an onboard expander. Will SAS cabling become a bottleneck?
  • If you're asking if this is the best way to achieve scale-out storage... Probably not. Your design has no storage-head high-availbility. What happens if your "brute" of a server has a motherboard failure? You'd lose access to EVERYTHING until it's repaired.
  • If you're asking if the hardware is supported... It depends. You didn't provide any hardware or component specifications. Really. What server, what controllers, which disks, which SSD manufacturer?
  • If you're asking if there are best-practices for what you're trying to do, yes, there are. (I'd be rethinking the number of disks in each vddv if I were you)
  • If you're asking which OS to use, I'd say none of them... but that's subjective. I'd suggest NexentaStor, OpenIndiana (with Napp-It?) or just plain Linux these days.

Are those the questions you wanted answered? Does that answer them?

Community
  • 1
ewwhite
  • 201,205