In Ceph cluster, for the same total raw volume, for example 32To each in 5 nodes, how to choose between 16 ssd of 2To or 8 of 4To? Any guidance rules?

Question

I'm designing a ceph cluster for mixing cephFS and rbd for my company( VM et file infrastructure).

In my set I need 32To raw storage by node. I start with 5 nodes.

The seller quotes propose me to choose beetween 16 ssd of 2 teras or 8 ssd of 4 teras by node.

I mean what is the impact on the IOPS managed by ceph in theses cases, the reconstruction delay etc. May question is highly related with CEPH, not general.

I've follow many guide to drive my choices in many aspects, including ceph documentation and books. But not sure to really have the answer of this question. The only clues I've found are things like 'bigger is better'...

Wich approach I need to follow to choose beetween theses 2 options ?

Here some details if needed : The network nics dedicated to cephs Vlans are 25Gb speed, redondants etc I've considered to multiply at least 4Gb Ram for 1 tera on OSD, so 128 by node to be large. SSD disks are enterprise fitted and read intensive.

Thanks for your help

Cheers

Ztevoz

Answer 1

If those two options are your only options and both SSD models are enterprise SSDs you should be fine with both. Points to consider are scalability, performance and failure domains. The more OSDs you have the more resilient you are to disk failure because less data has to be transferred during recovery. You would also benefit from more parallelization by having more disks which could increase performance. If you don't use all slots and have only 8 disks per node instead of 16 you can add more disks later if your capacity reaches its limit. Don't let your cluster become full, that will be very difficult to get out of. Also plan your capacity to sustain the loss of an entire host so your PGs can be recovered even if an OSD host goes down. And that host failure should also not fill up your cluster, of course.