3

I am developing a web application service and am looking to scale our internal servers.

Our current server has ~1TB of storage capacity. Users create ~10gb of data which is added to our database daily.

As you can see at our current rate, we would only be able to sustain this storage growth for ~100 days. We do not expect the rate of data creation to slow in the near future. Due to the nature of the information we are not looking to use commercial cloud storage (aws, google, microsoft, softlayer etc.)

We would like to build a server infrastructure that can be continually expanded (beyond the limits of 100TB). This would be a gradual process as needed, and would span multiple 4U server racks.

My question is that what would be the standard way to do this without over-complicating our software. I have looked into ZFS and openNAS, but there seems to be limitations with pooling and continually expanding storage.

What is the best way to build a homogenous storage architecture that can be continually expanded to support our storage needs?

Gordon
  • 31

1 Answers1

2

Even from my limited experience I should say that your problem is under-specified. For example, you don't talk about load, latency, and bandwidth requirements. Check this Q&A page for more: Can you help me with my capacity planning?

My experience is building and managing 140TB (and growing, 80TB primary + 60 backup) storage for our lab. We use it as a storage for research data (imaging data mostly, some genomics; no databases). System consists of two servers in different server rooms, and was designed to be expandable. Please take what I write with a grain of salt.

Our servers run FreeBSD and ZFS, 4U boxes with SAS expanders. Each box is stuffed with 4-5TB drives. Backup is newer and has 12x5TB drives, configured as RAID6 aka RAIDZ2.

Expandability is achieved via two routes:

  1. Each Z Pool (e.g. 12x5TB) can be expanded by adding extra 12 HDDs in parallel. Trick is that we'll have to add drives to storage in chunks of 60TB, we can't (AFAIK) add 30TB and six months later extra 30TB. If you are interested in ZFS yo ushould read up on terminology (many websites, including Oracle's and FreeBSD's)
  2. Using SAS allows us easily add extra 4U JBOD box with up to 45 drive slots, by connecting to main box's HBA/RAID card via single cable.

So, from what I know:

  • ZFS, when run on server-grade hardware, can be successfully used with almost no supervision for hosting 100TB of data and automatic off-site backups
  • Expansion of ZFS storage can be easily done when using SAS expanders, if bandwidth requirements are met
  • Expand-ability is not "smooth" in a sense that if you configure your zpool in blocks of 20TB you will have to add drives in chunks of 20TB or larger down the road. You can't just buy a 1TB hard drive and plug it in if you run some sort of RAID6/7. In case of RAID10 you can do that because you "block" is essentially 1 or 2 drives

Update

Current system bandwidth limitations are as follows:

  1. Network via copper Ethernet 1Gbps (whole campus) or 10Gbps (inside the rack)
  2. RAID60 arrays using HDDs: n x 160MB/s (in 4x configuration is it ~5Gbps)
  3. SAS3 protocol using 4x12=48 Gbps (connects backplane to HBA using SAS3 4x cables)

As you can see, we are currently limited by 1Gbps ethernet, if we work inside the rack (storage<->analysis server), we become limited by server's ability to swallow 5Gbps streams. Our data is mainly large files (5-500GB), so I am not sure about the IOPS. To improve bandwidth currently we need to install fiber optics in the building (our lab and server rooms are on different floors), then we become limited by the speed of our RAID60 arrays. Then we either have to add extra stripes or build SSD-based storage. This system was built to work instead of USB3-based HDD, which it beats.

aaaaa says reinstate Monica
  • 520