3

I want to build a file server that serves ~50 TB of content to its users. To maximize the server's throughput, I'm going to utilize the follow scenario.

  • 50 TB of HDD storage. All of the static files are residing here.
  • 6 TB of SSD storage. This will act as a cache for most popular contents.
  • A cache manager that decides what should reside on HDD or SSD.

Based on this architecture, the most popular files are copied to the SSD drives and served from there. The cache manager is a customized software, designed based on my application characteristics.

I had a few questions regarding this plan.

  1. Should I be worried about SSD write limits?
  2. Is there any cache framework that I can use to write my special-purpose cache manager, based on my own rules?
Sadjad
  • 133

2 Answers2

3

Although ZFS can do it, like ewwhite says, another solution might be bcache. I'm using that in a totally different scenario (2TB HDD and 128GB SSD in my laptop, using bcache makes loading Civ V a lot nicer ;-)), but it works very nicely.

Depending on how you serve the files, you might also want to consider something like Varnish, which you setup to use the SSD as the cache store.

Regarding using your own rules, don't do that. Lots of smart people have worked on this problem, you want to stand on their shoulders, IMHO.

Depending on how often you expect the most used content to change, I wouldn't worry about SSD write performance. Or put your SSDs in a RAID10 array to get even more performance out of them. Also, add a lot of RAM, so files can be cached in the kernel in memory block cache as well.

This all assumes a Linux machine, I guess.

Tim Stoop
  • 638
2

Don't reinvent the wheel. Use ZFS.

But you have other architectural concerns like networking, tuning, your client systems. It may also be helpful to describe the context for this and what you currently have in place.

ewwhite
  • 201,205