0

We're trying to upload 150TB into S3 using Commvault. However we're only able to upload about half a terrabyte a day with a 1.5Gbs dedicated line. We've also enabled the following recommendations:

Is there anything else we can do to increase the ingestion rate into S3?

Tim
  • 33,870
  • 7
  • 56
  • 84
Juraj Lišiak
  • 76

2 Answers2

1

So after opening up a ticket with Commvault and several hours of troubleshooting we found out the issue was hot spots on the on-premise SAN and the slow SATA disks from where the backups are stored and send over to S3. Once that was fixed, the internet pipe became the bottleneck.

Bottom line, we were able to scale out from 500GB a day to 10TB+ a day. S3 was able to scale out fairly linearly as more streams were added from the Commvault end.

Juraj Lišiak
  • 76
0

Have a read of this article. Latency can make a significant difference to S3 upload performance. What is your ping time to S3? What is your average object size? How many threads are you using?

S3 transfer acceleration uses CloudFront for uploads, which might make it a bit faster. I've read reports that it doesn't always give much benefit.

You could upload to an S3 region closer to you, then have the data replicated over. That will probably cost more.

You can also send a physical disk, using AWS import export.

Tim
  • 33,870
  • 7
  • 56
  • 84