1

Our latest project at work involves sending 100GB+ files from our office in Dallas to our client in Australia. I've been asked to put together the fastest, most reliable way of transferring these files. Usually this would be no sweat, but when they said 100GB I decided I needed to ask for advice. I'm also in a time crunch, so I have about 24 hours to put together a professional solution (no major software development).

It must be screaming fast, and it must have tolerance to internet service interruptions on both the upload and download side (upload resume and download resume). It also needs to be as turn-key as possible on the client download side. A GUI interface tool would be great.

My first thoughts were to use an Amazon S3 account with a 3rd party download/upload manager, but I'm not sure how fast Amazon's S3 bandwidth limitations are. Is EC2 faster? Also Amazon's website tended to be catering to software programmers - are there any reliable pre-rolled applications that support multipart uploads and downloads?

Scott Pack
  • 15,097
uberdanzik
  • 131
  • 2
  • 5

1 Answers1

2

The limiting factor is going to be the size of the pipe between you and your client. There is no technology that is going to be faster than another. Any modern file transfer mechanism will be able to saturate whatever links you have.

The first thing to do to cut down the transfer time is to efficiently compress your data. Experiment with different compression formats like rar or 7zip to see which is most effective.

Since all transfer methods are fairly equivalent in terms of speed, concentrate on restartability. SCP or SFTP have good restart capabilities and are easily scriptable. If the files ultimately need to end up at multiple offices of your client then BitTorrent is a good option because their offices can share pieces between themselves.

However:

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. —Tanenbaum, Andrew S. (1996). Computer Networks. New Jersey: Prentice-Hall. p. 83. ISBN 0-13-349945-6.

It's likely that the fastest way in terms of elapsed time is to use a USB hard drive or equivalent and just mail it or send it via courier. I suggest you do some tests with your client to see what kind of throughput you get, them do the math to see if a station wagon full of tapes hurtling down the highway is faster. (Or a USB hard drive on a jet, whichever you prefer.)

longneck
  • 23,272