0

I want to spawn a redis slave in AWS (Sydney), which needs ~20Mbit/s to catch up on the writes of the master located in germany.

Problem: I get only 8Mbit/s (1MB/s).

Tried bigger instance types after reading Bandwidth limits for Amazon EC2 but still no difference.

Tried to download random content hosted in australia: tadaa, 20Mbit/s on one download.

Then the idea: Start multiple wgets in parallel to the same host in germany. Tadaa: Every download gets 1MB/s, and the speed sums up to more than 150Mbit/s (didn't try to get more, but should have been possible).

How to get rid of the per connection speed limit?

Community
  • 1
edlerd
  • 864

1 Answers1

0

As mentioned in the comments: IO-bound due to the high latency.

For Redis slaves the solution is an ssh tunnel (with compression enabled) as described in this blog post [1]. Due to the compression, bandwidth is cut by a factor of 10, so in our case 1MB/s was enough.

For downloads the solution is aria2c, a cli download client, which can open up to 16 connections in parallel for big files. This way you can join the bandwidth of multiple connections.

[1] http://tech.3scale.net/2012/07/25/fun-with-redis-replication/

edlerd
  • 864