Speed up SFTP uploads on high latency network?

Question

I'm trying to transfer a set of large files internationally using SFTP, but I am finding my international partner can't get upload speeds above ~50k despite very good connections on either side. We can get multiple connections uploading at this speed (so not bandwidth?), but no single upload improves in speed, which is a problem as many files are several gb in size.

The SFTP is being hosted using the standard Apple OSX "Remote Login" SFTP system.

Is there a way to improve upload speeds, or is there a different SFTP host that would help? It's not clear to me if this is a configuration problem or an inherent limitation of the protocol.

(For security reasons I need to be using an end-to-end encrypted peer-to-peer connection -- no cloud services).

Answer 1

With OpenSSH sftp client (which you seem to use), you can use:

• -R switch to increase request queue length (default is 64)
• -B switch to increase read/write request size (default is 32 KB)

For a start, try to double both:

sftp -R 128 -B 65536 user@host

It probably does not matter much, which of them you increase.

Increasing either should help to saturate your high-latency connection. With the above settings, it will keep 8 MB worth of data flowing in the pipe at any time (128*64K=8M).

Note that this helps with big file transfers only. It won't have any effect, when transferring a lot of small files.

---

For some background and a discussion about other (GUI) SFTP clients, see the "Network delay/latency" section of my answer to Why is FileZilla SFTP file transfer max capped at 1.3MiB/sec instead of saturating available bandwidth? rsync and WinSCP are even slower.

For explanation why it helps with big files only, see What does the -C flag exactly do in scp? – It's about SCP, but most of it is true for SFTP too.

Answer 2

(You mention "high latency" in the question title, but not in the body text. Have you measured the actual latency, and what are the results?)

There's a patch to OpenSSH that explicitly improve throughput on a high-latency network link: HPN-SSH: (emphasis mine)

SCP and the underlying SSH2 protocol implementation in OpenSSH is network performance limited by statically defined internal flow control buffers. These buffers often end up acting as a bottleneck for network throughput of SCP, especially on long and high bandwith network links. Modifying the ssh code to allow the buffers to be defined at run time eliminates this bottleneck. We have created a patch that will remove the bottlenecks in OpenSSH and is fully interoperable with other servers and clients. In addition HPN clients will be able to download faster from non HPN servers, and HPN servers will be able to receive uploads faster from non HPN clients.

So, try to compile and use HPN-SSH on the receiving side, and see whether it improves your transfer speed.

Answer 3

I'm trying to transfer a set of large files internationally using SFTP

It hasn't been mentioned as an answer yet, but when transferring multiple files over a high-latency link, there's one really simple solution to get better performance:

Transfer multiple files in parallel.

And it is a solution that you even mentioned in your question. Use it.

Basically, the TCP protocol doesn't handle connections with a large bandwidth-delay product very well - a single connection can't keep enough data moving at any one time. See https://en.wikipedia.org/wiki/TCP_tuning

Since each connection is limited by the TCP protocol, just use more connections.

Answer 4

You could try and enable compression, and see if that helps.

From man sftp:

-C Enables compression (via ssh's -C flag).

And from man ssh:

-C Requests compression of all data (including stdin, stdout, stderr, and data for forwarded X11, TCP and UNIX-domain connections). The compression algorithm is the same used by gzip(1), and the “level” can be controlled by the CompressionLevel option for protocol version 1. Compression is desirable on modem lines and other slow connections, but will only slow down things on fast networks. The default value can be set on a host-by-host basis in the configuration files; see the Compression option.

It rather sounds as though the connection might be rate limited at some point along its path (or rather, that seems to me the simplest explanation for your 50kB/s per connection, but multiple such connections being possible), although it might not be a bad idea to make sure the disks on either side aren't a factor.

You could also run a quick pcap to see if there are any 'obvious' issues (such as a large number of retransmits) - but unless you had some confidence you would be able to address this, I would probably just see if enabling compression would help.

Answer 5

Speed up sftp transfers

Assuming your issues are network tuning and/or throttling per TCP connection, take a look at sftp using the lftp mirror subsystem

Network tuning on each end is a much bigger topic and would require a lot of back and forth, pushing the topic outside of the scope of ServerFault. For individual connections, the compression mentioned by iwaseatenbyagrue may help either way. This assumes the remote end allows compression.

Answer 6

In my case the sftp client was opening new connections rather than using the existing open connection. Switching to using the existing open connection made most operations a lot faster because the connection establishment had already been done. This makes a significant difference if latency is high because it takes several packet round-trip-times to establish a connection.

Throughput may be reduced using a single connection, depending on their server's configuration.

Answer 7

Not sure if this is an option for you, but have you tried pulling vs pushing the data to the international site? As well as either at different times to see if its an issue with contention for network resources?

Answer 8

We can get multiple connections uploading at this speed (so not bandwidth?)

It sounds like a configuration issue - either deliberately (as a way of upselling services without having to make any extra provision) or by accident (e.g. broken window scaling or overzealous traffic control). While you could parallelize the transfers you've told us nothing about what is at the other end of the connection or if its worth developing some simple scripts to handle sharding/reconstitution of files.

Tuning the queue size and compression are unlikely to have any significant impact, unless the cause is very badly written software (and openSSH does not into this category - not much point in using openssh with a longer request queue/larger block size unless the latency is over 250msec. You might consider trying with different clients from different locations to rule out a problem with the server.

My first call would be to identify which provider is to blame for the problem, ask them to fix the problem or switch to a different provider.