Is it possible to use rsync over sftp (without an ssh shell)?

Question

Rsync over ssh, works great every time.

However, trying to rsync to a host which allows only sftp logins, but not ssh logins, provides the following error:

rsync -av /source ssh user@remotehost:/target/

protocol version mismatch -- is your shell clean? (see the rsync man page for an explanation) rsync error: protocol incompatibility (code 2) at compat.c(171) [sender=3.0.6]

Here's the relevant section from the rsync man page:

This message is usually caused by your startup scripts or remote shell facility producing unwanted garbage on the stream that rsync is using for its transport. The way to diagnose this problem is to run your remote shell like this:

          ssh remotehost /bin/true > out.dat

then look at out.dat. If everything is working correctly then out.dat should be a zero length file. If you are getting the above error from rsync then you will probably find that out.dat contains some text or data. Look at the contents and try to work out what is producing it. The most com‐ mon cause is incorrectly configured shell startup scripts (such as .cshrc or .profile) that contain output statements for non-interactive logins.

Trying this on my system produced the following in out.dat:

ssh-dummy-shell: Command not allowed.

As I thought, the host is not allowing ssh logins.

The following link shows that it is possible to accomplish this task using fuse with sshfs - however it is extremely slow, and not fit for production use.

Is there any chance of getting rsync sftp to work?

Answer 1

Unfortunately not directly. rsync requires a clean link with a shell that will allow it to start the remote copy of rsync, when run this way.

If you have some way of running long-lived listening processes on the host you could try starting rsync manually listening for connections on a non-privileged port, but most techniques for doing that would require proper shell access via SSH too, and it relies on the hosts firewall arrangements letting connections in on the port you chose (and the host having rsync installed in the first place). Running rsync as a publicly addressable service (rather than indirectly via SSH or similar) is not generally recommended for non-public data though.

If you host allows scripting in PHP or similar and does not have it locked down so extra processes can not be execed by user scripts, then you could try starting rsync in listening mode that way. If your end is connectible (you are running SSH accessible to the outside world) you could try this in reverse - have a script run rsync on the server but instead of listening for incoming connections have it contact your local service and sync that way. This still relies on rsync actually being installed on the host which is not a given, or that you can upload a working copy, but does not have the security implications of running an rsync daemon in a publicly addressable fashion and talking to it over an unencrypted channel.

Messing around as described above may be against the hosts policies though, even if it works at all, and could get you kicked off. You are better off asking if a full shell can be enabled for that account and either abandoning rsync for that host or abandoning that host and moving elsewhere if they will not do that.

Answer 2

An alternative to using rsync is to instead use lftp (which can connect to sftp) and use the mirror command. For example one can do

lftp
~> open -u user,password sftp://host.com
~> mirror remotedir outdir
~> quit

Answer 3

Theoretically, yes. You can mount the remote filesystem on your local machine using FUSE. Then you can run a local copy of rsync between the mounted directory and the local directory. I've not personally tried this, but it should work in theory. It would likely be a lot less efficient that performing the rsync over SSH because it would need to transfer at least part of each file to perform the comparison.

Answer 4

Instead of Rsync, rclone supports syncing from/to and mounting SFTP network shares.

1. Install rclone
2. rclone config and follow the steps to add a new remote endpoint configuration. Input sftp at the protocol backend selection.
3. rclone sync local_path remote:remote_path (docs). This can cause data loss.

IMO, rclone provides a more automation-friendly CLI than LFTP.

I know this does not answer the original question but probably the needs of some people which come here.

Answer 5

a bit late, but here is how I do it, using sshfs

  source /scratch/slimdata/password.sh
  mkdir tmp_mnt
  echo $PASSWORD | sshfs user@host:dir tmp_mnt -o password_stdin
  rsync -rutL --delete tmp_mnt/ to_sync/
  fusermount -u tmp_mnt
  rmdir tmp_mnt

Answer 6

Option combining best of all

It appears to have the following advantages that other answers here don't have:

• fully benefits from SSH (secure)
• fully benefits from rsync (bandwidth-efficient protocol, all rsync options like bandwidth limitation)
• actually efficient (unlike sshfs which save the days sometimes but is still slow in practice)
• doesn't need arbitrary shell commands server-side
• doesn't need server to allow ssh tunnels
• doesn't need server to run a rsync daemon

I have not yet tested it, but I've successfully used all those features except rrsync.

How to do it

• create a key locally with ssh-keygen (openSSH feature)
• allow login only with that specific key, with an entry in server's ~/.ssh/authorized_keys (openSSH feature)
• associate this key to a specific command, with ~/.ssh/authorized_keys (openSSH feature) using as command the script rrsync distributed with rsync, something like command="$HOME/bin/rrsync -ro ~/backups/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-

Then you can rsync from client normally.

If you need more details

Restricting SSH Access to rsync | Guy Rutenberg

One step beyond link above

The command at the end of that page can be made shorter. In ~/.ssh/config create a stanza like this:

Host remote # can be host or ip or custom-label User user # login on remote host HostName optional-dns-resolvable-host-or-ip # if label used above IdentityFile ~/.ssh/id_remote_backup

then your rsync command from client

rsync -e "ssh -i $HOME/.ssh/id_remote_backup" -av user@remote: etc2/

becomes

rsync -av user@remote: etc2/

Answer 7

No. rsync works by running rsync on the other side and communicating with it, which means that some form of shell access is required.

Answer 8

An alternative would be to start rsync as daemon and connect to it through a SSH tunnel.