4

Please, excuse my horrible english. I'm triyng to find the way to clone a HDD server to a remote destination location, using SSH and, if possible, compress data on-the-fly to minimize unnecessary data transmission between them. The main objective is a server migration between physical & VPS appliances.

Broadband is approx 50 - 60 MB/s (transfers at 6-8 MB/s). Running a full-dd-copy estimate a 28 hours copy time. I want reduce that time as much as possible without reduce partition size in source server (i want source as-is).

Origin & destination servers are in offline status (both initiated in rescue mode).

Data on original server are 60 GB. Rest of partition is filled with zeros.

There is a way to archive that?

Until now i cloning entire disks (but not so large) with this command:

dd if=/dev/sda status=progress bs=10M conv=fsync | ssh user@ip dd of=/dev/sda

That works, but this time i'm trying to do it with too much data...

Any ideas?...

Thank you very much in advance to all. Best regards!

D.

Diego Ferrarese
  • 111

4 Answers4

7

Ok, guys... finally, server migration was executed with status SUCCESS! Data copy time was aprox 5hs 30Min. Command & total time:

dd if=/dev/sda bs=5M conv=fsync status=progress | gzip -c -9 | ssh user@DestinationIP 'gzip -d | dd of=/dev/sda bs=5M'
751593062400 bytes (752 GB, 700GiB) copied, 19185.8 s, 39.2 MB/s

I want to thanks to all for your comments, specially Tom Yan. Hope this help someone else! Best regards!

Diego Ferrarese
  • 111
0

Ok Guys. I return with new findings. Finally, I think i got it. Thanks to Tom Yan's directions, i realized that piping gzip -c at source side, and gzip -d at destination side did the trick.

I tried -C flag in ssh connection, with relatively no impact in speed.

So, after several tests this was the command that resolved my problem:

dd if=/dev/sda bs=4MiB conv=sync status=progress | gzip -c -9 | ssh user@DestinationIP 'gzip -d | dd of=/dev/sda bs=4MiB'

As Tom Yan's post, firstly (my bad) i not quote whole command line for the remote host and the the result not changed... so, i don't understand the reason of that quotes... of course, finally i put them... but not understand why...

Diego Ferrarese
  • 111
0

You should use conv=sync or conv=fsync in the destination section, like this:

dd of=/dev/sda bs=4MiB conv=fsync

and you can use pigz for speed up compression.

Correct command:

dd if=/dev/sda bs=100M status=progress | pigz -6 | ssh root@172.16.92.15 'pigz -d | dd of=/dev/sda bs=100M conv=fsync'
Tommiie
  • 5,704
Dmitriy Denisov
  • 1
0

You can replace gzip with pigz to get more throughput, so:

dd if=/dev/sda bs=4MiB conv=sync status=progress | pigz -c -9 | ssh user@DestinationIP 'pigz -d | dd of=/dev/sda bs=4MiB'
quaker
  • 1