Copying a large directory tree locally? cp or rsync?

Question

I have to copy a large directory tree, about 1.8 TB. It's all local. Out of habit I'd use rsync, however I wonder if there's much point, and if I should rather use cp.

I'm worried about permissions and uid/gid, since they have to be preserved in the copy (I know rsync does this). As well as things like symlinks.

The destination is empty, so I don't have to worry about conditionally updating some files. It's all local disk, so I don't have to worry about ssh or network.

The reason I'd be tempted away from rsync, is because rsync might do more than I need. rsync checksums files. I don't need that, and am concerned that it might take longer than cp.

So what do you reckon, rsync or cp?

Answer 1

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost. And being rsync, it can even restart part way through a large file. As others mention, it can exclude files easily. The simplest way to preserve most things is to use the -a flag – ‘archive.’ So:

rsync -a source dest

Although UID/GID and symlinks are preserved by -a (see -lpgo), your question implies you might want a full copy of the filesystem information; and -a doesn't include hard-links, extended attributes, or ACLs (on Linux) or the above nor resource forks (on OS X.) Thus, for a robust copy of a filesystem, you'll need to include those flags:

rsync -aHAX source dest # Linux
rsync -aHE source dest  # OS X

The default cp will start again, though the -u flag will "copy only when the SOURCE file is newer than the destination file or when the destination file is missing". And the -a (archive) flag will be recursive, not recopy files if you have to restart and preserve permissions. So:

cp -au source dest

Answer 2

When copying to the local file system I tend to use rsync with the following options:

# rsync -avhW --no-compress --progress /src/ /dst/

Here's my reasoning:

-a is for archive, which preserves ownership, permissions etc.
-v is for verbose, so I can see what's happening (optional)
-h is for human-readable, so the transfer rate and file sizes are easier to read (optional)
-W is for copying whole files only, without delta-xfer algorithm which should reduce CPU load
--no-compress as there's no lack of bandwidth between local devices
--progress so I can see the progress of large files (optional)

I've seen 17% faster transfers using the above rsync settings over the following tar command as suggested by another answer:

# (cd /src; tar cf - .) | (cd /dst; tar xpf -)

Answer 3

When I have to copy a large amount of data, I usually use a combination of tar and rsync. The first pass is to tar it, something like this:

# (cd /src; tar cf - .) | (cd /dst; tar xpf -)

Usually with a large amount of files, there will be some that tar can't handle for whatever reason. Or maybe the process will get interrupted, or if it is a filesystem migration, the you might want to do the initial copy before the actual migration step. At any rate, after the initial copy, I do an rsync step to sync it all up:

# cd /dst; rsync -avPHSx --delete /src/ .

Note that the trailing slash on /src/ is important.

Answer 4

This thread was very useful and because there were so many options to achieve the result, I decided to benchmark few of them. I believe my results can be helpful to others have a sense of what worked faster.

To move 532Gb of data distributed among 1,753,200 files we had those times:

• rsync took 232 minutes
• tar took 206 minutes
• cpio took 225 minutes
• rsync + parallel took 209 minutes

On my case I preferred to use rsync + parallel. I hope this information helps more people to decide among these alternatives.

The complete benchmark are published here

Answer 5

rsync

Here is the rsync I use, I prefer cp for simple commands, not this.

$ rsync -ahSD --ignore-errors --force --delete --stats $SRC/ $DIR/

cpio

Here is a way that is even safer, cpio. It's about as fast as tar, maybe a little quicker.

$ cd $SRC && find . -mount -depth -print0 2>/dev/null | cpio -0admp $DEST &>/dev/null

tar

This is also good, and continues on read-failures.

$ tar --ignore-failed-read -C $SRC -cf - . | tar --ignore-failed-read -C $DEST -xf -

Note those are all just for local copies.

Answer 6

The rsync command always computes checksums on every byte it transfers.

The command line option --checksum only relates to whether checksums of files are used to determine which files to transfer or not, ie:

-c, --checksum skip based on checksum, not mod-time & size"

The manpage also says this:

Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking its whole-file checksum, but that automatic after-the-transfer verification has nothing to do with this option’s before-the-transfer "Does this file need to be updated?" check.

So rsync also, always, calculates a checksum of the whole file on the receiving side, even when -c/ --checksum option is "off".

Answer 7

Whatever you prefer. Just don't forget the -a switch when you decide to use cp.

If you really need an answer: I'd use rsync because it's much more flexible. Need to shutdown before copying is complete? Just ctrl-c and resume as soon as your back. Need to exclude some files? Just use --exclude-from. Need to change ownership or permissions? rsync will do that for you.

Answer 8

rsync -aPhW --protocol=28 helps speed up those large copies with RSYNC. I always go rsync because the thought of being midway through 90GiB and it breaking scares me away from CP

Answer 9

rsync is great, but has issues with really large directory trees because it stores the trees in memory. I was just looking to see if they'd fix this problem when I found this thread.

I also found:

http://matthew.mceachen.us/geek/gigasync/

You could also manually break up the tree and run multiple rsyncs.

Answer 10

You definitely want to give rclone a try. This thing is crazy fast :

sudo rclone sync /usr /home/fred/temp -P -L --transfers 64

Transferred:       17.929G / 17.929 GBytes, 100%, 165.692 MBytes/s, ETA 0s
Errors:                75 (retrying may help)
Checks:            691078 / 691078, 100%
Transferred:       345539 / 345539, 100%
Elapsed time:     1m50.8s

This is a local copy from and to a LITEONIT LCS-256 (256GB) SSD.

You can add --ignore-checksum on the first run to make it even faster.

Answer 11

There are some speed-ups which can be applied to rsync:

Avoid

• -z/--compress: compression will only load up the CPU as the transfer isn't over a network but over RAM.
• --append-verify: resume an interrupted transfer. This sounds like a good idea, but it has the dangerous failure case: any destination file the same size (or greater) than the source will be IGNORED. Also, it checksums the whole file at the end, meaning no significant speed up over --no-whole-file while adding a dangerous failure case.

Use

• -S/--sparse: turn sequences of nulls into sparse blocks
• --partial or -P which is --partial --progress: save any partially transferred files for future resuming. Note: files won't have a temporary name, so ensure that nothing else is expecting to use the destination until the whole copy has completed.
• --no-whole-file so that anything that needs to be resent uses delta transfer. Reading half of a partially transferred file is often much quicker than writing it again.
• --inplace to avoid file copy (but only if nothing is reading the destination until the whole transfer completes)

Answer 12

When doing local a local directory copy, my experience is that "cp -van src dest" is 20% faster than rsync. As far as restartability, that's what "-n" does. You just need to rm the partially copied file. Not painful unless it's an ISO or some such.

Answer 13

ARJ IS SO OLD SCHOOL!! I really doubt that ARJ and/or rsync will give performance.

Definitely what I always do is use cpio:

find . -print | cpio -pdm /target/folder

This is almost fast than CP, definitely faster than tar and without pipeing anything.

Answer 14

tar would also do the job, but won't resume from being interrupted like rsync will.

Answer 15

For anyone in need to copy large amount of small files between two local mounts (in my case it was two NFS mounts of a NAS service from a cloud provider):

cp was painfully slow. When watching network throughput, I saw it could only saturate about 1 mbps of bandwidth. Then I tried with tar:

tar -pc /mnt/old-nas | tar -xpf - -C /mnt/new-nas

which could saturate the line fully, between 250-300 mbps.

Tar seems to perform much better when copying between two mountpoints with high latency.

Answer 16

If both storages are local, cp should transfer data near maximum possible speed. It is not necessary to use a synchronizer if the target directory is empty, but it brings benefits like restartability, possibility to exclude certain files etc.

rsync is strong in copying over network (delta transfer of big files). But rsync keeps its internal data in memory, which may cause problems with huge directory trees.

If you are interested in another synchronizer, you might have a look at Fitus/Zaloha.sh. It runs find on both directories and prepares scripts with cp commands. It keeps its internal data in files, not in memory. It is used as follows:

$ Zaloha.sh --sourceDir="test_source" --backupDir="test_backup"

If you want it to just generate the cp script (but not execute it, which would require extensive display and interaction), use the --noExec option.

Your use case presumably does not require generation of restore scripts: use the --noRestore option. Last, if you have the fast mawk installed, make use of it via the --mawk option.

Answer 17

What if you use ARJ?

arj a -jm -m1 -r -je filepack /source

where -jm -m1 are compression levels and -je makes it an executable. Now you have a encapsulated bash of files.

Then for extraction to the target map

filepack -y  

where the source map will be made (where -y is always accept, overwrite, skip etc)

One can then scp ftp the filepack to the target area and execute it, if that is possible.

Answer 18

Both will work just fine.