Is there An open source de duplication backup solution?

Question

I am looking for a solution for backing up about 30 of my servers that are a mix of centos, windows 2003 and 2008 machines.

I have a san in another data center with a 1 Gig link between the sites that im going to be backing up onto so this is purely a disk based backup solution.

What I am ideally looking for is some open source software possible with a nice gui that will allow me to have a backup cycle with full backups on a monthly basis that go back 6 months and weekly backups that are saved for one month. I'm hoping for de duplication as well so if a file is saved on the first backup the software creates a symlink to it rather then saving the data again in all the proceeding backups. It must also be smart enough to know when a file is deleted that the file is moved to the latest backup it exists in and all symlinks to it are updated.

I was looking at bacula but im unsure if it can do the deduplication or something similar. If it can will someone be kind enough to point me at a decent guide as to how to do it.

If it does not can someone recommend another solution open source or not?

score 9 · Answer 1 · answered Feb 17 '11 at 16:07

Something simple like rsnapshot may do what you want. It doesn't sound like you want what's traditionally referred to as deduplication. The synthetic full backup functionality using hard-links that rsnapshot provides sounds like exactly what you're asking for.

(On the traditional deduplication front you may want to look at the Opendedup project as a starting point. It's the most complete open source data deduplication project that I've seen. It uses a custom filesystem (SDFS) as the basis of its functionality. I tend to think that this isn't want you're looking for, but in the interests of completeness I mention it here.)

score 3 · Answer 2 · edited Sep 03 '12 at 11:43

3

Bacula writing to a ZFS filesystem that has de-duplication turned on, then write the snapshot from that host to a remote host with ZFS and apply that transaction log to that filesystem.

edited Sep 03 '12 at 11:43

user9517

117,122

answered Feb 17 '11 at 16:29

chris

12,104

score 2 · Answer 3 · answered Feb 17 '11 at 16:34

2

I have used backuppc with good success on a range of windows linux and mac machines. http://backuppc.sourceforge.net/ http://www.zmanda.com/backuppc.html (info on De-Duplication)

answered Feb 17 '11 at 16:34

Nick Downton

502
2
13

dtoubelis · Answer 4 · 2011-06-11T13:48:55.370

1

Here few links for de-duplication of top of my head:

OpenDedup - de-duplicating file system that works on Linux and Windows.
s3fs - de-duplicated file system backed by Amazon S3 service.

edited Jun 11 '11 at 13:48

answered Jun 10 '11 at 04:21

dtoubelis

4,797
1
31
32

score 1 · Answer 5 · edited Jun 17 '11 at 20:53

1

I ended up going with synbak http://www.initzero.it/products/opensource/synbak/ as it does the hardlinked copies using rsync and can pull them from remote servers unlike rsnapshot.

I'm also looking into using something like rwsync or deltacopy to setup rsyncd on windows machines so I can use synbak for windows machines as well.

edited Jun 17 '11 at 20:53

user9517

117,122

answered Jun 17 '11 at 20:43

user71173

135
1
1
4

score 1 · Answer 6 · answered Nov 30 '12 at 11:32

1

You now have Obnam which does that also, and much more !

answered Nov 30 '12 at 11:32

Anthony O.

674

Is there An open source de duplication backup solution?

6 Answers6