24

I work with a team to manage 500-600 rented Windows 7 computers for an annual conference. We have a large amount of data that needs to be synced to these computers, up to 1 TiB. The computers are divided into rooms and connected through unmanaged gigabit switches. We prepare these computers ahead of time with the Windows installation and configuration, plus any files that we have available to us before we send the base image in for replication by the rental company. Every year, we have presenters approach on site with up to gigs of data that need to be pushed to the room that they will be presenting in. Sometimes they only have a few files that are small sizes, such as a slide PDF, but can sometimes be much larger >5 GiB.

Our current strategy for pushing these files is using batch scripts and RoboCopy. For the large pushes, we actually use a BitTorrent client to generate a torrent file, and then we use the batch-RoboCopy to push the torrent into a folder on the remote machines that is being monitored by an installed BT client. Often times, this data needs to be pushed immediately with a small time window. We have several machines in a control room that are identical to the machines on the floor that we use for these pushes.

We occasionally have a need to execute a program on the remote machines, and we currently use batch and PSexec to handle this task.

We would love to be able to respond to these last minute pushes with "sorry, your own fault", but it won't happen. The BT method has allowed us to have a much faster response time, but the whole batch process can get messy when there are multiple jobs being pushed. We use Enterprise Ghost for other processes, and it doesn't work well in this large of scale, plus it is really quite expensive for a once-a-year task like this.

EDIT: There is a hard requirement that the remote machines on the floor are running Windows. The control machines do not have a hard OS requirement. I would really like to stay away from Multicast because of complications with upstream routers. Is Multicast or BitTorrent the better way to go on this? Is there another protocol that might work better?

WMIF
  • 340

4 Answers4

12

You really do want a Multicast File Transfer Program: UFTP, with decent documentation and proxt-style extensions for NAT/router traversal too.

dbush
  • 153
  • 9
Chris S
  • 78,455
5

You might want to look into murder

Murder is a method of using Bittorrent to distribute files to a large amount of servers within a production environment. This allows for scaleable and fast deploys in environments of hundreds to tens of thousands of servers where centralized distribution systems wouldn't otherwise function

.

Murder was/is developed by the folks at twitter, and they're using it daily to distribute files.

Jens Timmerman
  • 926
  • 4
  • 12
4

A new solution may have appeared: BitTorrent Sync

Later edit: Actually these days I would probably recommend git-annex assistant or syncthing but there are many alternatives

ptman
  • 29,862
1

I might have an idea for you that would help. Forgive me as I don't fully understand why this needs to be so complex, but if your need is to keep it simple for the end user and yet make the data quick and easy to reach inside a LAN, perhaps you could go with a NAS device. I have a Synology DS1812+ it can RSYNC with another Synology drive or various RSYNC capable devices, it has a bit torrent capable application called "Download Station", I know you can download torrent files from the drive bay and I believe you can also create or post a torrent file with that application to allow others to download a file they need. It has apps for mobile devices both Apple and Android. It also can do FTP transfers. This drive bay could give you the ability to send a file to it quickly then disperse it amongst the LAN quickly and easily. I suggest placing the data inside the LAN just to make it faster access for the LAN users, but the beauty of these NAS devices is that you can put them anywhere online as long as they have a fast internet connection. Perhaps one of the higher end Synology NAS devices would be a good fit for your needs.

Synology has a virtual interface you can look it over to give you a better idea if this would be useful for you. I will paste the link below to the virtual interface

http://www.synology.com/products/dsm_livedemo.php?lang=us

This device also gives people the ability access their data via a web interface or mobile device application.

I hope this helps out and like I said, forgive me if I don't understand the question correctly.

Frank R
  • 141