Herd is a torrent-based file distribution system based on Murder. It allows for quick and easy transfer of small and large files. You can probably use it for other things too.
Herd requires no extra Python modules and includes everything needed for destinations including its own (lightly modified) copy of BitTornado.
Herd has been updated from its fork Horde which removed limitations around large file transfers, peer seeding, and python integration. Herd now has the same capabilities as Horde in that regard.
Murder was Twitter's original BitTorrent-based file distribution system. It's pretty dependent on Capistrano and requires that a separate tracker process is started before you run the deploy task.
Herd spawns its own tracker in the background which makes it really trivial to integrate into whatever deployment system you like.
Herd needs Python > 2.5 and argparse. If you're using python 2.7+ nothing else is needed as argparse was added to the standard library.
Argparse, if needed, can be installed with easy_install or pip:
pip install argparse
easy_install argparse
Herd also requires that key based passwordless authentication is enabled on all of your target destinations.
git clone https://github.com/russss/Herd
cd Herd && sudo python setup.py install
Herd also supports pypi_server hosting.
With a hosts file that includes a list of the hosts you want to copy to:
herd myfile.tar.gz /path/to/destination.tar.gz hosts_file
Using a hosts list that is a single string comma separated:
herd myfile.tar.gz /path/to/destination.tar.gz --hostlist "host1,host2,host3"
Reading the host list from stdin.
cat /tmp/hosts_file | herd myfile.tar.gz /path/to/destination.tar.gz hosts_file
More options:
herd --help
Herd can also be imported as a python module. This makes integration into existing projects easy. One would simply need to:
import herd.herd as herd
herd.run_with_opts('localfile', 'remotefile', hostlist='server1,server2')