The rsync utility program is one of the most powerful (and dangerous) tools around when it comes to keeping directories and files synchronized. Many people fiddle around with it, get burned, and never touch it again - however, it's a really useful tool once you learn a few of its quirks.
In this document, we'll take a look at how rsync works and go through some examples ranging from the simple to the ridiculously complex. There are a myriad of ways to do all this stuff - this document is by no means an exhaustive study of rsync, just a handful of examples and some explanation. If you want the complete story on rsync, go get the O'Reilly book on it.
Keep these simple guidelines in mind when playing with rsync:
In this example, I'm just syncing one directory with another. The 'source' directory here happens to be an NFS mount of a remote machine's directory, but that doesn't matter - you can sync any two directories on your machine this way.
#!/usr/bin/ksh
#
# Use 'rsync' to keep local copies of
# patchinfo files up-to-date with master
# copies on remote.machine
RSYNC=/software/opt/sfw/bin/rsync
SOURCEDIR=/net/remote.machine/foo/bar
$RSYNC --archive --quiet --delete --delete-excluded \
${SOURCEDIR}/ /spare_1/patchinfos
Simple, eh? Just remember that trailing slash on the "from" directory and explicitly specify the destination directory (no variables that can get messed up) and you're done.
In addition to just syncing local directories (or things that appear as local directories, like NFS shares), you can use rsync to access files on remote systems. This is really what rsync is all about, and it's quite good at it.
The simplest way to do this is to operate rsync over an rsh connection to the remote machine. You can also do this over ssh, but that's left as an exercise for the student.
You don't have to know a lot about rsh to set up a simple password-less connection between two machines. Let's imagine a setup like this:
|
|
You're FRED, and you want to sync a local directory on your ALPHA machine from a directory accessible to BARNEY on the remote OMEGA machine. In setups like this, both FRED and BARNEY are likely to be special accounts set up for automated software execution, not home accounts used by actual people.
Log in as BARNEY on the OMEGA machine and create a file called .rhosts in BARNEY's home directory. Put this line in it:
ALPHA FRED
This has the effect of allowing FRED on the ALPHA machine to log in (rsh) as BARNEY on OMEGA without a password. As you can imagine, it's easy to create some fairly serious security loopholes this way, so don't do this lightly.
If ALPHA is actually something like alpha.central.foo.com, you need to explicitly spell all that out.
Once you've got BARNEY's .rhosts file in place, log into ALPHA as FRED. Then, try this:
rsh -l BARNEY OMEGA
You should immediately find yourself logged into OMEGA as BARNEY - no password prompt, no challenge/response, just "plop" and you're there, looking at a command prompt. If this is not what happens, seek professional help from your sysadmins - I'm not going to go into all the ways rsh can be fouled up.
Once you've got rsh set up for this automatic login behavior, you can run rsync on top of it. Here's a script that runs rsync over rsh to sync a remote machine directory over to a local directory:
#!/usr/bin/ksh
#
# Use 'rsync' to keep local copies of
# patchinfo files up-to-date with master
# copies on remote.machine
#
# This version of the script uses an 'rsh'
# session against the 'BARNEY' user account
# on OMEGA, rather than relying on
# the horribly slow NFS mounts.
LOCAL_RSYNC=/software/opt/sfw/bin/rsync
REMOTE_RSYNC=/opt/SENSrsync/bin/rsync
SOURCEDIR=/builds/foo/bar/quux
$LOCAL_RSYNC --archive --quiet --blocking-io \
--delete --delete-excluded \
--rsh="rsh -l BARNEY" --rsync-path ${REMOTE_RSYNC} \
OMEGA:${SOURCEDIR}/ /spare_1/patchinfos
The --rsh option specifies the command-line option FRED would use to establish an rsh session logged in as BARNEY on OMEGA, without the OMEGA machine name. Adjust as needed. The OMEGA:${SOURCEDIR}/ syntax alerts rsync to perform a remote connection rather than look at a locally-accessible directory. The --rsync-path option tells your local rsync where to find the rsync executable on the remote machine - when running rsync this way, your local rsync process fires up a partner rsync process on the remote system. The two rsync processes examine their respective directories and work together to move data and synchronize the systems.
The password-less login trick is not strictly required, but it's handy. You could fool around with hard-coding the remote account password in the script on ALPHA, but that's not generally a good idea. Setting up the .rhosts file on OMEGA means that FRED never needs to know BARNEY's password; the BARNEY password can be changed, too, without notifying FRED.
In situations where the remote machine is a production server and strict steps must be taken to avoid data loss or corruption, sysadmins often create the BARNEY account with group privileges that allow only read access to the file systems being read by rsync. Then, FRED (connected over rsh as BARNEY) can read the data but can't hurt it, even by accident.
In this example, I've got a huge amount of data stored on a remote machine, and I want to sync only selected files out of the remote directory for local storage on my machine. The remote machine has a directory that contains over 20,000 subdirectories that follow a naming convention like 10000-01, 234567-42, 654321-98, and so on. I want to sync up the following information to my local file system:
Here's the ksh script:
#!/usr/bin/ksh
#
# Let's see if we can rsync just the README and pkginfo
# and patchinfo files from the expanded patch directories
# over to local disk..
#
# This version of the script uses an 'rsh'
# session against the 'BARNEY' user account
# on OMEGA, rather than relying on
# the horribly slow NFS mounts.
LOCAL_RSYNC=/software/opt/sfw/bin/rsync
REMOTE_RSYNC=/usr/local/bin/rsync
SOURCEDIR=/patches/intpatchroot/all_patchdb/patchdb
begin=`date`
echo "Started rsync of patchinfo, pkginfo and READMEs at $begin"
# The first 'include' picks up the patch directory (123456-78)
# The second 'include' picks up pkg subdirectories (SUNWfoo)
# The next three includes grab pkginfo, patchinfo, and README
# The final 'exclude' ignores everything else (including
# subdirectory trees).
$LOCAL_RSYNC --archive --quiet --blocking-io \
--rsh="rsh -l BARNEY" --rsync-path ${REMOTE_RSYNC} \
--include "[0-9][0-9][0-9][0-9][0-9][0-9]-[0-9][0-9]/" \
--include "[0-9][0-9][0-9][0-9][0-9][0-9]-[0-9][0-9]/*/" \
--include "patchinfo" --include "pkginfo" --include "README*" \
--exclude "*" \
--delete --delete-excluded \
OMEGA:${SOURCEDIR}/ \
/spare_1/patchdata_rsh
end=`date`
echo "Finished rsync of patchinfo, pkginfo and READMEs at $end"
Ta-da! You can assemble some truly spectacular combinations of include and exclude directives; see the man page for rsync (or a good book) for more information.
In this example, I want to sync up selected subdirectories based on the contents of a couple of control files. The control files contain simple lines of text - strings in the form ######-## (one entry per line). The idea here is that some other process can manage the contents of the control files, and my script can dynamically adapt to the new set of requested subdirectories.
Here's the script:
The sort and sed game sucks the ######-## entries out of the two control files and dresses them up as rsync include expressions, prepending a forward slash and appending two asterisks to each ######-## entry. In other words 123456-78 becomes /12345678** in the $RSYNC_LIST file. The list of include expressions is fed into rsync as a file, rather than as a stack of command-line arguments.
The syntax and coding of include directives is weird and subtle - in this case, we're syncing selected subdirectories from /net/foo/bar into /spare_1/suspects, recursing all the way down each selected ######-## subdirectory.
If you pay attention and take some time to learn the subtleties of rsync, you'll find it to be one of the most useful and powerful tools in your toolbox. You can pull off some really spectacular miracles with rsync, and you can just as easily create titanic messes by being careless. Be careful!