Aaron Curley

Posted on October 20, 2013 by Aaron

10-21-2013 Update:  Corrected a few errors in the write-up and fixed the template configuration files.

Introduction

I remotely manage the computers & networks for various family members (parents, siblings, and grandparents).  To this end, I currently have router-to-router VPN links consistently established between home networks at five distinct geographic locations.  (These links are maintained using OpenVPN running on my OpenWrt-based gateway routers.)

As administrator of this geographically-diverse home network, I need to consistently back up my family’s data in case disaster strikes.  While there are many reputable cloud-backup solutions available, I prefer to trust my own skills to manage and protect my (and my family’s) data.

As we all know, a key backup strategy is to have off-site backups.  Fortunately, because my VPN spans multiple geographic locations, the off-site backup problem is easy – provided I can periodically synchronize files between computers at different locations.  Unfortunately, my VPN links are limited to approximately 300 KB/s due to the limited uplink bandwidth.  For me, this is where rsync comes in.

What is Rsync?

Rsync is a tool for *NIX.  From the rsync man page, “rsync … [is] … a fast, versatile, remote (and local) file-copying tool.” For me, attractive capabilities of rsync include:

• Delta-transfer algorithm – Only sends the differences between the “old” and “new” versions of the file when synchronizing over a network
• Supports compression of file data when sent via network
• Supports various modes for determining which files need to be transferred
  • “Quick check” comparison (file size & timestamp)
  • Checksum comparison (MD5)
• Transactional behavior (supports a mode where changes are not “committed” unless entire transfer was successful)
• Ability to resume partially-completed file synchronizations (without starting over)

Using “rsync”, the process of synchronizing files over slow VPN links can be made much less painful. The remainder of this entry documents installing and setting up rsync on Windows systems.

Installing Cygwin

I’m a Windows guy. Rsync is not (natively) a Windows tool.  While there are many ways to get rsync for Windows, the easiest way (in my opinion) is to install Cygwin.

(Honestly, even if you don’t care about running rsync, if you don’t have Cygwin installed on your Windows box, you probably should rectify that situation immediately.  Go do it now.  And yes, for those who are diehard Windows fans, I checked whether rsync can run on Subsystem for UNIX-based applications, and it doesn’t sound like it…bummer.)

Tips for Installing Cygwin so that it’s “rsync-ready”:

1. Download the Cygwin installer.
2. Verify the integrity of the Cygwin installer using GPG or whatever PGP tool you use.  For reference, Cygwin’s current public key’s is as follows:
   1. DSA key ID:  676041BA
   2. Fingerprint: 1169 DF9F 2273 4F74 3AA5  9232 A9A2 62FF 6760 41BA
3. Run the installer.  When you get to the point of selecting packages, be sure to install both the “rsync” and “cygrunsrv” packages.  You may also want “nano” if you’re not a “vi” person.
4. After installation, if your Windows user account contains spaces, eliminate those spaces from your Cygwin account and your home folder.  Do this by:
   1. Editing /etc/passwd to change the home directory and user name
   2. Renaming your home directory in /home
   3. (Re-)starting the Cygwin shell

Perform the above steps on *both* the rsync client and the rsync server.

Understanding Rsync Modes of Operation

Rsync can access files on a remote host using two different methods.  From the man pages, “There are two different ways for rsync to contact a remote system: using a remote-shell program as the transport (such as ssh or rsh) or contacting an rsync daemon directly via TCP.”

Of course, the advantages of using the remote-shell approach are obvious – no need to open additional ports and encryption of transmitted information.  However, in my case, I had no need of encryption (due to my VPNs) and I would have to open up ports regardless (my Windows boxes were not running any remote-shell daemon).  I therefore decided to get rsync working as a daemon on the server.

Installing Rsync as a Windows Service (Server-Side Only)

The equivalence to “daemons” in the Windows world are “services.” To install Rsync as a Windows service, a few steps are necessary and not exactly obvious.  In researching the problem, I found an online tutorial that provided instructions; however, some steps didn’t seem quite right to me from a security standpoint.  With some experimentation, I came up with a procedure that worked (on Windows 7, at least):

1. Create a new Windows user account called “rsync” with a random password & remember it for later.  This account will be used to run the rsync service.  Use the following settings:
   1. User cannot change password
   2. Password never expires
   3. No group memberships (i.e. remove the default “users” group).
2. Use secpol.msc –> Local Policies –> User Rights Assignment to configure the user privileges for the “rsync” user:
   1. Deny rsync “log on locally”
   2. Deny rsync “access computer from network”
   3. Deny rsync “log on through remote desktop services”
   4. Allow rsync “log on as a service”
   5. Note:  Not all Windows editions support secpol.msc.  There are ways to set these configuration settings using other mmc snap-ins, but doing so is outside the scope of this blog.
3. Using Windows Explorer’s file permission editor, grant the Windows rsync account FULL CONTROL rights to any folder that will be synchronized remotely via rsync.
4. From the Cygwin shell as root, run ‘mkpasswd > /etc/passwd’ to update the password file with the latest MS Windows user account list.
   1. Note:  If you had to manually update the /etc/passwd file to remove spaces within your Cygwin username, execute ‘mkpasswd >> /etc/passwd’ instead and then manually edit the file to remove the duplicate information added.  This way you won’t overwrite your previous changes.
5. Generate the rsyncd.conf configuration file:
   1. Download the above template file.
   2. Edit the template configuration file to add the folders you wish to make available via the rsync daemon.  Note that your “c:\” drive is mounted under the /cygdrive/c/ Cygwin path.
   3. Copy the file to the Cygwin path /etc/rsyncd.conf
   4. Use chmod to set the file permissions to 644 (rw- r– r–).
6. Generate the rsyncd.secrets file:
   1. Download the above template file.
   2. Edit the file to set the rsync daemon password.  Since we’re using plaintext communication (not counting any VPN) don’t use a password you use anywhere else.
   3. Copy the file to the Cygwin path /etc/rsyncd.secrets
   4. Use chown to change the owner of this file to “rsync”
   5. Use chgrp to change the group of this file to “root”
   6. Use chmod to set the permissions to 660 (rw- rw- —).
7. Install the Windows service using the following command within an administrator command prompt:
   C:\cygwin\bin\cygrunsrv.exe -I "rsync" -p /cygdrive/c/cygwin/bin/rsync.exe -a "--config=/cygdrive/c/cygwin/etc/rsyncd.conf --daemon --no-detach" -f "Rsync daemon service"
   1. Note: This assumes you’ve installed Cygwin to C:\cygwin. You should adjust the paths above if you installed Cygwin somewhere else.
8. Use services.msc to:
   1. Configure the now-installed rsync service to run under the “rsync” account.  You’ll need the random password you typed in earlier.
   2. Configure service autostart settings as desired.
   3. Start the rsync service.
9. Edit your firewall settings to allow inbound port TCP 873 for rsync.exe.

Rsync on the Client

At this point, we have Cygwin installed on both the client and the server machines.  We also have rsync running as a daemon on the server machine as a Windows service.  So, all we need to do now is run rsync on the client machine.

There are, however, some complications.  First and foremost, there are a boatload of rsync options available, many of which we’d like to invoke during each and every rsync operation.  Secondly, due to the /etc/rsyncd.secrets file, we will need to provide a password during every rsync connection, which may be undesirable.  Finally, for lengthy transfers, a network glitch may interrupt rsync, and we may wish to have it automatically restarted for us after a brief period of waiting.

Fortunately, we can automate all of the above (and more) by writing a few simple shell scripts:

(IMPORTANT: Read the cautionary note below before attempting to use any of these scripts.)

• rsync.sh – Calls rsync with a set of default options (specified at the start of the script file). If run in “remote” mode, automatically loads the rsync password for the remote host from a file (named hostnamehere.rsyncpw) in the home directory. The script also implements a –loop option for auto-retrying failed synchronizations.
  Usage: ./rsync.sh local|remote [--loop] [additional rsync options] source dest
  1. Note: For the script to work, when specifying the remote host, use the format “username@remotehostname::FoldernameHere/”

  Example:
  ./rsync.sh remote --dry-run /cygdrive/c/myfiles/ aaron@desktop::myfiles/

• rsync_local.sh – Provides a short cut for calling the above “rsync.sh” script; always provides the “local” option to the script.  This allows for a syntax such as:
  ./rsync_local.sh --dry-run /cygdrive/c/myfiles/ /cygdrive/c/myfiles2/
• rsync_remote.sh – Provides a short cut for calling the above “rsync.sh” script; always provides the “remote” option to the script. This allows for a syntax such as:
  ./rsync_remote.sh --dry-run /cygdrive/c/myfiles/ aaron@desktop::myfiles/
• rsync_batch.sh – Automates the execution of multiple separate rsync operations using a separate file containing a list of synchronizations to perform.
  Usage: ./rsync_batch.sh check|sync timestamp|checksum rules_file [additional rsync options]
  • check|sync – Option “check” runs the batch set of rsync commands with the –dry-run option to prevent any changes from being made. Option “sync” actually performs the synchronization.
  • timestamp|checksum – Option “timestamp” uses timestamps and file sizes (the default mode) for file comparison. Option “checksum” uses the MD5 checksum for file comparison.
  • rules_file – Specifies the rules file to use for the batch synchronization. See the example.

For ease of use, after downloading the scripts, drop them in your Cygwin home directory, and use chmod +x to mark them executable.

CAREFUL: Being a file synchronization program, if used incorrectly, rsync can easily damage/destroy large amounts of your data. So, be VERY careful when running rsync with or without the above scripts. I AM NOT RESPONSIBLE for any damage to your computer and/or files through the use of the scripts provided above!

I highly recommend that you always first invoke rsync with the –dry-run option to view the likely results of the synchronization without actually changing anything.

Conclusion

I hope this guide helps anyone setting up rsync on their Windows systems.