Synchronizing Files

File synchronization between people and our own computers / devices has an interesting history and tradition.  In 2014, we all use dropbox or one of its competitors, along with whatever corporate standard our companies want us to use.  Many of us have dozens of accounts on several free services.

Among the reasons we need to copy stuff are:

• “protection” – making sure we can recover our stuff if a device breaks, gets lost, stolen or your house burns down.
• “safety,” – making sure all and only the people who can get at the stuff should be allowed and do have access.
• “sharing” – quickly, simply, elegantly getting grand-mom that video or picture of her granddaughter playing soccer.
• “collaboration” – quickly, simply, seamlessly making live updates to that presentation, document, source code, video, or meeting.

Side Bar:  The Career of Ray Ozzie

One of my favorite biographies about the captains of our software industry (and file synchronization fanatic) is that of Ray Ozzie.  In one respect, Ray is a “one trick pony” who kept re-implementing file synchronization his entire career.  In 1974 he worked on PLATO where he helped develop email, messaging, forums, multiplayer games, and other file / message sharing.  In 1979 he developed file serving (sharing) on Data General Nova.  1983 – 1997 he worked on Lotus Notes, which is really just a glorified file sharing system.  Then he founded Groove Networks to do file sharing over the public Internet.  In 2005 – 2008 at Microsoft, he developed “Windows Live Mesh,” which became Windows Live SkyDrive, a system which is, (wait for it) file sharing over the public Internet.

Syncing Photos On Kiosk Monitors at The eBay Bellevue Office

When eBay set up its third, big “Seattle” (really Bellevue) office in 2013, I was asked to help set up the monstrous “Kiosk” (public) monitors to display interesting content to employees walking the halls.  I was called in late to the project, after the hardware, software, and networking were chosen and installed.  Each 62” monitor has a $50 Rasperry Pi embedded system   $1,000 mac mini driving the display. (I would have chosen the Rasperry Pi, but as mentioned, I was not consulted until they needed someone to “put something up on the displays.”)  The monitors themselves are actually quite beautiful and blend in well with the offices’ interior design.

In keeping with the motif of having each floor of the building decorated, named, and themed, the monitors display content specific to their location and themed appropriately.  Every 5 seconds a new photo or web page is displayed.

Syncing Files to the Monitors

When I first implemented the system I wanted to enable every employee in the building to submit photos that could be displayed.  So I created a file share on a pair of corporate-network connected, and centrally-administered file servers running on virtual machines in a data center.  Developers at eBay have an awesome, internal “cloud” (I hate that word) service similar to amazon web service AWS.  We can provision virtual machines, storage, and set up clusters of machines very quickly.

I chose the “WebDAV” protocol because it works with Windows, Apple MacOS, and Linux.  I wrote up screen-shot-heavy instructions on our wiki for all three operating systems and announced the share to everyone in the office.  Among the hundreds of people I contacted, 2-3 submitted some photos and one person added a few photos after I pleaded with him for help.  (Note to self:  If your request does not help employee with her review score do not expect help on your project.)

Dev Cloud Glitches

The AWS-like internal service happened to have huge, frequent outages that took down all of my virtual machines and storage for long periods of time.  Although the service is usually quite reliable, it happened to have three major outages while I was building this system, two of which required me to re-image and re-configure my VMs and storage.

But worst of all was re-visiting each and every mac mini via that silly Virtual Network Computing (VNC) remote desktop system Apple uses.  If there were only a handful of devices, I suppose the task would not be so annoying.  But there are a whole bunch of these monitors and  I am not very good at OS/X.  I noticed OS/X runs sshd (encouraging) but I don’t understand in which text files those gooey applications (photo screen saver on OS/X) stores its configuration.  When OS/X cannot access a WebDav mounted share, it gives up and does not re-try to mount the drive periodically. So, again, I had to VNC in to each device to re-mount the share.

Rsync to the rescue!  Loose Coupling

I have always been a big fan of the rsync algorithm because of its elegance and also because of the efficiency for both network traffic and CPU by its use of rolling checksums for blocks.  I look for excuses to drop in rsync to solve problems.  And one fell right into my lap.

"Special Programming" on the monitors for high-level exec visits forced the single photos folder to split into separate folders for "classes" (locations) of Kiosk machines.  My system accommodated this change easily.  It is, after all, a single point of deployment.  I just split the single directory into separate directories. But I also thought about a better, "loosely coupled" design and decided to copy each monitor's content locally to the mac mini machines that drive the monitors.  This local "cache" of photos provides immunity to network and  server failures, enabling each monitor machine to display local content from its cache when it cannot “see” the server.  When the network or server or storage issue is repaired, the content synchronizes.  Yippee!  I get to use rsync. 

Mac OS/X Implementation

First, I needed to re-mount the central photos folder each time the machine restarts or loses its mount.  For the reboot case on OS/X one simply puts an entry in /etc/rc.common:

##################################
# Mount eBay pictures directory #
##################################
mount_webdav http://hostname.domain/folder /mnt/seapics

This same command must be re-run periodically in case the remote server becomes unavailable for any reason.

Then we need a wrapper around rsync that can be run out of cron (the scheduled task system on OS/X) that includes mutual exclusion to prevent several instances of the same script from running at the same time:

rsync script on each kiosk monitor machine

#!/usr/bin/env bash
# sync-photos.sh -- synchronize eBay photos between display and server source
# Instead of a license we offer this blessing:
#
# May you do good and not evil;
# May you find forgiveness for yourself and forgive others;
# May you share freely, never taking more than you give.
 
ORIGIN=/mnt/seapics/3rd-floor           # server pictures location
SERVER=/Users/mwyle/Pictures/3rd-floor  # local cache of pictures
LOCKFILE=/var/run/rsync-photos-lock     # lock file for mutex
# Interrupted, killed, die, exit? Clean up mutex lock!
trap "/bin/rm -f $LOCKFILE" EXIT SIGHUP SIGINT SIGTERM
if shlock -f ${LOCKFILE} -p $$  ; then
  rsync -a $ORIGIN $SERVER
else
  echo photo sync already running; ${LOCKFILE} held by $(cat ${LOCKFILE})
fi

 

And finally, we need an entry in root's crontab on each server to synchronize its photos every few minutes:

*/10 * * * * /var/root/sync-photos.sh

All of the components used in this solution run on Linux and Windows in addition to OS/X.  Therefore, this solution would work with thousands of Kiosk monitors driven by MS-Windows machines, Rasperry-Pi embedded systems, Google Chromcast dongles, or most other systems used to drive public Kiosk monitors.