ShotGo

Download

Make it go.

1. Use rsync to synchronize the photo data (in my case, 30GB in 10,000 files) to my server,
2. Use rsync to squirt a copy of Shotwell's sqlite3 metadata database up to my server,
3. Use ssh to invoke shotgo on the server to synchronize Piwigo's mysql database, and
4. Click on Piwigo's "maintenance/Update albums information" function.

How the metadata is synchronized:

• Each photo in Shotwell lives in an Event (a hierarchically arranged category). That event hierarchy is mirrored in Piwigo.
• Photos with the Shotwell tag hide are not mentioned at all in Piwigo's metadata. Assuming the web server is securely configured, this means 'hide' photos aren't ever served over the web, regardless of any Piwigo misconfiguration.
• Photos with the Shotwell tag public are given Piwigo Privacy Level 0, so that guests can see them. All other photos are given level 2, which means a user must have Privacy Level "friends" or above to see it.
• Shotwell tags and rotation information is propagated into Piwigo.

Optionally: Piwigo will sometimes point the browser directly at the photo source directory such as when the original photo is small enough that it doesn't need downconverting. That means that the entire photo source tree must be visible to all comers, logged in or not, which means that the non-public photos are only as secret as their URLs. I think that's a security flaw in Piwigo.

We can't do much about it for non-public photos, but photos tagged hide shouldn't be web-accessible at all, even if they're living in the file tree accessible by the web server. To get that property,

• Invoke shotgo with the --hide-with-chmod flag to chmod all of the photos tagged hide with mode 0400. This means that the owner can read them, but not the web-server user (www-data on Debian).

Limitations.

• Currently, shotgo replaces the entire images table every time it runs. My assumption is that the Piwigo site is not used to manually edit or manage photos; the script synchronizes in one direction only and will happily destroy any manual work done via the Piwigo interface.
• Shotgo should use Piwigo APIs to synchronize the database, if that's the Right Way to diddle with Piwigo, rather than reaching into its database guts. I don't feel too guilty about this behavior, since I'm sort of lobotomizing poor Piwigo anyway. :v)
• At a minimum, expose the "update albums information" function as a Piwigo WS API so I can automate the entire publishing process.
• Crop and image enhancements applied in Shotwell are ignored. Fixing this would require exporting intermediate versions of the modified data to the web server, which would probably require tighter Shotwell integration.
• Create/preserve meaningful Piwigo "available" dates, so that the "new photo" icons are meaningful.
• Selection of ``key photos'' for events are not yet carried over.

Motivation.

But of course I want to share my photos with the family. I've posted a few to Flickr (with Shotwell's publish tool) and SMS'ed a few one-offs. What I really want is a way to do all the management in Shotwell, and then simply synchronize (not publish) the photos to my web server.

In my case, I already back up my photo album from my laptop to my closet server. I can have my web server NFS-mount the photos read-only, so the bulk of the data, the photos themselves, don't require any extra transfers or management; the one rsync script that does backup does all the work.

Choosing publishing software.

I played with Gallery3, which is looking pretty slick, but its permission model wasn't flexible enough for my needs. I want all of my photos to be visible in their Shotwell event (category) hierarchy to friends and family, and I want guests to see same hierarchy, but with all of the non-public photos vanished. Gallery has a per-category permission model, which prevents this sort of layout without (as far as I can tell) duplicating all the public data.

I played with OpenPhoto. Its software design scared me a little bit: it writes ImageMagick temp files right into the html/ directory, alongside HTML files, and writes thumbnails and other data into the directories that hold the original photos. The former carries the scent of poor security design; the latter interfered with my desire to have a pristine data directory untouched after photo import. (Metadata comes and goes, but the ``negatives'' should be forever.)

I played with PhotoShow. This software is wonderfully easy to install: no database configuration, it just moves right into the filesystem. By preparing a tree of symlinks into a directory with execute/no-read permissions, I could convince PhotoShow to see the category layout from Shotwell but the actual image data from the pristine source directory. So I built my first script around photoshow. It worked fine at 10 photos, but once I imported all 10,000, it simply melted. Hitting its front page caused it to touch every file in the entire tree; it appears to be fundamentally unscalable. (This sort of this is, after all, what databases are for.) Darn.

Piwigo was actually the first package I tried, since Shotwell ships with a publish plugin for it. Unfortunately, "publish" isn't really what I want to do: it requires that I know which photos I haven't already published. And it also involves reorganizing the photos at publish time.

Piwigo also has some sort of mechanism for syncing to a local (to the webserver) file tree, which I played with, but of course it created metadata that reflected the on-disk layout, since it knew nothing of shotwell's metadata, such as its category (event) tree.

After some experimentation, I discovered that:

1. Piwigo's interface satisfies my goals for family-vs-guests.
2. Piwigo has a pretty clean, well-normalized database schema.
3. It dynamically creates all necessary thumbnails and web-sized images perfectly transparently, and stores them in an appropriate out-of-the-way directory.
4. While it doesn't take too kindly to directly munging its database records, one click on a maintenance function causes it to invalidate its cache databases and cope with the new data.
5. Its performance is fantastic, even after a full-database cache invalidation.

Things I learned.

From my quick introduction, it's a fantastic point in the design space: it's a serious high-level language, with features comparable to C# and Java, including static type safety, runtime type information, and automatic memory management. On the other end, it admits a very tidy, fast implementation, with no runtime VM or garbage collector; in fact, it's easy to make libraries in Vala that you can call from C. This feels like the language C++ wanted to be, if we knew then what we know now.

I can't wait to try it. Maybe my project will be to translate ShotGo from Python to Vala. :v)

Shotwell's TagTable.
I hope someone on the Shotwell team attends to the horrifically-designed TagTable; it completely misses the purpose and power of using a relational database. It stores a tag as a single row

("kitten","photo1,photo7,photo10,photo93")

("kitten","photo1")
("kitten","photo7")
("kitten","photo10")
("kitten","photo93")