Backing Up With Aperture
This article was Originally published in March 2007 by MacZealots magazine.
Apple's Aperture application is an extremely versatile digital photography workflow tool suited to quickly comparing, rating, organizing, finding, and presenting images immediately after the shoot. Much of Aperture's strength and ease of use comes from it's non-destructive workflow based around a sophisticated library and versioning system. In Aperture, once an image has been imported into a project in the library, the original file, called a master, is never modified. As images are adjusted and edited, instead of the space-intensive duplicate-edit-save cycle of Photoshop and other tools, Aperture's workflow stores changes as very small recipe files called versions. When a particular version of an image is displayed or exported, Aperture reads the master from disk and applies the recipe. Changes to a recipe can be made and unmade at any time and in any order, resulting in a non-linear, non-destructive, and very flexible workflow.
To support this workflow Aperture's libraries are complex. Versions are collected together into albums and albums and masters are stored in projects. Two different kinds of folders allow albums to be organized inside projects and projects to be grouped together. Thumbnails and previews assist the rapid display of images for navigation and a database provides fast access to a very large number of images. Metadata, such as captions, keywords, and ratings is stored alongside image data as well as in the database.
This sophistication and flexibility brings with it some challenges. The first is that of horsepower. By deferring image processing to the moment of display, Apple has ensured that its customers will want the fastest computer they can find, especially one that can process adjustments in hardware. Another, less glamorous but equally important challenge is how to protect Aperture's library against loss or damage. Destructive workflows with embedded metadata make backing up simple but very space-hungry: just copy every version of every image. Aperture's library doesn't have most of the images, just masters and recipes, and metadata is stored separately from the images. So what is the best approach to backing it all up?
Make A Copy Of The Library
The icon for an Aperture library looks like a box (or even a front-loading washing machine) and one approach to backing up the data it contains is simply to treat it as a monolithic chunk of storage and use the Finder to make a duplicate. That could take a while: 40,000 12MB images will fill a 500GB drive and take three to four hours to copy. And as soon as the library exceeds the size of the drive it will either have to be split into separate libraries -- very inconvenient for finding images-- or put onto an even bigger drive. Since the whole purpose of a back up is to have a recent copy of valuable data, copying the library is a task that will have to be completed often. However, the size of a typical library will likely prevent more than one copy a day being made.
Introduced in Aperture 1.5, support for referenced masters addresses some of the limitations of library size. By default, master image files are stored inside the library as managed masters. The benefit of managed master, as the name suggests, is that Aperture takes care of all of the file management. However, the cost is a very rapidly growing library. Referenced master image files can be stored anywhere: on any fixed, removable, or remote disk volume that the user choses and with any folder structure imaginable. In this situation the Aperture library stores only references to those files. This keeps the library at a much more reasonable and convenient size (still not exactly small -- it contains thumbnails and previews for all the image versions), but at the cost of an added file management burden for the photographer.
[Possibly a sidebar]A library with referenced masters will be faster to copy than one with only managed masters, but the duplicate will not include the referenced master images themselves. That must be done separately. use of referenced masters make backing up more complex and so more prone to error (and neglect).
There are two ways to restore images from a copy of the library. The first, quick and risky, is to make the backup copy the current Aperture library by just double-clicking it to launch Aperture. Then export the project or images that are needed and close the library. This is risky because Aperture writes to the library as part of its normal management function, leaving the backup no longer an accurate copy. The second way to restore, slow and safe, is to copy the backup (and any external masters) onto another drive and then open the new copy.
Because of the amount of time consumed, neither backing up nor restoring by copying the library look like an appealing way to protect valuable image data. Fortunately, Apple's product team have designed in a better way: vaults.
Use A Vault Instead
Aperture provides a handy way of backing up in the form of a feature called vaults. Any number of vaults can be created for a library, each one containing a complete copy of all the critical data in the library. By providing for multiple copies, Aperture allows the photographer to keep one or more back ups on removable drives and so stored and rotated off-site. A vault omits all the image thumbnails, all the previews, and the database. It's a bare-bones version of the library and so is somewhat smaller and faster to copy.
Vaults also have a trick up their sleeve. In the user's Library folder is the Application Support folder. That contains a folder called "Aperture", and inside that are several files that store all the centralized settings that Aperture needs. These include the keywords list, keyword sets, watermarks, plug-ins, custom folder naming presets, adjustment presets, and metadata presets. When a vault is written, all of that is written into a folder on the vault called Settings. Copying a library does not preserve this information because none of it is stored inside a library. The effect of this is that a vault will contain all the keywords in the keyword HUD, not just those stored alongside the images in this library.
Vaults also support incremental updates. While synchronizing with the Aperture library, only the updated information is copied, so allowing even very large libraries to be backed up frequently once the initial vault has been created and filled. One final chance for the careless photographer is also provided: all deleted images are put into folder alongside the vault so even an accidental deletion followed by a vault synchronization is not a total loss.
Unfortunately, an entire vault must be imported to get back as little as a single image. As of Aperture 1.5 there is no way to restore anything less. That's potentially a huge amount of data -- only a little less than the library -- so it can be very time-consuming. Worse, since vaults omit the thumbnails, the previews, and the database that the library contains, restoring from a vault requires a lengthy rebuilding step to regenerate the missing information. This can be a shock to a photographer working on deadline and trying to dig herself out of a hole. Even at a rate of one preview and one thumbnail a second, 40,000 masters will take more than eleven hours to process.
And vaults don't include referenced masters. Those are still the responsibility of the photographer.
How About Exporting Projects?
Aperture projects typically contain from between a few to a few thousand master images, depending on how they are used. The photographer defines exactly what a project represents for their workflow: a few shots of one person, a day's shooting, an entire trip, a visit to one client by three people, a city, a package delivered to a client, or practically anything that needs to group master images together. Alongside the masters, projects also contain albums (groups of versions), metadata, light tables, web sites, books, previews, and thumbnails. It is natural then to want to back up and restore these very reasonably-sized and complete collections of valuable work as a single chunk of data.
By using the Export Project function, any Aperture project can be exported very quickly to any mounted volume. The export is fast because it's just a copy of a piece of the library. The only material difference between an exported project and the original in the library is that the exported project also contains extra masters that are needed to recreate images from other projects that are used in its albums.
As a back-up a project is a good choice. Projects are usually kept fairly small because that helps for speed and flexibility in Aperture, and that also makes it much easier to fit onto back up media than a copy of the entire library. One disadvantage of backing up with projects is that unlike vaults Aperture does not support incremental updating, so making frequent project back ups will take some time -- a good reason for keeping projects small. Automator cannot be used to automate the process either: it has no ability to select or copy an Aperture project, so projects must always be manually backed up.
Another advantage over vaults is that projects with referenced masters can be exported with all the masters included. Just select the option in the export dialog to consolidate all the masters into the export:
Note that the option appears even if the project contains no referenced masters. A little confusing.
Restoring images from an exported project is straightforward and fast. The entire project can be imported into the library simply by dragging and dropping onto Aperture's library pane. And because a project already contains all the thumbnails and previews there is no wait while Aperture recreates them, as it must with a vault. The project is ready for use as soon as the data has been copied. A fairly large project with 1000 12MB images would take only about five minutes to restore from a Firewire drive. One small downside to restoring a project is that if a project that contains images in its albums that come from masters in other projects is imported, then those extra masters will come in too. This will result in duplicates of master images already in the library, which can be confusing if it is not expected.
The Paranoid Do Much More
So far I have made the assumption that restoring all the Aperture-specific information is sufficient for full image recovery. This is not the case if Aperture itself is lost or cannot be used. Even without being able to run Aperture, a library can still be opened and examined easily on Mac OS X. Control-click on the icon and select Show Package Contents to browse with the Finder and locate the master image files.
If high resolution JPG previews have been generated for image versions, then those will be in the library too, each stored in the same folder as the master, so for greater security it can be worth spending the time and disk space to generate previews for the best images in each project.
For the highest level of paranoia, masters and versions should be exported from Aperture at full size and depth in a portable, lossless format such as TIFF and stored on a universally-readable medium such as a DVD or a FAT32 hard drive. However this will be very time consuming and will use a huge amount of space: ten versions of one image in Aperture use only a little more than the space occupied by one, the master. If exported, the ten images will use full ten times as much storage. Exporting does have the advantage that it can be automated with Automator, and there is full control over the size, type, and quality of the images.
Alongside the images, all the metadata can be exported from Aperture as an XML file or a spreadsheet. However, exporting versions will omit one potentially important piece of information: how were the versions created from the master? That is encoded in the library, and that data is only useful to Aperture. A copy of the Aperture folder inside the Application Support folder should also be made: that includes the keyword list and keyword sets.
There is also the issue of organization that should not be glossed over. If exported images are used for backing up, then file naming and folder organization will have to be very carefully managed or else there will be no connection to the existing library contents, especially as time goes on and more and more images are added. To help this, Aperture offers automatic folder organization and image naming as part of the export process.
Backing up images only by exporting them is not wise, but as a complement to exported projects it adds another level of security.
What Do I Recommend?
There are several good ways to back up Aperture images.
The simplest, but definitely not the fastest, is to upload them to a service like Flickr, Amazon's S3, or Digital Railroad. Now that export plugins are available for many online photo services, the price and convenience make this an attractive option. However the modest bandwidth of most internet connections prevents this from being a universal solution. That 40,000 image library I have been using as an example would require more than 100 days of non-stop uploading to transfer over a 512k DSL connection.
Often a significant amount of work has just been completed -- such as a series of adjustments, or key wording, or a light table layout -- and it would be very valuable to make a quick copy to guard against any sort of crash or corruption (or stupidity). I recommend making a copy of that project every once in a while, either on the desktop, or on a fast local drive. Start it before you go to get coffee, and by the time you are done, it will be done.
For full back ups, forget about Aperture and duplicate the whole disk to at least a pair of Firewire drives that are rotated off site. Any time that widespread image loss has occurred the disk almost certainly has other problems so you will want to restore everything on that drive anyway. I use a utility called SuperDuper! to clone my drive each day. It works quickly, reliably, and automatically and creates a copy of my drive that I can boot from immediately. Best of all, it is incremental, so each daily back up takes only ten to thirty minutes. It has rescued me from two drive failures and several instances of stupidity over the last few years.
For the most complete and paranoid procedure, do all of the above, but for highly-rated images also export versions as TIFFs and include all the metadata in an XML or text file. Add JPG thumbnails at various sizes so that the back up itself can be indexed and browsed later on any platform.
Ultimately backing up is only one of a number of tactics that you should employ to protect your images. It is important to have a workflow that extends before and beyond Aperture so that the collection and archiving of images are also carefully managed. People failures contribute to at least as many disasters as do equipment failures, and worse, their timing and outcomes are far less predictable. Part of the rationale for backing up is to gain insurance for the unexpected as well as the unpredicted.
Apple's Aperture application is an extremely versatile digital photography workflow tool suited to quickly comparing, rating, organizing, finding, and presenting images immediately after the shoot. Much of Aperture's strength and ease of use comes from it's non-destructive workflow based around a sophisticated library and versioning system. In Aperture, once an image has been imported into a project in the library, the original file, called a master, is never modified. As images are adjusted and edited, instead of the space-intensive duplicate-edit-save cycle of Photoshop and other tools, Aperture's workflow stores changes as very small recipe files called versions. When a particular version of an image is displayed or exported, Aperture reads the master from disk and applies the recipe. Changes to a recipe can be made and unmade at any time and in any order, resulting in a non-linear, non-destructive, and very flexible workflow.
To support this workflow Aperture's libraries are complex. Versions are collected together into albums and albums and masters are stored in projects. Two different kinds of folders allow albums to be organized inside projects and projects to be grouped together. Thumbnails and previews assist the rapid display of images for navigation and a database provides fast access to a very large number of images. Metadata, such as captions, keywords, and ratings is stored alongside image data as well as in the database.
This sophistication and flexibility brings with it some challenges. The first is that of horsepower. By deferring image processing to the moment of display, Apple has ensured that its customers will want the fastest computer they can find, especially one that can process adjustments in hardware. Another, less glamorous but equally important challenge is how to protect Aperture's library against loss or damage. Destructive workflows with embedded metadata make backing up simple but very space-hungry: just copy every version of every image. Aperture's library doesn't have most of the images, just masters and recipes, and metadata is stored separately from the images. So what is the best approach to backing it all up?
Make A Copy Of The Library
The icon for an Aperture library looks like a box (or even a front-loading washing machine) and one approach to backing up the data it contains is simply to treat it as a monolithic chunk of storage and use the Finder to make a duplicate. That could take a while: 40,000 12MB images will fill a 500GB drive and take three to four hours to copy. And as soon as the library exceeds the size of the drive it will either have to be split into separate libraries -- very inconvenient for finding images-- or put onto an even bigger drive. Since the whole purpose of a back up is to have a recent copy of valuable data, copying the library is a task that will have to be completed often. However, the size of a typical library will likely prevent more than one copy a day being made.
Introduced in Aperture 1.5, support for referenced masters addresses some of the limitations of library size. By default, master image files are stored inside the library as managed masters. The benefit of managed master, as the name suggests, is that Aperture takes care of all of the file management. However, the cost is a very rapidly growing library. Referenced master image files can be stored anywhere: on any fixed, removable, or remote disk volume that the user choses and with any folder structure imaginable. In this situation the Aperture library stores only references to those files. This keeps the library at a much more reasonable and convenient size (still not exactly small -- it contains thumbnails and previews for all the image versions), but at the cost of an added file management burden for the photographer.
[Possibly a sidebar]A library with referenced masters will be faster to copy than one with only managed masters, but the duplicate will not include the referenced master images themselves. That must be done separately. use of referenced masters make backing up more complex and so more prone to error (and neglect).
There are two ways to restore images from a copy of the library. The first, quick and risky, is to make the backup copy the current Aperture library by just double-clicking it to launch Aperture. Then export the project or images that are needed and close the library. This is risky because Aperture writes to the library as part of its normal management function, leaving the backup no longer an accurate copy. The second way to restore, slow and safe, is to copy the backup (and any external masters) onto another drive and then open the new copy.
Because of the amount of time consumed, neither backing up nor restoring by copying the library look like an appealing way to protect valuable image data. Fortunately, Apple's product team have designed in a better way: vaults.
Use A Vault Instead
Aperture provides a handy way of backing up in the form of a feature called vaults. Any number of vaults can be created for a library, each one containing a complete copy of all the critical data in the library. By providing for multiple copies, Aperture allows the photographer to keep one or more back ups on removable drives and so stored and rotated off-site. A vault omits all the image thumbnails, all the previews, and the database. It's a bare-bones version of the library and so is somewhat smaller and faster to copy.
Vaults also have a trick up their sleeve. In the user's Library folder is the Application Support folder. That contains a folder called "Aperture", and inside that are several files that store all the centralized settings that Aperture needs. These include the keywords list, keyword sets, watermarks, plug-ins, custom folder naming presets, adjustment presets, and metadata presets. When a vault is written, all of that is written into a folder on the vault called Settings. Copying a library does not preserve this information because none of it is stored inside a library. The effect of this is that a vault will contain all the keywords in the keyword HUD, not just those stored alongside the images in this library.
Vaults also support incremental updates. While synchronizing with the Aperture library, only the updated information is copied, so allowing even very large libraries to be backed up frequently once the initial vault has been created and filled. One final chance for the careless photographer is also provided: all deleted images are put into folder alongside the vault so even an accidental deletion followed by a vault synchronization is not a total loss.
Unfortunately, an entire vault must be imported to get back as little as a single image. As of Aperture 1.5 there is no way to restore anything less. That's potentially a huge amount of data -- only a little less than the library -- so it can be very time-consuming. Worse, since vaults omit the thumbnails, the previews, and the database that the library contains, restoring from a vault requires a lengthy rebuilding step to regenerate the missing information. This can be a shock to a photographer working on deadline and trying to dig herself out of a hole. Even at a rate of one preview and one thumbnail a second, 40,000 masters will take more than eleven hours to process.
And vaults don't include referenced masters. Those are still the responsibility of the photographer.
How About Exporting Projects?
Aperture projects typically contain from between a few to a few thousand master images, depending on how they are used. The photographer defines exactly what a project represents for their workflow: a few shots of one person, a day's shooting, an entire trip, a visit to one client by three people, a city, a package delivered to a client, or practically anything that needs to group master images together. Alongside the masters, projects also contain albums (groups of versions), metadata, light tables, web sites, books, previews, and thumbnails. It is natural then to want to back up and restore these very reasonably-sized and complete collections of valuable work as a single chunk of data.
By using the Export Project function, any Aperture project can be exported very quickly to any mounted volume. The export is fast because it's just a copy of a piece of the library. The only material difference between an exported project and the original in the library is that the exported project also contains extra masters that are needed to recreate images from other projects that are used in its albums.
As a back-up a project is a good choice. Projects are usually kept fairly small because that helps for speed and flexibility in Aperture, and that also makes it much easier to fit onto back up media than a copy of the entire library. One disadvantage of backing up with projects is that unlike vaults Aperture does not support incremental updating, so making frequent project back ups will take some time -- a good reason for keeping projects small. Automator cannot be used to automate the process either: it has no ability to select or copy an Aperture project, so projects must always be manually backed up.
Another advantage over vaults is that projects with referenced masters can be exported with all the masters included. Just select the option in the export dialog to consolidate all the masters into the export:
Note that the option appears even if the project contains no referenced masters. A little confusing.
Restoring images from an exported project is straightforward and fast. The entire project can be imported into the library simply by dragging and dropping onto Aperture's library pane. And because a project already contains all the thumbnails and previews there is no wait while Aperture recreates them, as it must with a vault. The project is ready for use as soon as the data has been copied. A fairly large project with 1000 12MB images would take only about five minutes to restore from a Firewire drive. One small downside to restoring a project is that if a project that contains images in its albums that come from masters in other projects is imported, then those extra masters will come in too. This will result in duplicates of master images already in the library, which can be confusing if it is not expected.
The Paranoid Do Much More
So far I have made the assumption that restoring all the Aperture-specific information is sufficient for full image recovery. This is not the case if Aperture itself is lost or cannot be used. Even without being able to run Aperture, a library can still be opened and examined easily on Mac OS X. Control-click on the icon and select Show Package Contents to browse with the Finder and locate the master image files.
If high resolution JPG previews have been generated for image versions, then those will be in the library too, each stored in the same folder as the master, so for greater security it can be worth spending the time and disk space to generate previews for the best images in each project.
For the highest level of paranoia, masters and versions should be exported from Aperture at full size and depth in a portable, lossless format such as TIFF and stored on a universally-readable medium such as a DVD or a FAT32 hard drive. However this will be very time consuming and will use a huge amount of space: ten versions of one image in Aperture use only a little more than the space occupied by one, the master. If exported, the ten images will use full ten times as much storage. Exporting does have the advantage that it can be automated with Automator, and there is full control over the size, type, and quality of the images.
Alongside the images, all the metadata can be exported from Aperture as an XML file or a spreadsheet. However, exporting versions will omit one potentially important piece of information: how were the versions created from the master? That is encoded in the library, and that data is only useful to Aperture. A copy of the Aperture folder inside the Application Support folder should also be made: that includes the keyword list and keyword sets.
There is also the issue of organization that should not be glossed over. If exported images are used for backing up, then file naming and folder organization will have to be very carefully managed or else there will be no connection to the existing library contents, especially as time goes on and more and more images are added. To help this, Aperture offers automatic folder organization and image naming as part of the export process.
Backing up images only by exporting them is not wise, but as a complement to exported projects it adds another level of security.
What Do I Recommend?
There are several good ways to back up Aperture images.
The simplest, but definitely not the fastest, is to upload them to a service like Flickr, Amazon's S3, or Digital Railroad. Now that export plugins are available for many online photo services, the price and convenience make this an attractive option. However the modest bandwidth of most internet connections prevents this from being a universal solution. That 40,000 image library I have been using as an example would require more than 100 days of non-stop uploading to transfer over a 512k DSL connection.
Often a significant amount of work has just been completed -- such as a series of adjustments, or key wording, or a light table layout -- and it would be very valuable to make a quick copy to guard against any sort of crash or corruption (or stupidity). I recommend making a copy of that project every once in a while, either on the desktop, or on a fast local drive. Start it before you go to get coffee, and by the time you are done, it will be done.
For full back ups, forget about Aperture and duplicate the whole disk to at least a pair of Firewire drives that are rotated off site. Any time that widespread image loss has occurred the disk almost certainly has other problems so you will want to restore everything on that drive anyway. I use a utility called SuperDuper! to clone my drive each day. It works quickly, reliably, and automatically and creates a copy of my drive that I can boot from immediately. Best of all, it is incremental, so each daily back up takes only ten to thirty minutes. It has rescued me from two drive failures and several instances of stupidity over the last few years.
For the most complete and paranoid procedure, do all of the above, but for highly-rated images also export versions as TIFFs and include all the metadata in an XML or text file. Add JPG thumbnails at various sizes so that the back up itself can be indexed and browsed later on any platform.
Ultimately backing up is only one of a number of tactics that you should employ to protect your images. It is important to have a workflow that extends before and beyond Aperture so that the collection and archiving of images are also carefully managed. People failures contribute to at least as many disasters as do equipment failures, and worse, their timing and outcomes are far less predictable. Part of the rationale for backing up is to gain insurance for the unexpected as well as the unpredicted.
The Bagelturf site welcomes Donations of any size