Hazards of Referenced Masters -- Bone-Headedness

Hello, I stumbled onto your page looking for help with Aperture., I'm new to digital photography and I think I accidentally moved my, master files to a removable hard drive and then deleted them, thinking, that they were saved in a vault somewhere else. Obviously I don't know, what I'm doing, but is there anyway to get aperture to let me work, with the library images that it has on my hard drive? All of my, pictures are there, in good enough resolution, but aperture won't let, me do anything with them because I don't have the masters. No, exporting, no emailing, no editing, nothing., Am I stuck with looking at these pictures forever and that is it?, Thanks a bunch for any help.

I see a quite a few postings on message boards and get emails from Aperture users who have done disastrous things to their referenced masters because they didn't realize that the files were still part of their Aperture library. Referenced masters are master image files stored outside the Aperture library. This feature allows a small internal hard drive, such as on a laptop, to maintain a very large image library where the originals are stored on a separate removable drive or central storage system. In the other kind of master storage, managed masters, the masters live inside the Aperture library itself. While not impossible, damaging managed masters takes some persistence and the barrier formed by the library protects them against most bone-headed errors.

That referenced masters live outside the library leaves them prone to several kinds of abuse. Moving them by hand is harmless, unless the move goes to another volume. That will break the connection with the library and require a reconnect. Aperture's Referenced File Manager does this well, but it is very fussy about restoring the connection. If the image has been edited, for instance, it will likely not reconnect. If the pixel dimensions have changed, it will not reconnect. If the file size has changed, it will not reconnect. And this is where the big problems start. Since Aperture only checks these things when reconnecting, problems can go undetected for a very long time. A reconnect is needed and suddenly many masters (and hence their versions) are effectively lost.

The best medicine, then, is prevention. So how do you go about protecting referenced masters? They could be stored anywhere and called anything -- what kind of barriers can be constructed to protect them?

Part 1: Name Defensively

When you import, rename the masters in such a way that all of them can be seen to be masters. Prefixing with "MAS" or an equivalent short word will make masters instantly recognizable and you will no longer feel compelled to trash them in haste.

The input screen allows a selection for the version name and can optionally rename the masters:
defensive11
Above that setting is a block of information that gives confirmation of the change if an image is selected:
defensive15
Another recommendation is to rename the masters in such a way that all of them have unique names. For instance, add the current date to the name given by the camera. This will ensure that as the camera or card numbering rolls over, the images still have unique names. While not critical, this defensive step may help in the future when it is necessary to list or index all of the images. Ensuring unique naming now will obviate managing duplicates and messing with hierarchies later on.

At the bottom of the Version Name drop-down the Edit... option allows the naming schemes to be edited. Here is how MAS prefix used above is defined:
defensive9

Part 2: Manage, then Relocate

Always import new images into the library as managed masters as a first step. Then edit, cull, rate, tag, stack as usual. Then finally move the masters out of the library using the Relocate command to a reserved area of your disk and add another prefix to show that they are referenced, such as "REF". Finally, possibly much later, delete the rejects.
defensive10
You can relocate masters in two ways: either by selecting individual images and from the File menu going to Relocate Masters For Library... or by control-clicking on a project and selecting Relocate Masters for Project to relocate an entire project full of images at once:
defensive14
Relocating the masters also has the ability to rename as it moves. To relocate and rename at the same time, set up a new Name Format preset from the Relocate masters sheet:
defensive8
By clicking on the Name Format drop-down and selecting Edit... Give the new name format a name and set it up something like this:
defensive7
Then select that new name format and do the relocate. As the files are moved, the names will be changed. Here is a referenced master on the disk after it was relocated (I used a slightly different prefix than above in this example to show this image is referenced, omitting the dash):
defensive17
The original image was called 6830-1.JPG. Importing added MAS-2005-04-20 and relocating added REF.

Why work this way? This workflow keeps all the images that are still being worked on in one place so they can easily be found with a smart album that shows only managed masters. This workflow means that if it's managed, then you're not done with it. The library becomes a staging area. Once relocated and renamed, the master files are immediately identified as being referenced from their name and you know that they have already been processed and so are ready for use or to be archived. And, since importing into the library makes a copy, the originals are still on the card or disk they came from and another layer of corruption insurance has been created.

Creating a smart album to show only managed files is straight forward. Create a new smart album by clicking on the magnifying glass next to the library (so it will apply globally) and add a File Status filter:
defensive12
Then filter on Managed status and check the Ignore stack groupings box so that stacks don't hide any images:
defensive13

Part 3: Organize Masters For Growth

Relocating and renaming masters imported into Aperture helps to ensure that they will not be accidently altered, misplaced, or deleted as referenced files.

But when they are relocated, where should they be put? Everyone has a different system for doing this. Often that system arose from a need to either find images or process them, but these are requirements that Aperture does not have of disk storage: the library and its tools take over that organizational role.

What is left to organize? There must be some logic to the folder structure. My answer is to organize around change and minimize risk. What can change?
Adding Storage
The first thing that changes is that the disk fills up and more is needed. Unless you have a RAID system that can be transparently expanded, you must either add a new disk to your computer and split the masters across both, or replace the old disk with the new one and copy everything over. Which is the better approach depends on how the masters have been organized, so ideally your master organization is planned according to your plans for expansion. Do you have any plans for expansion?

If you are storing referenced masters primarily chronologically, say by month, then you add a new drive and all the new images get put on the new drive and the old ones stay on the old drive. It's a quick upgrade and you are unlikely to accidently delete or damage anything. Further, you can stop backing up the old drive: it will never change. Just keep the old off-site back up until you get rid of the smaller drive a few years from now. One small catch is that the addition of the new drive will happen mid-month. So do you have two July folders, one on each disk, or move the July folder to the new disk and then continue adding to it? I recommend the latter, and will be looking at how that can be achieved in this article.

If you are not storing referenced masters primarily chronologically, then the approach is different. Masters organized by client and then by project cannot be handled the same way as the strictly chronological system because any client could ask for another project and overrun an already-full disk. In this case it makes sense to copy everything over and stop using the old disk. Copying between disks can take a little while, even with fast disks -- about an hour per 100GB -- so this is something that may take some planning. The catch with this method of storage expansion is that backing up will need to include both drives now, so don't forget to change your settings or procedures to do that.

Both method of adding storage require copying, and Aperture can do it for you. In fact you should always use Aperture to do the copying. In that way Aperture always knows where its library masters are at all times and reconnecting is never needed.

To move referenced masters to another drive using Aperture, relocate them using exactly the same system of organization that was in use on the old drive. You already have a preset for this because the current organization or referenced masters was built with it.
Archiving
In deciding how to organize referenced masters there is more to consider than just storage expansion. The other change that occurs is that archiving is needed: some images no longer need to be at-hand and can be stored more cheaply or in a place that is not immediately available. These are not back-ups (copies stored short-term as insurance that you hope to never need) -- they are archives (originals stored long-term with the expectation that they will be needed).

I described one folder systems based primarily on date, and another based on client and project. But there are others. Which one is best? In my mind the best folder system is one that Aperture can create and maintain and that can be adequately backed up incrementally.

That may sound restrictive because you may not want to look at a library organized that way, but remember that the folder organization on the disk does not have to follow the library organization at all. For instance your library may be organized by client and then city (both using blue folders) and then by project because your work involves travel to different locations for each client to shoot vacation accommodation. But since renovation is common, you almost never need access to images that are more than three years old. So you organize your referenced masters on the disk by year and project (using Finder folders) and archive a whole year at a time to DVDs or a hard drive each time you start a new year. Note that the library still contains the thumbnails and the metadata for all images, allowing you to view, tag, and find those other images at any time.

Organize masters to reflect how you archive images and manage storage. Organize the library to reflect how you find and work with images.

Part 4: Write Protect

For maximum paranoia against accidental modification or deletion, write protect the master image files. This can be easily achieved from the Finder.

From the Finder I select the top-level folder of the folder structure to protect and press command I to bring up the Info window. I open up the permissions part at the bottom, and it looks like this:
defensive1
By changing the Details pop-ups to Read Only and applying it to all enclosed items, the settings are propagated to all the files and folders:
defensive2
Now if I try to modify a file, I get a dialog like this:
defensive3
I get an opportunity to override by authenticating, but usually I would not want to, just accepting the OK button. If I drag a file to the trash, I get the same dialog.

To change everything back, I use the same procedure, this time setting the permissions to Read and Write for myself.
The Bagelturf site welcomes Donations of any size