Referenced
Fraser Speirs Shares His Workflow
2008-01-07
Fraser Speirs (of FlickrExport fame) shares his Aperture workflow:
He also has a more recent blog entry entitled When I Delete a Photo (almost never, basically). I have a different philosophy: I reject many photos and delete all the rejects about once a month.Someone asked me recently about how I work through my photographs after I shoot. When I go out to photograph, I shoot a lot - most people I go out with are usually surprised at the number of frames I produce. It’s not uncommon for me to take the kids to the park and come home with 150-300 images.I find that I can usually edit down 350 images to around 50 in 40-60 minutes and I thought I would share how I go about this.
|
Aperture: Recover Images With File Juicer
2007-12-28
File Juicer is a small Mac OS X application that extracts images, movies, text, and other useful data from practically anything. It's useful to Aperture users in two important ways: it can recover usable images from the library if the masters are lost and and can scrounge deleted images from memory cards.
One of the hazards of working with referenced image masters is that their management is the responsibility of the owner. Accidental deletions are not that uncommon, and if that happens then while Aperture can display the images, it cannot export or otherwise use any of the versions that are based on the lost masters. If the masters are truly lost -- no back ups, nothing in the trash -- then whatever images can be found become valuable.
If high resolution previews were generated, then these can be extracted from the Aperture library by simply selecting the thumbnail images in the browser and dragging them to the desktop. They will be in JPEG format and at a size and resolution that depends on the settings in Aperture's preferences:
If there are no previews, then attention turns to the thumbnail files that Aperture stores in each project. It is these images that are used to display the on-screen thumbnails in the browser pane and as placeholder images in the viewer while Aperture processes the RAW image. The files that contain the thumbnails are called AP.Tinies, AP.Minis, and AP.Thumbnails and contain images at 32, 256, and 1024 pixel sizes respectively. They are also present in exported projects, but not in vaults.
To get to the thumbnails, I control-click on the library and select Show Package Contents. Then I navigate down to the project of interest and open that with a control-click and Show Package Contents:
The AP.Thumbnails file is one big chunk of binary data, but inside there are complete JPEG images. File Juicer will go into it and locate and extract the JPEGs without knowing the format of the file.
I launch File Juicer and check that the preferences are set to include JPEG images (at least):
I also make sure that the extracted files will be stored somewhere sensible, such as on the desktop, because I don't want the extracted images put inside my Aperture library:
With the selections I have made, File Juicer will put each image type into a separate folder and create a parent folder for those. It will also get an HTML index file for easy browsing. To start scanning for images, I drop the AP.Thumbnails file from my project onto the main File Juicer window and wait for it to process:
After processing I get a new folder on my desktop containing the images:
And I can either open the jpg folder and browse the image icons in the Finder (or watch a slide show), or click on the index.html icon and see all the images in a browser window as a panoramic display:
Now my images have been extracted, I can reimport them into Aperture and sort through them. They will be smaller then the originals -- only up to 1024 pixels on a side-- and there will be one image per version. So a single lost master will result in five recovered JPEGs if it had five versions in that project. This is good because I get my adjusted images, albeit at low resolution.
Since File Juicer is scavenging for JPEGs rather than following any information that Aperture provides, there are some side-effects. The first is that there may be old images or possibly corrupted images in the folder of JPEGs. The second is that the names of the images are sequential and bear no relationship to the order in which they were taken or anything else. The third is that there is no EXIF or other metadata in the JPEGs, so all the keywords, camera and shooting data are lost.
File Juicer will also recover RAW and other images that have been deleted from camera cards, so if masters have been lost it is possible that they can be obtained that way. The process is very similar to the thumbnail recovery described here, except that there is an extra step at the beginning where File Juicer creates a disk image of the card and scans that.
The File Juicer web site has a page dedicated to its use with Aperture, and one about RAW image formats.
Aperture: What Are The Pros And Cons Of Managed vs. Referenced Masters?
2007-11-19
Here is my list of pros and cons for referenced and managed masters:
Pros for Managed
- No file management needed
- Low risk
- All masters backed up with the library
- Can use vaults for complete back up
- Always have everything with you
Cons for Managed
- Potentially huge library
- Other applications can't access images
- Can archive by project only
- Adding storage requires that everything is copied
Pros for Referenced
- Organize and store the masters where you like
- Access them from other applications
- Masters can be backed up separately from the rest of the system
- Potentially sharable among many users and systems
- Can archive in many different ways
- Can be simple and fast to add storage
Cons for Referenced
- Must organize, manage, and protect the masters
- Risk of being caught by permissions problems
- Risk of Aperture not reconnecting them
- Must remember to back them up separately from everything else
- Not all images are with you all of the time if using a laptop
- Vaults don't have all the data
- Offline referenced masters cause filter difficulties
Aperture: Don't Import Unsupported Images As Referenced Masters
2007-10-26
A short Knowledgebase article from Apple:
Another good reason for using managed masters, or just not dealing with unsupported cameras at all.Referenced images from cameras that are newly supported sometimes cannot be connected after you update to a version of Mac OS X that supports them.
Aperture: How Do I Consolidate The Images I Have Shared With iPhoto Without Creating A Problem For iPhoto?
2007-10-03
You can do this very easily using Aperture's Consolidate command. Select the images and go to File > Consolidate Master... or for an entire project control click on the project and choose Consolidate Masters for Project...:
While consolidating usually moves the master files into the Aperture library (so causing iPhoto grief), there is an option to copy the files instead. This leaves the originals intact:
Of course now you have two masters on your disk, one in iPhoto and one in Aperture. Once you have confirmed that Aperture has the photos in its library as expected, you can delete the iPhoto copies.
Aperture: Hazards of Referenced Masters -- Bone-Headedness Part 4
2007-07-19
This is the last in a series of short articles about how to protect referenced masters from one of their worst natural enemies: bone-headedness. From Part 1: The best medicine, then, is prevention. So how do you go about protecting referenced masters? They could be stored anywhere and called anything -- what kind of barriers can be constructed to protect them?
From the Finder I select the top-level folder of the folder structure to protect and press command I to bring up the Info window. I open up the permissions part at the bottom, and it looks like this:
By changing the Details pop-ups to Read Only and applying it to all enclosed items, the settings are propagated to all the files and folders:
Now if I try to modify a file, I get a dialog like this:
I get an opportunity to override by authenticating, but usually I would not want to, just accepting the OK button. If I drag a file to the trash, I get the same dialog.
To change everything back, I use the same procedure, this time setting the permissions to Read and Write for myself.
Part 4: Write Protect
For maximum paranoia against accidental modification or deletion, write protect the master image files. This can be easily achieved from the Finder.From the Finder I select the top-level folder of the folder structure to protect and press command I to bring up the Info window. I open up the permissions part at the bottom, and it looks like this:
By changing the Details pop-ups to Read Only and applying it to all enclosed items, the settings are propagated to all the files and folders:
Now if I try to modify a file, I get a dialog like this:
I get an opportunity to override by authenticating, but usually I would not want to, just accepting the OK button. If I drag a file to the trash, I get the same dialog.
To change everything back, I use the same procedure, this time setting the permissions to Read and Write for myself.
Aperture: Hazards of Referenced Masters -- Bone-Headedness Part 3
2007-07-14
This is a series of short articles about how to protect referenced masters from one of their worst natural enemies: bone-headedness. From Part 1: The best medicine, then, is prevention. So how do you go about protecting referenced masters? They could be stored anywhere and called anything -- what kind of barriers can be constructed to protect them?
But when they are relocated, where should they be put? Everyone has a different system for doing this. Often that system arose from a need to either find images or process them, but these are requirements that Aperture does not have of disk storage: the library and its tools take over that organizational role.
What is left to organize? There must be some logic to the folder structure. My answer is to organize around change and minimize risk. What can change?
If you are storing referenced masters primarily chronologically, say by month, then you add a new drive and all the new images get put on the new drive and the old ones stay on the old drive. It's a quick upgrade and you are unlikely to accidently delete or damage anything. Further, you can stop backing up the old drive: it will never change. Just keep the old off-site back up until you get rid of the smaller drive a few years from now. One small catch is that the addition of the new drive will happen mid-month. So do you have two July folders, one on each disk, or move the July folder to the new disk and then continue adding to it? I recommend the latter, and will be looking at how that can be achieved in this article.
If you are not storing referenced masters primarily chronologically, then the approach is different. Masters organized by client and then by project cannot be handled the same way as the strictly chronological system because any client could ask for another project and overrun an already-full disk. In this case it makes sense to copy everything over and stop using the old disk. Copying between disks can take a little while, even with fast disks -- about an hour per 100GB -- so this is something that may take some planning. The catch with this method of storage expansion is that backing up will need to include both drives now, so don't forget to change your settings or procedures to do that.
Both method of adding storage require copying, and Aperture can do it for you. In fact you should always use Aperture to do the copying. In that way Aperture always knows where its library masters are at all times and reconnecting is never needed.
To move referenced masters to another drive using Aperture, relocate them using exactly the same system of organization that was in use on the old drive. You already have a preset for this because the current organization or referenced masters was built with it.
I described one folder systems based primarily on date, and another based on client and project. But there are others. Which one is best? In my mind the best folder system is one that Aperture can create and maintain and that can be adequately backed up incrementally.
That may sound restrictive because you may not want to look at a library organized that way, but remember that the folder organization on the disk does not have to follow the library organization at all. For instance your library may be organized by client and then city (both using blue folders) and then by project because your work involves travel to different locations for each client to shoot vacation accommodation. But since renovation is common, you almost never need access to images that are more than three years old. So you organize your referenced masters on the disk by year and project (using Finder folders) and archive a whole year at a time to DVDs or a hard drive each time you start a new year. Note that the library still contains the thumbnails and the metadata for all images, allowing you to view, tag, and find those other images at any time.
Organize masters to reflect how you archive images and manage storage. Organize the library to reflect how you find and work with images.
Part 4 has been posted.
Part 3: Organize Masters For Growth
Relocating and renaming masters imported into Aperture helps to ensure that they will not be accidently altered, misplaced, or deleted as referenced files.But when they are relocated, where should they be put? Everyone has a different system for doing this. Often that system arose from a need to either find images or process them, but these are requirements that Aperture does not have of disk storage: the library and its tools take over that organizational role.
What is left to organize? There must be some logic to the folder structure. My answer is to organize around change and minimize risk. What can change?
Adding Storage
The first thing that changes is that the disk fills up and more is needed. Unless you have a RAID system that can be transparently expanded, you must either add a new disk to your computer and split the masters across both, or replace the old disk with the new one and copy everything over. Which is the better approach depends on how the masters have been organized, so ideally your master organization is planned according to your plans for expansion. Do you have any plans for expansion?If you are storing referenced masters primarily chronologically, say by month, then you add a new drive and all the new images get put on the new drive and the old ones stay on the old drive. It's a quick upgrade and you are unlikely to accidently delete or damage anything. Further, you can stop backing up the old drive: it will never change. Just keep the old off-site back up until you get rid of the smaller drive a few years from now. One small catch is that the addition of the new drive will happen mid-month. So do you have two July folders, one on each disk, or move the July folder to the new disk and then continue adding to it? I recommend the latter, and will be looking at how that can be achieved in this article.
If you are not storing referenced masters primarily chronologically, then the approach is different. Masters organized by client and then by project cannot be handled the same way as the strictly chronological system because any client could ask for another project and overrun an already-full disk. In this case it makes sense to copy everything over and stop using the old disk. Copying between disks can take a little while, even with fast disks -- about an hour per 100GB -- so this is something that may take some planning. The catch with this method of storage expansion is that backing up will need to include both drives now, so don't forget to change your settings or procedures to do that.
Both method of adding storage require copying, and Aperture can do it for you. In fact you should always use Aperture to do the copying. In that way Aperture always knows where its library masters are at all times and reconnecting is never needed.
To move referenced masters to another drive using Aperture, relocate them using exactly the same system of organization that was in use on the old drive. You already have a preset for this because the current organization or referenced masters was built with it.
Archiving
In deciding how to organize referenced masters there is more to consider than just storage expansion. The other change that occurs is that archiving is needed: some images no longer need to be at-hand and can be stored more cheaply or in a place that is not immediately available. These are not back-ups (copies stored short-term as insurance that you hope to never need) -- they are archives (originals stored long-term with the expectation that they will be needed).I described one folder systems based primarily on date, and another based on client and project. But there are others. Which one is best? In my mind the best folder system is one that Aperture can create and maintain and that can be adequately backed up incrementally.
That may sound restrictive because you may not want to look at a library organized that way, but remember that the folder organization on the disk does not have to follow the library organization at all. For instance your library may be organized by client and then city (both using blue folders) and then by project because your work involves travel to different locations for each client to shoot vacation accommodation. But since renovation is common, you almost never need access to images that are more than three years old. So you organize your referenced masters on the disk by year and project (using Finder folders) and archive a whole year at a time to DVDs or a hard drive each time you start a new year. Note that the library still contains the thumbnails and the metadata for all images, allowing you to view, tag, and find those other images at any time.
Organize masters to reflect how you archive images and manage storage. Organize the library to reflect how you find and work with images.
Part 4 has been posted.
Aperture: Hazards of Referenced Masters -- Bone-Headedness Part 2
2007-07-10
This is a series of short articles about how to protect referenced masters from one of their worst natural enemies: bone-headedness. From Part 1: The best medicine, then, is prevention. So how do you go about protecting referenced masters? They could be stored anywhere and called anything -- what kind of barriers can be constructed to protect them?
You can relocate masters in two ways: either by selecting individual images and from the File menu going to Relocate Masters For Library... or by control-clicking on a project and selecting Relocate Masters for Project to relocate an entire project full of images at once:
Relocating the masters also has the ability to rename as it moves. To relocate and rename at the same time, set up a new Name Format preset from the Relocate masters sheet:
By clicking on the Name Format drop-down and selecting Edit... Give the new name format a name and set it up something like this:
Then select that new name format and do the relocate. As the files are moved, the names will be changed. Here is a referenced master on the disk after it was relocated (I used a slightly different prefix than above in this example to show this image is referenced, omitting the dash):
The original image was called 6830-1.JPG. Importing added MAS-2005-04-20 and relocating added REF.
Why work this way? This workflow keeps all the images that are still being worked on in one place so they can easily be found with a smart album that shows only managed masters. This workflow means that if it's managed, then you're not done with it. The library becomes a staging area. Once relocated and renamed, the master files are immediately identified as being referenced from their name and you know that they have already been processed and so are ready for use or to be archived. And, since importing into the library makes a copy, the originals are still on the card or disk they came from and another layer of corruption insurance has been created.
Creating a smart album to show only managed files is straight forward. Create a new smart album by clicking on the magnifying glass next to the library (so it will apply globally) and add a File Status filter:
Then filter on Managed status and check the Ignore stack groupings box so that stacks don't hide any images:
Part 3 has been posted.
Part 2: Manage, then Relocate
Always import new images into the library as managed masters as a first step. Then edit, cull, rate, tag, stack as usual. Then finally move the masters out of the library using the Relocate command to a reserved area of your disk and add another prefix to show that they are referenced, such as "REF". Finally, possibly much later, delete the rejects.You can relocate masters in two ways: either by selecting individual images and from the File menu going to Relocate Masters For Library... or by control-clicking on a project and selecting Relocate Masters for Project to relocate an entire project full of images at once:
Relocating the masters also has the ability to rename as it moves. To relocate and rename at the same time, set up a new Name Format preset from the Relocate masters sheet:
By clicking on the Name Format drop-down and selecting Edit... Give the new name format a name and set it up something like this:
Then select that new name format and do the relocate. As the files are moved, the names will be changed. Here is a referenced master on the disk after it was relocated (I used a slightly different prefix than above in this example to show this image is referenced, omitting the dash):
The original image was called 6830-1.JPG. Importing added MAS-2005-04-20 and relocating added REF.
Why work this way? This workflow keeps all the images that are still being worked on in one place so they can easily be found with a smart album that shows only managed masters. This workflow means that if it's managed, then you're not done with it. The library becomes a staging area. Once relocated and renamed, the master files are immediately identified as being referenced from their name and you know that they have already been processed and so are ready for use or to be archived. And, since importing into the library makes a copy, the originals are still on the card or disk they came from and another layer of corruption insurance has been created.
Creating a smart album to show only managed files is straight forward. Create a new smart album by clicking on the magnifying glass next to the library (so it will apply globally) and add a File Status filter:
Then filter on Managed status and check the Ignore stack groupings box so that stacks don't hide any images:
Part 3 has been posted.
Aperture: Hazards of Referenced Masters -- Bone-Headedness Part 1
2007-07-06
Hello, I stumbled onto your page looking for help with Aperture., I'm new to digital photography and I think I accidentally moved my, master files to a removable hard drive and then deleted them, thinking, that they were saved in a vault somewhere else. Obviously I don't know, what I'm doing, but is there anyway to get aperture to let me work, with the library images that it has on my hard drive? All of my, pictures are there, in good enough resolution, but aperture won't let, me do anything with them because I don't have the masters. No, exporting, no emailing, no editing, nothing., Am I stuck with looking at these pictures forever and that is it?, Thanks a bunch for any help.
I see a quite a few postings on message boards and get emails from Aperture users who have done disastrous things to their referenced masters because they didn't realize that the files were still part of their Aperture library. Referenced masters are master image files stored outside the Aperture library. This feature allows a small internal hard drive, such as on a laptop, to maintain a very large image library where the originals are stored on a separate removable drive or central storage system. In the other kind of master storage, managed masters, the masters live inside the Aperture library itself. While not impossible, damaging managed masters takes some persistence and the barrier formed by the library protects them against most bone-headed errors.
That referenced masters live outside the library leaves them prone to several kinds of abuse. Moving them by hand is harmless, unless the move goes to another volume. That will break the connection with the library and require a reconnect. Aperture's Referenced File Manager does this well, but it is very fussy about restoring the connection. If the image has been edited, for instance, it will likely not reconnect. If the pixel dimensions have changed, it will not reconnect. If the file size has changed, it will not reconnect. And this is where the big problems start. Since Aperture only checks these things when reconnecting, problems can go undetected for a very long time. A reconnect is needed and suddenly many masters (and hence their versions) are effectively lost.
The best medicine, then, is prevention. So how do you go about protecting referenced masters? They could be stored anywhere and called anything -- what kind of barriers can be constructed to protect them?
This the first of a four-part series on protecting referenced masters.
The input screen allows a selection for the version name and can optionally rename the masters:
Above that setting is a block of information that gives confirmation of the change if an image is selected:
Another recommendation is to rename the masters in such a way that all of them have unique names. For instance, add the current date to the name given by the camera. This will ensure that as the camera or card numbering rolls over, the images still have unique names. While not critical, this defensive step may help in the future when it is necessary to list or index all of the images. Ensuring unique naming now will obviate managing duplicates and messing with hierarchies later on.
At the bottom of the Version Name drop-down the Edit... option allows the naming schemes to be edited. Here is how MAS prefix used above is defined:
Part 2 has been posted.
I see a quite a few postings on message boards and get emails from Aperture users who have done disastrous things to their referenced masters because they didn't realize that the files were still part of their Aperture library. Referenced masters are master image files stored outside the Aperture library. This feature allows a small internal hard drive, such as on a laptop, to maintain a very large image library where the originals are stored on a separate removable drive or central storage system. In the other kind of master storage, managed masters, the masters live inside the Aperture library itself. While not impossible, damaging managed masters takes some persistence and the barrier formed by the library protects them against most bone-headed errors.
That referenced masters live outside the library leaves them prone to several kinds of abuse. Moving them by hand is harmless, unless the move goes to another volume. That will break the connection with the library and require a reconnect. Aperture's Referenced File Manager does this well, but it is very fussy about restoring the connection. If the image has been edited, for instance, it will likely not reconnect. If the pixel dimensions have changed, it will not reconnect. If the file size has changed, it will not reconnect. And this is where the big problems start. Since Aperture only checks these things when reconnecting, problems can go undetected for a very long time. A reconnect is needed and suddenly many masters (and hence their versions) are effectively lost.
The best medicine, then, is prevention. So how do you go about protecting referenced masters? They could be stored anywhere and called anything -- what kind of barriers can be constructed to protect them?
This the first of a four-part series on protecting referenced masters.
Part 1: Name Defensively
When you import, rename the masters in such a way that all of them can be seen to be masters. Prefixing with "MAS" or an equivalent short word will make masters instantly recognizable and you will no longer feel compelled to trash them in haste.The input screen allows a selection for the version name and can optionally rename the masters:
Above that setting is a block of information that gives confirmation of the change if an image is selected:
Another recommendation is to rename the masters in such a way that all of them have unique names. For instance, add the current date to the name given by the camera. This will ensure that as the camera or card numbering rolls over, the images still have unique names. While not critical, this defensive step may help in the future when it is necessary to list or index all of the images. Ensuring unique naming now will obviate managing duplicates and messing with hierarchies later on.
At the bottom of the Version Name drop-down the Edit... option allows the naming schemes to be edited. Here is how MAS prefix used above is defined:
Part 2 has been posted.
Aperture 1.5: Use Smart Albums To See Which Referenced Masters Are Available
2007-01-07
One problem with keeping referenced masters on removable media such as DVDs or Firewire drives is knowing what is actually on them. Mounting the disk or drive brings the masters on line and the badges change from this:
to this:
But it is difficult to see just these images among the thousands and register which ones are on the drive that just mounted.
An email from Johan Elzenga suggested an easy way to see just the referenced images that are on the mounted drive: use a pair of smart albums. This first one finds all offline images. It is set up to work only on images for my 2006-05 project:
The File status setting is found in the action (cog) menu on the top right.
Its sibling shows all masters that are referenced and online. Since managed masters are always online, the Match setting at the top is set to All and two conditions are needed:
This makes the album only show masters that are online and referenced and in the 2006-05 project. And with either of these I can also use additional filtering, searching, and sorting in the browser window where the images are displayed to narrow my choice further.
The neat thing about these smart albums is that they will change their contents as disks are mounted and removed. Put in a DVD, watch the album. Eject it and put an another. Repeat.
to this:
But it is difficult to see just these images among the thousands and register which ones are on the drive that just mounted.
An email from Johan Elzenga suggested an easy way to see just the referenced images that are on the mounted drive: use a pair of smart albums. This first one finds all offline images. It is set up to work only on images for my 2006-05 project:
The File status setting is found in the action (cog) menu on the top right.
Its sibling shows all masters that are referenced and online. Since managed masters are always online, the Match setting at the top is set to All and two conditions are needed:
This makes the album only show masters that are online and referenced and in the 2006-05 project. And with either of these I can also use additional filtering, searching, and sorting in the browser window where the images are displayed to narrow my choice further.
The neat thing about these smart albums is that they will change their contents as disks are mounted and removed. Put in a DVD, watch the album. Eject it and put an another. Repeat.
Aperture 1.5: Recover From Broken References to Edited Files
2006-12-16
If referenced masters are edited outside of Aperture without performing Open In External Editor and then the reference is broken, it may not be possible for Aperture to reconnect to the master, even by using the Referenced File Manager.
The reason for this is that to ensure the master being reconnected is the same image as the original, Aperture checks the file size. If editing changes the file size then the Reconnect button on the Referenced File manager will not be enabled.
Of course doing anything to masters behind Aperture's back like this is Bad News and is asking for trouble. To modify an image, whether referenced or managed, always use Open In External Editor, edit the image, save it (it goes into the library as a managed master), and then relocate the new master out of the library.
There are two ways out of this situation. The easy way is to delete the image with the broken reference (don't delete the referenced master!) and then reimport the master. This also loses all the adjustments and metadata, so may not be a suitable solution. The hard way, which I describe below, patches Aperture's library so that the file size information is correct and Aperture will reconnect the broken reference. This method does keep all the metadata and adjustments intact.
To allow Aperture to reconnect the master you have to correct the file size information that is in Aperture's database. To do that you have to get the correct file size, find and edit the image's apfile buried in the library, and then rebuild the database.
The inspector pane (I) shows the file name and the location of the master if it has been set up to do this:
Quit Aperture and navigate to your Aperture library in the Finder. Open the library using control-click Show Package Contents and navigate down into the library, through the blue folder hierarchy, and to the project folder:
Control click the project and Show Package Contents, then navigate to the image folder inside the project via the import group folder.
There it is: Pine tree chopping24.JPG.apfile
Press Return and note the number displayed before the date. That is the correct file size in bytes:
In this case the number is 2128603. The Finder will show a larger file size because it includes space used by the resource fork:
Close and save the apfile.
Repeat this for each of the images you need to recover and then close the library and project packages.
Once done, the images that had their sizes fixed should be reconnectable.
The reason for this is that to ensure the master being reconnected is the same image as the original, Aperture checks the file size. If editing changes the file size then the Reconnect button on the Referenced File manager will not be enabled.
Of course doing anything to masters behind Aperture's back like this is Bad News and is asking for trouble. To modify an image, whether referenced or managed, always use Open In External Editor, edit the image, save it (it goes into the library as a managed master), and then relocate the new master out of the library.
There are two ways out of this situation. The easy way is to delete the image with the broken reference (don't delete the referenced master!) and then reimport the master. This also loses all the adjustments and metadata, so may not be a suitable solution. The hard way, which I describe below, patches Aperture's library so that the file size information is correct and Aperture will reconnect the broken reference. This method does keep all the metadata and adjustments intact.
To allow Aperture to reconnect the master you have to correct the file size information that is in Aperture's database. To do that you have to get the correct file size, find and edit the image's apfile buried in the library, and then rebuild the database.
Find The Image apfile
Each master has an apfile, an XML file that describes the master. To find the apfile, inspect the image in Aperture and note its project, its location in the blue folder hierarchy, its import group, and its master file name. My example image is in the Tree Cutting project:The inspector pane (I) shows the file name and the location of the master if it has been set up to do this:
Quit Aperture and navigate to your Aperture library in the Finder. Open the library using control-click Show Package Contents and navigate down into the library, through the blue folder hierarchy, and to the project folder:
Control click the project and Show Package Contents, then navigate to the image folder inside the project via the import group folder.
There it is: Pine tree chopping24.JPG.apfile
Find the correct file size for the image master
Open the Terminal application and type "ls -l " (that is lower case L, lowercase S, space, hyphen, lowercase L, space). Locate the image file in the finder and drag the image file icon onto the terminal window and let go:Press Return and note the number displayed before the date. That is the correct file size in bytes:
In this case the number is 2128603. The Finder will show a larger file size because it includes space used by the resource fork:
Correct the file size in the apfile
Open the apfile using a text editor or XML editor (drag and drop the file onto the application icon of a text editor like TextWrangler in the dock). Locate the item fileSize and replace the size with the size you determined using the terminal above.Close and save the apfile.
Repeat this for each of the images you need to recover and then close the library and project packages.
Rebuild the Aperture database
Locate the Aperture application icon and option-command double-click. Choose to rebuild the database and wait for it to complete.Once done, the images that had their sizes fixed should be reconnectable.
Aperture 1.5: The Hazards of Referenced Masters -- Archives on DVDs
2006-12-14
The introduction of referenced masters in Aperture 1.5 came to many people with a feeling of relief. I can keep my masters where I want them to be! I can burn masters to DVD for archiving! I can use all those Firewire drives to store my images! In other words, I can keep on working the way I used to work: manually organizing storage and applying a cataloging application to that storage so that it is possible to find things.
As people who have been emailing me are finding out, there are hazards to using referenced masters. Aperture is not a cataloging application like iView: it's a librarian and a workflow tool. And using it like it is a cataloging application is going to result in problems.
The difference between a library and a catalog is important. A catalog is part of a library and simply works as an index, telling you where things are. A library actually holds the things that are referenced by the catalog. In the case of Aperture there are two options for the library: either store the images in the same place as the catalog as managed masters and have Aperture do the work, or store the masters externally in other places as referenced masters and you do the work. What work? There's nothing to do. Right?
Let's say you have been shooting for years and have diligently been burning everything to DVDs and cataloging them with Aperture. The day comes when you want to upload all the five star images in your collection to pBase. You create a smart album for the whole library and make it select the five star images. All the thumbnails appear and you select them and go to export them. But:
Well that is because the disk with the images on is not in the drive. But which disk to use? Control-click on an image and try to open in the Finder helps a little:
But that is only the information about one image and I have hundreds spread across many DVDs. The Referenced File Manager is more of a help:
It at least shows me the volume names of the images I have selected, so I know which disks to put in.
But wait. To export all of my hundreds of five-star images in one go I must mount all of the DVDs that contain those images at the same time! So if I have five DVDs that contain the images, I need five DVD players all plugged into my computer at the same time. The same is true of Firewire disks. Unless I can mount all of them at once, I cannot export a selection of images that spans them.
There are two possible solutions. The first is to make the images managed just for the export and then relocate them again when done. That is easier said than done when the media is read-only. The second is to split the export up into chunks so that each chunk only uses one DVD. An equally large pain because I have to do that manually: there is no filter that lets me chop up the images into groups by storage volume.
I have described one of the hazards of referenced masters. You do have work to do, you may just not realize it yet.
As people who have been emailing me are finding out, there are hazards to using referenced masters. Aperture is not a cataloging application like iView: it's a librarian and a workflow tool. And using it like it is a cataloging application is going to result in problems.
The difference between a library and a catalog is important. A catalog is part of a library and simply works as an index, telling you where things are. A library actually holds the things that are referenced by the catalog. In the case of Aperture there are two options for the library: either store the images in the same place as the catalog as managed masters and have Aperture do the work, or store the masters externally in other places as referenced masters and you do the work. What work? There's nothing to do. Right?
Let's say you have been shooting for years and have diligently been burning everything to DVDs and cataloging them with Aperture. The day comes when you want to upload all the five star images in your collection to pBase. You create a smart album for the whole library and make it select the five star images. All the thumbnails appear and you select them and go to export them. But:
Well that is because the disk with the images on is not in the drive. But which disk to use? Control-click on an image and try to open in the Finder helps a little:
But that is only the information about one image and I have hundreds spread across many DVDs. The Referenced File Manager is more of a help:
It at least shows me the volume names of the images I have selected, so I know which disks to put in.
But wait. To export all of my hundreds of five-star images in one go I must mount all of the DVDs that contain those images at the same time! So if I have five DVDs that contain the images, I need five DVD players all plugged into my computer at the same time. The same is true of Firewire disks. Unless I can mount all of them at once, I cannot export a selection of images that spans them.
There are two possible solutions. The first is to make the images managed just for the export and then relocate them again when done. That is easier said than done when the media is read-only. The second is to split the export up into chunks so that each chunk only uses one DVD. An equally large pain because I have to do that manually: there is no filter that lets me chop up the images into groups by storage volume.
I have described one of the hazards of referenced masters. You do have work to do, you may just not realize it yet.
Interview Number Two With Joe Schorr
2006-11-16
O'Reilly Digital Media has now posted a second interview with Joe Schorr, Apple's Product manager for Aperture. In this interview he talks about referenced files.
Aperture: How Do I Delete Rejects?
2006-11-16
What you are missing is that you have Primary Only turned on. It's very easy to do this by accident because the shortcut is the S key. Here is the button:
If turned on, whatever you do will only affect the primary (thick white outline) image, not all the others (thin white outline). Here are six images in the viewer (not thumbnails -- just a very small viewer pane) with Primary Only turned off:
And here is primary Only turned on. Notice that only the primary is shown with a line around:
Here are two ways of deleting rejects, each with its own advantages and disadvantages:
Delete Rejects In The Current Project or Album
Press keys in this sequence: Control 8, Command A, Command Delete, Return.
Control 8 shows just the rejects, removing any other filter in use. Command A selects all of them. Command Delete deletes them. Return accepts the dialog box. And the state of the Primary Selection Only does not matter to Command Delete. There is a catch with this method of deleting though -- this will not delete rejects that are inside stacks. But the second method below gets those too.
Delete Rejects In The Entire Library
I have a smart album that I set up to delete rejects across the entire library:
Notice that it was created with the Library selected so it applies across all projects. And that the Ignore Stack Grouping is checked. That lets it look inside stacks and means that I don't have to mess with opening and closing stacks all over to clear out the rejects.
I select the smart album, then command A, command Delete, Return. It can be good to command option B before all that to maximize the browser in order to give the images one last check.
Aperture: How Do I Delete Referenced Masters?
2006-10-29
There is a misunderstanding here that is causing some confusion. By dragging your photo folders into Aperture you created a project inside the library and also caused Aperture to copy those images into its library: your images are actually managed, not referenced. The dialog you saw:
Shows this to be the case. If you were attempting to delete any referenced images, then you would have seen this dialog:
which includes an option to trash the masters that are outside the library.
In order to import your images as referenced masters you will have to use the File > Import > Images or File > Import > Images Into a Project items from the menu. Once you do that to bring up the dialog makes sure that the Store Files: option is set appropriately, in your case to In their current location:
But your question raises an interesting problem. What if I were to delete my library images but not move the referenced files to the trash? Where does that leave me? It leaves me with a problem. I now have masters on a disk somewhere that are not connected to the library in any way. So if I want to delete them or use them somehow later I am out of luck. I have no way to find or isolate them to delete them. I will call these orphaned masters.
How To Recover Orphaned Masters
Here is a folder on my disk that contains referenced files. There are sixteen of them:
I imported all sixteen by reference so that the masters were left in this folder and the Aperture library just contained pointers to them. Then I deleted three images, but did so without letting Aperture move the masters to the trash, so in this folder are three orphaned masters. But which three?
Here is how to find out. First in Aperture I select everything that could be in the folder (project, album, import session, or whatever does it) and then relocate those to some temporary location, choosing to move the masters, not copy:
Now look at what is left in the original folder:
Just the three that Aperture did not know about: the three I had deleted. So now I can delete them for real or move them or do whatever else I want. Once that is complete I can use relocate again to restore my masters to their original position if that is what I want.
Aperture 1.5: Burning Masters to DVD and The Referenced File Manager
2006-10-24
Now that Aperture supports image masters outside of the library (referenced masters), it is possible to store these large files on removable media. Accessing referenced masters on entire drives, such as networked volumes and Firewire drives this is straight forward: just mount the volume.
However, for write-only media such as DVDs and CDs there is a hiccup that will require a trip to Aperture's Referenced File Manager. I will burn a CD with some images to illustrate. Here are my images in the Grid view:
They are currently all inside my library (managed), so to put them on a CD I have to make them referenced with the Relocate Masters. First I put a blank CD in the drive first so it shows up on the desktop and I will be able to specify it as a destination for my masters:
I select all of my images and go to File > Relocate Masters and choose a folder scheme that stores them by date. Aperture does not split a group of files across several disks, so it is necessary to make sure that all my files fit on a single disk --easy in this case because there are only a few.
I press Relocate Masters and my files are been copied to the CD. But they are not really there yet. They are on a disk image that the Finder has created for me that mimics the CD. Here is the Finder view of this disk, all ready to burn:
So the next step is to actually burn the files onto the disk. Once I have chosen a name for this disk (Masters) and the burn is complete I have my finished disk:
Of course that's a terrible name for a disk if I were doing this for real. I'd pick something with a date in it and make sure that the disk name was unique.
But now there is a problem. If I try to do anything with the images on the CD, they are tagged to show that the master cannot be found. Not only does Aperture not have the masters in its library, but it knows that it does not know where the masters are. They are disconnected:
So where are my masters? Why can't Aperture find them? If I Iook on the CD they are definitely there:
So what has happened?
Aperture has recorded the location of the masters as being on the disk image the Finder created for me instead of on the CD I just burned. So the library references point to locations on my hard disk that no longer have the master files. Somehow I have to correct those references.
It is time to fire up the Referenced File Manager and tell Aperture where the masters really are. I'll work with just six images first, by selecting them and control-clicking:
The window that appears has two halves. The upper half shows the status of the files that are referenced. This is the "problem" half:
On the left are the volumes that the images are supposed to reside on. Clicking on a volume will bring up the selected files that were last seen on that volume on the browser in the center. Red means that they cannot be found. And on the right is a pane that details the selected image from the browser:
The lower half of the window is the "solution" half, known as Reconnect Options:
It is a standard column file browser. I can reconnect the image I have selected in the upper half to its master by using this browser and checking that the image matches. I locate the selected master image like this:
After a short delay (and a confusing one too) the Reconnect and Reconnect All buttons become enabled. If I click on Reconnect, then the status on the top half changes, since one out of five is now reconnected:
And the file name in the upper browser turns black from red. If I had to do this one at a time it would be horribly tedious. But I don't. If I select all the remaining images in the upper half, and then navigate to just one of them in the lower half, pressing Reconnect All will do exactly that and all will be reconnected with their master files:
Once repeated for all the remaining images in this project, the thumbnails look like this:
If I eject the CD, the icons change to indicate that the masters are offline, but not disconnected:
Now what if I decide that I don't want the masters on the CD any more? It may be that I have pulled this CD out of storage and want to use the images. It's way too slow to try to work with the masters this way. I simply make the images managed again by using Consolidate Masters. They are copied off the CD into the library and Aperture uses them from that location. I can pop out the CD.
But when I am done with that project I don't want the images in the library any more. I can't delete them or I will lose everything about them. I don't want to burn another CD or DVD with my masters on again -- I already have a perfectly good copy in storage.
If I try to relocate them, then it does not work:
There is a way to do it. First create a temporary folder on your desktop called Temp. Then relocate all the masters into that folder using the same naming scheme that you used on the CD or DVD. That step gets them out of the library. Next, drag that folder to the trash. Yes really. No need to empty the trash.
Select all the images in the grid view and Open up the Referenced File Manager. Select All Volumes from the list of volumes and click the Verify button. Aperture will suddenly realize that it has lost the masters and they will appear on the browser pane. Now on the lower half of the window navigate to the CD or DVD and reconnect all the images.
Once you are happy that everything is OK, empty the trash.
However, for write-only media such as DVDs and CDs there is a hiccup that will require a trip to Aperture's Referenced File Manager. I will burn a CD with some images to illustrate. Here are my images in the Grid view:
They are currently all inside my library (managed), so to put them on a CD I have to make them referenced with the Relocate Masters. First I put a blank CD in the drive first so it shows up on the desktop and I will be able to specify it as a destination for my masters:
I select all of my images and go to File > Relocate Masters and choose a folder scheme that stores them by date. Aperture does not split a group of files across several disks, so it is necessary to make sure that all my files fit on a single disk --easy in this case because there are only a few.
I press Relocate Masters and my files are been copied to the CD. But they are not really there yet. They are on a disk image that the Finder has created for me that mimics the CD. Here is the Finder view of this disk, all ready to burn:
So the next step is to actually burn the files onto the disk. Once I have chosen a name for this disk (Masters) and the burn is complete I have my finished disk:
Of course that's a terrible name for a disk if I were doing this for real. I'd pick something with a date in it and make sure that the disk name was unique.
But now there is a problem. If I try to do anything with the images on the CD, they are tagged to show that the master cannot be found. Not only does Aperture not have the masters in its library, but it knows that it does not know where the masters are. They are disconnected:
So where are my masters? Why can't Aperture find them? If I Iook on the CD they are definitely there:
So what has happened?
Aperture has recorded the location of the masters as being on the disk image the Finder created for me instead of on the CD I just burned. So the library references point to locations on my hard disk that no longer have the master files. Somehow I have to correct those references.
It is time to fire up the Referenced File Manager and tell Aperture where the masters really are. I'll work with just six images first, by selecting them and control-clicking:
The window that appears has two halves. The upper half shows the status of the files that are referenced. This is the "problem" half:
On the left are the volumes that the images are supposed to reside on. Clicking on a volume will bring up the selected files that were last seen on that volume on the browser in the center. Red means that they cannot be found. And on the right is a pane that details the selected image from the browser:
The lower half of the window is the "solution" half, known as Reconnect Options:
It is a standard column file browser. I can reconnect the image I have selected in the upper half to its master by using this browser and checking that the image matches. I locate the selected master image like this:
After a short delay (and a confusing one too) the Reconnect and Reconnect All buttons become enabled. If I click on Reconnect, then the status on the top half changes, since one out of five is now reconnected:
And the file name in the upper browser turns black from red. If I had to do this one at a time it would be horribly tedious. But I don't. If I select all the remaining images in the upper half, and then navigate to just one of them in the lower half, pressing Reconnect All will do exactly that and all will be reconnected with their master files:
Once repeated for all the remaining images in this project, the thumbnails look like this:
If I eject the CD, the icons change to indicate that the masters are offline, but not disconnected:
Now what if I decide that I don't want the masters on the CD any more? It may be that I have pulled this CD out of storage and want to use the images. It's way too slow to try to work with the masters this way. I simply make the images managed again by using Consolidate Masters. They are copied off the CD into the library and Aperture uses them from that location. I can pop out the CD.
But when I am done with that project I don't want the images in the library any more. I can't delete them or I will lose everything about them. I don't want to burn another CD or DVD with my masters on again -- I already have a perfectly good copy in storage.
If I try to relocate them, then it does not work:
There is a way to do it. First create a temporary folder on your desktop called Temp. Then relocate all the masters into that folder using the same naming scheme that you used on the CD or DVD. That step gets them out of the library. Next, drag that folder to the trash. Yes really. No need to empty the trash.
Select all the images in the grid view and Open up the Referenced File Manager. Select All Volumes from the list of volumes and click the Verify button. Aperture will suddenly realize that it has lost the masters and they will appear on the browser pane. Now on the lower half of the window navigate to the CD or DVD and reconnect all the images.
Once you are happy that everything is OK, empty the trash.
Aperture 1.5: Relocate and Consolidate
2006-10-06
Now that Aperture 1.5 is here, it is possible to store the masters outside of the library. For some this is just perfect: their library is huge and big parts of it could be put on Firewire drives, or a server, or on DVDs. Right-clicking on an individual project shows the two new options that take care of this: Relocate Masters for Project and Consolidate Masters For Project:
You don't get these options for albums or blue or brown folders, just for projects and selections of thumbnails (to manage selections of thumbnails you will have to use the menu bar and select the File menu: the control-click menu does not show these options). Masters for the entire library can also be relocated and consolidated.
Relocate means move the masters from where ever they are to somewhere else and reference them.
Consolidate means bring the masters back from where ever they are to the library and manage them.
It's only the masters that are affected by these operations, so there is still data left in the library: thumbnails, adjustments, versions, keywords and other metadata. But the masters are usually huge in comparison, so moving them out makes the most difference when library space is at a premium.
A project can contain a mix of referenced (masters outside the library) and managed (masters inside the library) images through failures during relocation or consolidation, imports to projects, or selections of images being moved in or out of the library. If you edit an image in an external editor, the new image is always put into the library and managed. So if you want it somewhere else, you will have to do that manually.
Lets break up my library by relocating the masters for my Vacation project. Relocate Masters for Project brings me to this dialog:
I've chosen a folder called Media on a volume called Turkey that is on a server. Using the Subfolder Format pop-up I can choose the way that the individual master files from the project are stored in the Media folder. Choosing None gives me this organization:
And the masters (my masters are all JPGs) are stored flat. But I actually had a problem doing this relocate. For some reason Aperture thought there was insufficient space to store some my files on that volume and so left them in the library. Three of the thumbnails don't have the badge that shows they are referenced. You can see one of them below:
That one doesn't have a badge, but how would I know if I had thousands of images which ones were managed or not? It is easy because I can filter on the file status:
By picking File Status and then selecting Managed, I can see the three that were not relocated:
The other options here, Online and Offline refer to the volumes that are storing the referenced masters. If I had a closet full of Firewire drives and only three of them plugged in to my machine I could easily see which images were on those drives and which were in the closet by using those filter selections.
I can still work with my referenced images exactly the same way I can work with my managed images. I can create versions, do adjustments, crop, rotate. It might be a little slower in my case because my master is on an aging G3 iMac at the end of a 100Mb network, but otherwise the experience will be the same.
I will consolidate the files that were relocated and so make them managed again. By selecting the project and Consolidate Masters For Project I get this dialog:
The warning here is because I could have other Aperture libraries that also reference the same masters. If I move them, they will not be usable from those other libraries. I click Continue, move them all back, and the badges all disappear. My Media folder is empty again.
Relocating again and selecting Image Year/Month/Day for the subfolder gets me this organization inside the Media folder on my server:
This corresponds to the date the images were shot. I can also use Relocate Masters For Project to relocate the masters at any time without consolidating them. They are simply moved from one place to another. If I now choose Project as the subfolder name, then they are all moved out of the year/month/day folders and into a folder called Vacation without coming back into the library over the network. So this is very fast on a remote server:
It's very flexible. Not only can I move the masters about willy-nilly, but I can also make my own structure out of building blocks by selecting Edit... from the subfolder pop-up. Here I have redefined the preset Project Name:
That results in the following organization:
There is more. Because Aperture keeps a lot of information about the referenced files in the library, the referencing is very robust. It will even find them if you give them new names or move them somewhere else on the same volume. If the project contains master files that have the same name Aperture renames one of them like this:
Note that relocate and consolidate operating on a project don't take account of any filtering: they work on all the masters in the project at once. Filtering can be used though: just filter, select the images, and use the File menu to relocate or consolidate.
You don't get these options for albums or blue or brown folders, just for projects and selections of thumbnails (to manage selections of thumbnails you will have to use the menu bar and select the File menu: the control-click menu does not show these options). Masters for the entire library can also be relocated and consolidated.
Relocate means move the masters from where ever they are to somewhere else and reference them.
Consolidate means bring the masters back from where ever they are to the library and manage them.
It's only the masters that are affected by these operations, so there is still data left in the library: thumbnails, adjustments, versions, keywords and other metadata. But the masters are usually huge in comparison, so moving them out makes the most difference when library space is at a premium.
A project can contain a mix of referenced (masters outside the library) and managed (masters inside the library) images through failures during relocation or consolidation, imports to projects, or selections of images being moved in or out of the library. If you edit an image in an external editor, the new image is always put into the library and managed. So if you want it somewhere else, you will have to do that manually.
Lets break up my library by relocating the masters for my Vacation project. Relocate Masters for Project brings me to this dialog:
I've chosen a folder called Media on a volume called Turkey that is on a server. Using the Subfolder Format pop-up I can choose the way that the individual master files from the project are stored in the Media folder. Choosing None gives me this organization:
And the masters (my masters are all JPGs) are stored flat. But I actually had a problem doing this relocate. For some reason Aperture thought there was insufficient space to store some my files on that volume and so left them in the library. Three of the thumbnails don't have the badge that shows they are referenced. You can see one of them below:
That one doesn't have a badge, but how would I know if I had thousands of images which ones were managed or not? It is easy because I can filter on the file status:
By picking File Status and then selecting Managed, I can see the three that were not relocated:
The other options here, Online and Offline refer to the volumes that are storing the referenced masters. If I had a closet full of Firewire drives and only three of them plugged in to my machine I could easily see which images were on those drives and which were in the closet by using those filter selections.
I can still work with my referenced images exactly the same way I can work with my managed images. I can create versions, do adjustments, crop, rotate. It might be a little slower in my case because my master is on an aging G3 iMac at the end of a 100Mb network, but otherwise the experience will be the same.
I will consolidate the files that were relocated and so make them managed again. By selecting the project and Consolidate Masters For Project I get this dialog:
The warning here is because I could have other Aperture libraries that also reference the same masters. If I move them, they will not be usable from those other libraries. I click Continue, move them all back, and the badges all disappear. My Media folder is empty again.
Relocating again and selecting Image Year/Month/Day for the subfolder gets me this organization inside the Media folder on my server:
This corresponds to the date the images were shot. I can also use Relocate Masters For Project to relocate the masters at any time without consolidating them. They are simply moved from one place to another. If I now choose Project as the subfolder name, then they are all moved out of the year/month/day folders and into a folder called Vacation without coming back into the library over the network. So this is very fast on a remote server:
It's very flexible. Not only can I move the masters about willy-nilly, but I can also make my own structure out of building blocks by selecting Edit... from the subfolder pop-up. Here I have redefined the preset Project Name:
That results in the following organization:
There is more. Because Aperture keeps a lot of information about the referenced files in the library, the referencing is very robust. It will even find them if you give them new names or move them somewhere else on the same volume. If the project contains master files that have the same name Aperture renames one of them like this:
Note that relocate and consolidate operating on a project don't take account of any filtering: they work on all the masters in the project at once. Filtering can be used though: just filter, select the images, and use the File menu to relocate or consolidate.
The Bagelturf site welcomes Donations of any size