File Lifecycle: Understanding Workflow

Introduction

Lifecycle is a concept that helps you understand how to take care of your digital data. As the name implies, lifecycle describes how old and how active your data is. This understanding lets you design systems and processes that keep your images well protected. The lifecycle concept is also important because it helps you understand your own workflow. You can use this knowledge to increase your efficiency at the same time as you increase security and reduce cost.

The lifecycle concept has been a standard tool in the information technology world for decades. In this section of the dpBestflow® website, we’ll take those concepts and make them understandable to the average digital photographer.

Lifecycle and Workflow

At its core, lifecycle is tied directly to your workflow. There is a natural progression of your images from the camera, through the computer, through output, and into a permanent home. The decisions you make about how to store the images, what to do with them, and what software to use change as the images “grow up”.

There are certain steps that are necessary when the images are first captured, and steps that should be done when they are first downloaded to a computer. Most people’s images then go through a works-in-progress stage, where they are sorted and then adjusted. And once the images are ready, they can be published in some way - sent to a friend or client, printed, included in a movie or multimedia production, or posted on a website. The final storage for your images should be a “permanent” home in the archive - a safe and orderly repository for the digital data.

Understanding lifecycle is key to understanding workflow. It lets you know why a certain task is the best thing to do right now, and it lets you know when a step is coming out of order, or when the backup systems you are using are not right.

Lifecycle helps you to create a workflow that has a beginning, middle and end. It lets you create a path your images can travel and lets you understand exactly what you are doing along the way.

There are two areas where understanding lifecycle will help you most. The first is helping you understand the most efficient way to construct a workflow. For instance, the concept of “Ingestion” helps you to understand the set of steps that should be taken the first time you download your images. And it also helps you understand the work order for this sub-group of steps.

Lifecycle and storage

Breaking your workflow into lifecycle phases lets you create storage systems that are the most secure - at the lowest cost - for any particular phase. Your undelivered works-in-progress, for instance, need a different kind of backup protection than do images that were shot, delivered and archived years ago.

Working files typically require the fastest storage in your arsenal, which will also be the most expensive. And protecting these files with off-site storage can be tricky, since media files are generally too large to be easily sent over the internet in bulk. Additionally, works-in-progress often need some kind of versioned backups. If something undesired happens to the original file, you may want to roll back to a previous version.

Archived files, by contrast, can typically be stored on less expensive media. Often, they can be stored on energy-efficient, low-performance drives, since you won’t need to access the files that often. And it’s much easier to protect archived files with inexpensive off-site backup by copying the images to drives and carrying the drives offsite periodically. You can also make use of inexpensive optical media for a disaster recover backup copy.

We use the lifecycle concept to discuss storage throughout the entire website.

The five phases of image lifecycle

We break Lifecycle into five basic phases. Here they are:

• Capture
• Ingest
• Working
• Publish
• Archive

Capture

Capture is the first phase of your workflow. It encompasses all the work you do and the decisions you make when you shoot the pictures. These include camera selection, file format, white balance, and how much you shoot. The Capture phase may also include multi-image capture, such as HDR, time-lapse, or panoramic shooting.

While Capture is the most easily understood phase of image lifecycle, it’s still important to examine in the lifecycle context. The work you do to prepare for a good capture (particularly for JPEG shooters) can be a critical factor in making good photos.

Ingestion

Ingestion refers to the set of steps you take when you first transfer the images from the camera to the computer, and many of these steps can probably be automated by the software you are already using. In many cases, the ingestion phase is where photographers can add productivity most easily.

The Ingestion stage is an important opportunity to use computers for what they do best: performing repetitive tasks quickly and reliably without a lot of user involvement. These steps can include downloading, renaming, adding bulk metadata, adding rendering settings, backing up, conversion to DNG (if desired), and loading into an image editor. With a little set-up work on the front end, all of this may be a few clicks away, even for a shoot with a large number of photos.

The Ingestion phase is also a delicate time in the life of an image file. There is generally only one copy, so loss is a real possibility. We’ll suggest some image handling methods in the Ingestion phase that can add lots of protection.

Ingestion for video

The lifecycle concept works nearly identically for both stills and video. One difference is that some video ingestion is best accomplished by creating a disk image of the original media. Another difference can occur at the tail end of the Ingestion process. There are a series of processes that most video workflows require that are a bit different from photo workflows. In many cases there is a set of steps that can’t be automated, but should be considered to be part of the ingestion process.

These steps can include audio track replacement, logging, transcription, transcoding and checksumming.

Working

We use the term “Working files” to describe image files that are works-in-progress. These are files that have gone through the Ingestion process, but are not yet put away into their permanent home.

Examples of working files

There are a number of common examples of Working files.

• This could include files that have been downloaded on location, and are on their way back to the studio. 
• Or perhaps these could be files that have been Ingested and are waiting to be culled before being Archived. 
• If you use a DNG workflow, but don’t do the conversion as part of the Ingestion process, these working files may be the proprietary raw files awaiting conversion to DNG. 
• A master TIFF file that has not been finalized is another example of a Working file. 
• Some files are Working files indefinitely. If you use one big Lightroom or Aperture catalog to manage your whole photo collection, that catalog is a work-in-progress forever. 
• Video project files are Working files until the video has been delivered and the project has been Archived. 

All of these above examples share some characteristics. They occupy a middle ground between Ingestion and Archive. The automated processes are done, but the files are not yet ready to Archive. In this phase, all the files need some work done to them that must be performed by a human. And in each of these cases, the file itself is either being changed or has not yet reached the Archive.

Publish

From a lifecycle perspective, the term “Publish” refers to any product or output that you create that is sent to someone else. This might be delivery files of your images, a finished video or multimedia piece, prints, files uploaded to the internet, or some other creative work. The essential characteristic is that the image is sent outside of your own system.

Over time, your most valuable images and videos will probably be Published in multiple ways. Perhaps a client uses the images in some kind of advertising or promotion, and you also use the images in your portfolio.

From a lifecycle perspective, the Publish phase is important for several reasons:

• It’s the ultimate payoff for the creative work.
• The work is moving outside your control.
• You may wish to preserve an exact copy of the published file for a business record.
• There are copyright registration implications, particularly for the US market.

Archive

We use the term “Archive” for any files that have been put in their “permanent” homes and safely backed up in a secure, long-term way. In most cases, Archived files should be expected to never change. There are a few exceptions to this: you might want to update the metadata of Archived files, or you might need to migrate the files to a new format eventually. But for the most part, Archive files simply don’t change.

When you use the Archive lifecycle concept to construct your image collection and your workflow, you gain significant advantages. You make a place that your images can actually flow to. And you separate out the unchanging files from the ones that are in flux.

Professional IT managers have long known that it’s much cheaper and easier (and more secure) to store files that don’t change, than files that do. You can create simpler storage systems, and you can validate the integrity of the files with total certainty.

The Archive process includes transferring the file(s) to a permanent home, creating checksums, secure backup, and periodic validation.

File lifecycle vs. image/footage lifecycle

At this point, it’s important to make a distinction between the way we handle a file and the way we think about the underlying image or footage. A photo or video file may be ready to archive even though the finished image or finished video has not even been delivered yet. As a matter of fact, the original files may be ready to archive even before the raw image conversion has been finalized.

Wait, what?

At first that may not seem to make sense. How can a file be ready to Archive even before it’s finished? The important thing to keep in mind is the nature of Parametric Image Editing and Non-Linear Video Editors. When you adjust a raw image with Lightroom, Aperture or Capture One, you never alter the original file, you simply reinterpret it. (The same happens with Final Cut or PremierePro). Each of those programs treats the source image/footage as a read-only file. This means that you can Archive the file, and you can continue to open and work with it in PIEware. When you need to Publish the file in some way, you simply make a new copy of the adjusted image or video, leaving the original file untouched.

Real advantages

There are some huge advantages when you separate how you treat the file from how you think about the image or footage. Remember that the cheapest and safest place to store your data is in the Archive. You reduce your risk and your cost by moving files rapidly into the archive.