Archive File Formats - Video

Introduction

Due to the large file sizes and the use of numerous codecs, archiving video is a complex topic. The dependency on specific codecs creates a potential for footage becoming inaccessible. And project files carry an inherent risk of obsolescence, as NLE software and the formats it understands are constantly moving targets.

Most users choose to archive their original camera media as well as digital master files of their completed programs after editing. recommends that you archive both your source footage and multiple self-contained video files of the completed project. It is also a good idea to store copies of the project file that contains the instruction set on how the source files were assembled.

For client work, archive decisions often come down to a matter of cost. Some clients are willing to pay for archives while others are not. Additionally, many clients assume that the production company has archived everything, so be sure to clarify assumptions. The use of longterm media formats such as LTO tape and optical media has made this process easier, but it is still neither easy nor cheap.

Create a digital master of the finished production

When a project is done, you’ll often make several digital files for delivery. Typically these files are heavily compressed as they are intended for playback on portable media devices or the Internet. While you will probably want to archive these files, you’ll also want to save a digital master file with the least compression that is practical.

A QuickTime movie using a low-compression scheme is a good format choice for your digital master file. Popular codec choices include the Apple ProRes 422 (HQ), Avid DNXHD, Cineform, or Animation codecs. These files may be very large, but they ensure a high-quality digital file that can be used to make additional digital derivatives.

Archiving a master to tape

It’s very common for people who make or commission video to render and archive a master copy of the production to some form of tape. This is frequently done even though there may be multiple copies of the digital files on various hard drives or other media. A finished master copy written to tape provides a backup version that may not be subject to the same codec uncertainties that other digital copies have.

If you don’t own a system capable of creating a digital tape, you could send a master digital copy to a production house and subcontract the process. You could also write the file out to Blu-ray disc to create a high-quality version that is not so dependent on installed codecs.

Camera original files

In addition to the finished master version, it will generally be advisable to archive the camera original files. There are a few key considerations when creating these archives.

• Maintaining file structure: It is a best practice to maintain an exact copy of the file structure of the original source media files when copying to a backup device. The additional metadata and folder structures are often needed to assist editing software in properly importing and interpreting the camera data, particularly for AVCHD.
• Redundancy: If you have only one copy, it is not backed up. Be sure that the media exists in at least two locations and uses at least two formats of backup. While hard drives are cheap, they are not a permanent solution. Many turn to optical media such as Blu-ray discs or tape-based archives like LTO, DLT, and AIT.
• Use software: Many use software tools to ensure a complete copy of the disk image. Tools like Final Cut Pro X have a camera archive feature that supports certain formats (not DSLR cameras, however). Another popular choice is ShotPut Pro (www.imagineproducts.com/ShotPut.html) which can automate copying of tapeless media to up to three separate locations. It also has a great set of tools for verifying copies and, creating additional backups to optical formats.

Project files

The project file created by your nonlinear editing software truly is intellectual property. If you ever need to make a change to the video after the creation of the digital master, it will probably be easiest to do so from the original project file. Of course, this is only so if the project file can be opened, and all the clip files can be easily reattached. You will want to take great care to ensure that the project file is archived to multiple locations and multiple media types.

At the end of a project, it is also a good idea to export additional versions of a project file. Over time, manufacturers often evolve their project file formats. Some even drop support all together (such as Apple did in its initial transition from Final Cut Pro 7 to version X). Most editing tools can export an EDL (edit decision list) and XML (eXtensible Markup Language) file. Storing a copy of each with the archived project is a good idea.

Verify your project archive restoration

As you come up with ways to archive your project and the associated media, it’s important that you double-check your ability to restore the project at a later date. Project files expect the source media to be in a particular place, and their ability to reset those connections to new locations varies by application. In some cases this will be nearly automatic, and in others may not even be possible.

Before you archive a lot of project and media files, you’ll want to make a test. Save a test project out to your proposed archive location, and then go through the steps to open it up again. Ideally this restoration test would take place on a different computer or with the project sitting on a different drive than the original location.

Removing unused media from the project file

When you archive a completed project and the associated media, you will almost certainly have a bunch of stuff that was not used in the finished product. This may include a lot of unused source footage, and it may also include temporary render files. In some cases, it will make sense to include all of this with the archived project, and in some cases you may want to remove some of the files to save space.

There are a number of factors to balance here. These include the total size of the data, the value of the project, the prospective need to repurpose the project or footage, the need to revisit the project, and more.

For a high-value project that only takes up a couple hundred gigabytes, it probably makes sense to archive everything. If the project has limited audience and a ton of transcoded footage, then it might make sense to trim off the unused transcoded footage when you send the project to archive.

In the end, you’ll have to make the decision of what to cut according to your own valuation of the project in the context of the archiving costs.

Imported/transcoded files

If your editing system transcodes or re-wraps the files into a new format, it is important to back that media up as well. Editing tools like Final Cut Pro X have a complex event and project structure that must be fully preserved with all media in exactly the same place. Other tools have greater flexibility, but the easiest way to open and revise a project is with the project completely archived.

A video editing project may also contain items like music or graphics. These should also be backed up as part of the project. Most nonlinear editing tools offer a project manager capability. These features can remove or trim unused footage as well as consolidate intermediate files like renders and caches. They can also target a specific drive or directory for backup.

Format obsolescence

A major challenge with regard to the preservation of digital video files is the long-term readability of file formats. This is especially true since there are so many competing formats and codecs in the video editing market.

The sheer number of compressed and raw formats leads to many potential problems. We have seen rapid evolution with video codecs as well as camera acquisition formats. Even if workflows exist to recover discontinued formats, it often means re-compressing or reprocessing the assets which can lead to a loss in quality.

Adobe is currently trying to establish the Cinema DNG standard (http://labs.adobe.com/technologies/cinemadng/). This effort is very similar to the DNG efforts they implemented for digital photography. They have yet to experience a wide-spread adoption by the industry (to the same level as MXF has for acquisition formats). However the effort has merit and bears close observation as it could standardize the industry and remove several barriers in the process.

Preserving a legacy system

Okay, we hate to even bring this up, but it may be necessary for some people. In some cases, it may be necessary to actually preserve legacy software or even an older computer in order to retain access to project files. If you make a big change of computer platform or NLE software, your older projects may be totally locked out of the new system. For projects with long-term value, this might present a real problem.

The biggest challenge here is that the old computer system may stop working at some point, and you might not notice until you have an actual need for it.

There are any number of issues which could make that old computer sitting in the storage area non-functional.

Software often relies on the OS for lots of its capability. When you upgrade your OS, it may make the software non-functional.

Any video archive that relies on legacy software or hardware presents ahigh inherent risk of invisible failure.