RAW

Introduction

Raw files have some unique capabilities for digital imaging. They provide the highest quality as a capture format. They capture the data the camera sensor can provide at the highest bit depth and the highest dynamic range. As such, they allow for maximum adjustment in post-production.

The high quality and flexibility that raw photography provides come at a price. Raw files are larger than JPEG files, and they typically require more attention in post-production processing.

There are a number of important considerations in setting up a raw file workflow and an archive based on raw files, due to the proprietary nature of these files. There are some difficulties presented by the variations in raw file structure. And there are some important challenges to address in the preservation of the adjustments you make to raw files.

Anatomy of a raw file

The vast majority of camera raw files have a number of traits in common. These include the basic file format type and the structure of the data inside the file. Within these broad similarities, there are a lot of smaller variations which can present some difficulties.

TIFF/EP

Most raw files are a variation of the TIFF/EP standard that was developed for electronic photography (as digital photography used to be known.) This is a particular flavor of TIFF that allows for the storage of mosaiced image data and a preview. It also outlines some tags in the file that can help imaging software know how to read out the image data and render it.

Bayer pattern sensor data

A large majority of still cameras that shoot raw images capture and store the image as Bayer pattern data, which is called a mosaiced image. In these cameras, each sensor element sees and saves information about one color only, typically red, green or blue. When the raw file is opened with the proper software, the missing two values for each pixel are interpolated from surrounding pixels. This results in a full RGB color value for each pixel.

Each digital camera sensor records the mosaiced data in a slightly different way. Differences include the basic color response of the sensor, as well as variations in the color mosaicing.

Other sensor types

There is at least one other method for encoding raw image data, and that is what is done by the Foveon chip, currently used in Sigma cameras. This sensor design uses multiple layers to record a red, green and blue value for each pixel, rather than the Bayer pattern method.

A JPEG preview

All modern cameras that shoot raw create a JPEG version of the image during the shooting, and place it inside the raw file itself. This is the image you see on the back of the camera. The JPEG is created using the settings that have been chosen in the camera. The camera will use any contrast, color balance or specialized “looks” when it creates the JPEG preview, but in most cases this has no effect at all on the raw information: it’s just one interpretation of the raw image data.

Note that when you see a histogram on the back of your camera, this is being created from the embedded JPEG, not the raw image data. This means that the histogram will not actually tell you whether you have really clipped highlights or crushed the blacks in the raw image.

Metadata

There is also a bunch of metadata in a raw file. At minimum, raw files include the technical metadata that indicates how a raw file was encoded, so that software knows how to decode it. There will also be EXIF shooting data that indicates date created, shutter speed, aperture, lens, metering mode and more.

Many cameras allow you to assign some kind of creator information: at minimum, you can usually tag the file with the photographer’s name. Some newer cameras, such as the Nikon D4, allow a broad range of IPTC metadata to be assigned in-camera. In most cases, however, descriptive metadata, such as keywords, location and more are typically added in post-production.

Private Maker Notes

Raw files will often contain a type of metadata called Private Maker Notes. This includes information that the manufacturer may consider to be proprietary for a host of reasons. Sometimes the information may be considered proprietary technology, or sometimes it may simply be something the manufacturer doesn’t want to disclose to the consumer.

Proprietary raw considerations

Raw image technology is a fast-moving area, and a major focus of development for camera companies. This has led to a bit of a mess with respect to raw file standardization. As camera companies implement new sensors in cameras, or develop new ways to render color or reduce noise, the structure of the raw file can change. Let’s take a look at some of the characteristics and ramifications of proprietary raw formats.

Proprietary raw files are undocumented

The changes outlined above often happen without proper documentation for third-party applications. Camera companies can also change the way they structure their files, moving the elements of the file around or changing the way they encode different bits within the file. This situation can make it dangerous for any third-party application to make any changes to a raw file, since those changes may inadvertently do some damage to the data in the file.

Raw files are typically “read only”

The undocumented nature of most raw files leads to a caution that is exercised by most third-party raw file software. In an effort to avoid corrupting undocumented files, most applications will treat them as “read only”. This means that the file can be opened, but the changes you make to the file are not saved back to the file itself.

From the standpoint of the raw image data, the read-only status is perfectly fine. The mosaiced image data does not lend itself to manipulation by third parties anyway. But the other types of data in the file – the metadata and the embedded JPEG preview – are a different story. There are plenty of reasons one might want to alter these.

You might want to alter the embedded JPEG preview, for instance, to reflect the adjustments made by the user when editing the file. This would allow the end-result image to travel along with the file. And if you create descriptive metadata such as a keyword, or administrative metadata such as a copyright notice, then it may be highly desirable to attach these terms to the file as well. In many cases, it is simply not possible to make these changes to proprietary files with third-party software.

Software from the manufacturer of the camera may be able to alter both the embedded JPEG preview, as well as the embedded metadata. Nikon Capture, for instance, can do both of these. This makes a favorable argument for using camera-native applications, but many photographers find the workflow these provide to be unsuitable.

Software must be updated for new raw versions

Each new camera model typically creates a raw file that is different from its predecessors, even though it may have the same file extension. In order to open these new variants, your software must be updated to “understand” the new file structure. In the first years of raw photography, this waiting period could be months long, creating a frustrating situation for the photographer.

Software manufacturers have gotten much better about working with camera companies to support new raw variants as the cameras are released. But a structural problem remains. Typically, software manufacturers will stop adding new camera support once a new version of their software is released. Anyone purchasing a new camera after this time is generally required to buy the new version of the software in order to open the new files.

If budget allows, dpBestflow® suggest staying current on your imaging software. The improvements with each new generation of software are typically well-worth the price of upgrade for the professional photographer.

But we also recognize that software upgrade can get expensive, particularly for those who must upgrade several machines, or for those who must upgrade entire suites of software simultaneously. Additionally, it’s not uncommon for one upgrade to trigger the need for another one. So a software upgrade may force an upgrade in your OS or your computer hardware. This can add a lot of expense, as well as a lot of IT overhead to the price of a camera.

We therefore urge photographers to be informed about software support for raw files prior to the purchase of a camera, so that they understand the upgrade ramifications for their entire systems.

Raw file “color space”

Most digital camera that shoot raw offer options for choosing the color space, but this is misleading. Raw images do have their own native color characterization, but this is a different than a standard ICC profile such as sRGB or Adobe RGB. The color profile of the camera itself is known as “scene-referred” color, and describes the relationship between the real world and the file created by the camera.

When you select Adobe(1998) or sRGB in your camera’s menu, you are choosing an output for the scene-referred color. These settings tell the camera how to convert the raw color data into a regular RGB file, whether it is a JPEG that the camera creates or whether it is the JPEG preview inside the raw file.

This setting you make in camera creates the JPEG and it is also written to the file as an instruction for opening the raw file in the manufacturer’s software. There is nothing, however, that requires you to use that color space when opening the file, and most third-party applications actually ignore it.

READ MORE ABOUT SCENE REFERRED COLOR IN KARL LANG”S WHITE PAPER

READ MORE ABOUT COLOR SPACE IN THE COLOR SECTION

Raw conversion software

Since raw image data is read-only, raw files must typically be run through some kind of conversion software prior to any usage. This software demosaics the file, typically producing a full-color RGB file. It will also apply a white balance to the file, which is a necessary part of the process. The software will also make a number of other color and tonal transformations to the image data as it creates the full-color version.

It’s important to note that each raw file processing application will create a different look to the image. This is similar to the way different films produced different results. The look created by a piece of software is an important part of the field of competition between imaging applications.

There are important implications for the different renderings produced by each software package. If you take time to adjust an image in a program, you will likely want to save that work. You can do this by creating a new file – a JPEG or TIFF with the changes “baked in” to the new file. But this results in the creation of lots of new files, which can be pretty confusing to manage. Alternatively, you can save a record of the settings you’ve used, and make sure to use them the next time you want to view or render the file.

Sidecar files

In most cases, the best way to save the settings you’ve used to adjust your proprietary raw file is by means of a sidecar file. This is typically a text file that lives in the folder next to the raw file, and it is used by the imaging application to remember and reapply the settings. Adobe XMP files are an example of such files. Sometimes the sidecar files will be made on a folder basis, with one text file representing all adjustments for each image in the folder.

Long-term accessibility

In the relatively short history of raw photography, there have been some formats that have become obsolete, and therefore largely inaccessible. The early Kodak cameras used a patented method to record color, and the software to read it has not been updated to work with modern computers. This has put the images at risk of inaccessibility.

For modern DSLR cameras (generally those produced after the year 2000), the risk of inaccessibility is significantly less. Even if commercial software packages eventually drop support for very old cameras, open source software will probably continue to provide access. The program dcraw, written by Dave Coffin, includes support for a very large number of cameras. Because it is open source, it is unlikely to simply go away. The codebase can be picked up and modified in the future by any number of independent developers.

DNG as a raw file container

If the above issues are of concern, then you might want to consider using the digital negative or DNG file format to store your raw image data. DNG was developed by Adobe to address the issues presented by multiple undocumented raw image file types. It offers a documented structure and replaceable JPEG previews as well as user-created metadata, including the rendering settings. The next page outlines the structure, uses and creation of DNG files as a standardize raw file container.

READ MORE ABOUT DNG IN THIS SECTION