Raw File Formats

Proprietary raw

Raw file format (not an acronym) is raw data from a digital camera sensor that has not been demosaiced, meaning that a raw file is only composed of either a red, green, or blue value at each pixel location.

Rather than having the camera demosaic the raw data to create a rendered file, saving the raw data allows the photographer to control the demosaicing process with special software that offers control over white balance, color saturation, exposure (to a degree), bit depth, and color space, noise reduction, and even lens correction and other "parameters". This is why these types of software are often referred to as Parametric Image Editing software.

Although raw files are much larger than JPEG files, the extra measure of control and potential quality is a worthwhile trade-off for most professional photographers. Most raw formats are proprietary and undocumented. Proprietary raw formats are problematic because they must be reverse engineered by third party software vendors before they can be opened in any software other than the manufacturer's software.

This feature along with lack of documentation makes it quite possible that many of these formats will become unreadable in standard software at some point in the future. Currently, there are over 200 varieties of proprietary raw file formats. Until camera manufacturers agree on a standardized raw format, this number increases with every new camera release.

Figure 1 It is difficult to provide an accurate description of the content of proprietary raw file formats. Although most are loosely based on the TIFF/EP standard, the organization of the data varies and is undocumented. Some camera makers including Nikon, Canon, and Sony encrypt portions of the file.

Introduction to DNG

There are some “Urban Myths” related to converting to DNG. The primary one being that you leave something behind when you convert to DNG. The answer is no. Everything is intact plus a few additional things are included; for example - camera model, sensor type, camera serial number, spectral characteristics of the sensor and lens information.

Figure 2 DNG, as opposed to proprietary raw formats, is openly documented. This makes it easier for third party software vendors to support since there is no mystery as to where the various types of data are located. In addition, DNG contains a data verification function by means of an embedded MD5 hash; it also has the ability to contain a full size JPEG preview with an accurate rendition of the image adjustments, and can even contain the original proprietary raw file as an option.

Things you should know about DNG

• DNG is a container.
• DNG holds all that data that proprietary raw files hold.
• DNG also holds additional information that allows any third party software to understand where the file came from and how it needs to be processed based on the PIE work done.
• DNG can hold a JPEG preview file that accurately reflects the PIE adjustments made to the raw image data. This JPEG preview is controlled by a menu preference in the DNG converter. The ability for a DNG file to hold an accurate JPEG preview (showing updated PIE adjustments) is a unique advantage. Proprietary raw files can only hold the original camera generated JPEG preview—no matter what PIE adjustments are subsequently made to the image. Having an accurate preview built into the DNG file is an important feature if you use cataloging software and wish to view your images as you have adjusted them. These ready made JPEG files can be quickly copied out with certain applications such as PhotoMechanic and Expressions Media Pro. This saves time rather than processing out raw files to JPEG through PIEware.
• The JPEG preview, when it exists, can be updated. When and how that is done has some significant workflow implications. We will discuss those shortly when we talk about points of conversion.
• Of the three JPEG preview options, the full sized preview option is the best for maximum workflow usefulness. When you select this option, a medium size preview is also created which can allow faster preview rendering in browsers or cataloging applications.

Pros for converting to a DNG workflow

• DNG is a standard openly documented format.
• DNG is suitable for archiving image data since it is an open source format.
• DNG contains information about the camera that made the file and how the file needs to be processed (interpreted). This means that if you use a newly released camera with newly released proprietary raw format files that are converted to DNG, these DNGs can be processed in any older software that supports DNG processing.
• DNG can contain a JPEG preview that accurately reflects PIE adjustments.
• The DNG conversion process includes image data verification that tells you if your data is good or not.
• DNG contains a verification hash that allows for continuing data verification

Cons for converting to a DNG workflow

• While this is becoming less true, converting files to DNG tends to lead to an Adobe centric workflow. For example, some third party programs do not recognize a DNG since they do not properly parse the file data.
• Current lack of complete support by the industry as a whole.
• The ability to create and update accurate Jpeg previews can slow PIEware. Consequently, it is best done as a batch process.

Early DNG binding: Convert on ingestion

Some choose to convert during ingestion (which we call early DNG binding). This requires awareness of a few issues and making decisions about how to handle these details during the rest of the workflow.

If you use Bridge/Camera Raw, PIE work is automatically written to the DNG as XMP data. This makes your PIEwork portable since it is now contained within the DNG file. When the DNG is opened on another computer either in Bridge or Lightroom, the PIEwork will accompany the file. Bridge/Camera Raw gives you the option to update the JPEG previews immediately as you do your PIE work (Camera Raw Preferences: DNG File Handling). Although this insures that the JPEG previews are rebuilt to reflect your PIE work, this can slow your work progress unless you have a very fast workstation. We recommend that you don’t turn on this option; instead wait until you have finished adjusting your files. JPEG previews can be updated within Bridge or with the stand-alone DNG converter. We recommend that you run the newly adjusted DNG files back through the DNG converter since it can run in the background. If you open all the files in Camera Raw through Bridge to run the conversion, your workstation will be tied up until that task is finished.

An important item to be aware of is that you will need to save the new DNG files in a new folder, otherwise you will have a second copy of each DNG interleaved with the original files (i.e. the converter will add a _1 just before the file extension). This is true whether you use Bridge/Camera Raw, or the DNG converter.

DNG and Lightroom

Lightroom gives you the choice (Lightroom: catalog settings) to automatically write PIE XMP to the DNG files, or manually updating the XMP data (Metadata: Save Metadata to File). We recommend using this command to update the XMP at the end of your PIE session. This ensures that the image adjustments (as well as any metadata additions) are written into the DNG files. A person who only works on one computer may not see the need to write XMP to the DNG, BUT, storing all your PIE work in a catalog will limit your options and makes the file less portable. For example, if working in Lightroom and the XMP is not written to the file, the PIE work done in Lightroom will not appear in the file. This means that if the file is viewed in Bridge or in Lightroom on another computer, your work will not show up.

Figure 3 Lightroom has another option to update both the DNG preview (meaning the JPEG preview), and Metadata (meaning the XMP data) with one command found under the Metadata menu (Metadata: Update DNG Preview & Metadata).

As with converting to DNG within Bridge/Camera Raw, converting to DNG within Lightroom will tie up your workstation until the task is complete. However, Lightroom conveniently rewrites the DNG files, so you do not have to create a new folder and delete the old folder, as you need to do when using the stand-alone converter or Bridge/Camera Raw. This is the best option, in our opinion, because the PIE work will show up in Bridge and Lightroom on any other computer. Plus, since the JPEG preview has also been updated, your image adjustments will show up in other browsers and cataloging applications that don’t parse raw files.

Late DNG binding: Convert after optimization

workflow preference is to convert to DNG after we have gone deeper into the workflow. We edit, rate, rename, and do a good first round (or two) of PIE work to where the image files are at least 90% optimized. More work can always be done, but at this stage, we should feel good enough about how the images look that we can show them. This is the best workflow point to convert to DNG. Here are the options:

Figure 4 Adobe DNG converter

Save Options

• Save Folder: It’s best to save in a new folder. We use the same folder name but indicate that these are DNG files. For example 09016 TomBloom cr2 becomes 09028 TomBloom DNG. If there are nested folders, we check the option to preserve subfolders. This is a convenient feature only available in the stand-alone DNG converter.
• Select name for converted images: We stick with the default which is “Document Name”. This preserves our original file name and adds the DNG extension. For example; RNA_09028_0123.cr2 becomes RNA_09028_0123.DNG.

Preferences

Figure 5 Adobe DNG converter preferences

JPEG Preview

We always choose the full size JPEG preview. This choice will save a medium size preview as well. The full size preview will reflect any sharpening applied in Camera Raw or Lightroom (the medium size JPEG preview is not sharpened). “Full size” means the JPEG will have the file dimensions set in Camera Raw or Lightroom. For instance, if you set Camera Raw to enlarge or reduce the native file, the JPEG preview will reflect that size choice as well as the ppi resolution setting.

Compression

It’s lossless; we recommend that you keep this default setting.

Image Conversion Method

We recommend that you choose Preserve Raw Image. The raw image is undemosaiced image data. Once the image data has been processed into mosaiced data, it becomes a linear image, which is another way of saying that it has become similar to a standard raster image like TIFF.

There is some confusion about what a linear DNG is, since most people assume that DNG is always a raw file. However the Adobe DNG specification allows for a demoisaiced (linear) DNG that is essentially just like a TIFF file. Since it is demosaiced, it is three times larger than an undemosaiced (raw) DNG, and it has the PIE work baked in. Creation of a linear DNG is more commonly done by non-Adobe PIEware, such as DxO, and Capture One PRO as a means of preserving their rendering of raw files.

Embed Original Raw File

Choosing this preference stores the original proprietary raw file inside the DNG container. This is advantageous since you have preserved the proprietary raw file. You now have the option to extract it and process it in the manufacturer’s software, or any other software that doesn’t open DNG files. Another advantage is that the proprietary raw file is verified in the conversion process. The hash that allows for continuing verification of the DNG now also extends this extra security to the proprietary image data. The disadvantage is that it essentially doubles the size of the DNG. If you shoot a lot of files, you may find storing all that data to be a challenge, and of course moving files around will take twice as long.

Our conclusions about the DNG workflow

The DNG files have proven to be significantly more useful than the proprietary raw files in our workflow. We like the fact that:

• The metadata is safely stored in the file, not hanging outside the image file in a sidecar file.
• We have a full size optimized JPEG file available in the DNG container. This provides us with an accurate view of the image in our DAM applications and gives us the option to quickly copy them out if we, or our clients, need good high resolution files for general use.
• We have verified our image data by running our files through the DNG converter.
• We can verify these files long into the future, which is currently not possible for the proprietary raw files.