Disk Configurations

Disk configurations - Logical volumes

When you add one or more hard disks to a computer system, or replace disks in a system, you have several options for the configuration of those disks.

• Each disk can appear to the computer as a single disk.
• A single disk may also be partitioned into several volumes with each volume behaving like an individual physical disk.
• Multiple disks may also be "spanned" so that they appear to be a single disk.

Let's look at these different configurations.

Disk volumes

Every hard disk has a volume structure that lists the size of the available space, and maps out how the data is broken up and stored on the disk. The volume structure determines what a computer "sees" when a disk is connected. In most cases, a single device will show up as a single volume, although that's not always the case. A single device could be partitioned into multiple volumes, and it's even possible to have multiple devices act as a single volume through drive spanning. (More on both of these below.)

Partitions

The most common volume structure for a single hard drive is a single partition, where the drive appears to the computer to be a single drive. It's possible to partition a drive so the computer sees it as two or more volumes. Partitioning separates sections of the drive from each other so each has its own volume information and directory. Some use partitions as a strategy for organizing their media or to create an additional bootable backup of the operating system.

Should I partition my drive?

While it used to be pretty common for advanced computer users to partition drives for performance reasons, that's less common now. We generally recommend that there's little reason to partition a drive, except for certain backup tasks, and those who need to wring every possible performance advantage out of a drive (and who are willing to spend a lot of time to achieve this performance.)

Some backup arrangements need to have an entire volume set aside. Apple's Time Machine, for instance, will not let you use a single volume to backup more than one computer. If you want to send Time Machine backups from multiple computers to a single drive, you'll first want to make a partition on the drive for each computer. This is a reasonably common arrangement for bootable backup software, and it's the most common reason to partition drives.

Hard drive formatting

Hard drives may be formatted in several ways, depending on the hardware and operating system.

Macintosh

For Intel Mac computers, drives should be formatted as GUID Partition Table in order to be bootable. An Intel Mac can read a drive formatted as Apple Partition Map, but can't boot from a drive with this format. An Intel Mac can also read and write, but not boot from, a FAT32 drive (a PC format that is also used on camera memory cards). The newer extended File Allocation Table (exFAT) format is also supported for both reading and writing under OS 10.6 or later.

For Power PC Macs, (G5 and earlier), the drive must be partitioned as Apple Partition Map in order to boot from it. A PowerPC Mac with operating system later than 10.4 can read a GUID- formatted drive. PPC Macs can also read FAT32 drives.

Both Intel and PPC Macs can read from Windows NTFS-formatted drives, but can't write to these drives.

PC

PC drives should be formatted as NTFS, unless you know you need to share the drive with someone on a Mac. You can use FAT32 for more compatibility, but keep in mind it's a very old file system. For instance, FAT32 does not support files any larger than 4GB. Windows computers can also be made to read Macintosh-formatted drives (Apple Partition Map or GUID) by using the program MacDrive. A newer Windows format, exFAT is available, starting with Windows 7, and also offers built-in compatibility with the Mac OS.

Read more about formatting drives for Mac and PC in this section

JBOD

The acronym JBOD stands for "Just a Bunch of Disks", and it refers to a configuration of multiple disks that retain their identity as individual disks. Each physical disk is a logical disk, as shown in Figure 3. It is the simplest arrangement for multi-disk storage, and the easiest to configure, upgrade, and repair. JBOD is a good configuration for photographers to use as primary storage for a couple reasons.

FIGURE 3 In a JBOD disk arrangement, each disk shows up as a single disk.

• It’s simple to purchase and set up. Making an informed choice on the best RAID implementation, on the other hand, is not something that should be done without a good understanding of the hardware and software under the hood. Few photographers are well equipped to make this decision properly. If you plan to use the drive for video editing, however, seek out the help of an informed professional who can ensure the drive meets the performance requirements for HD video workflows.
• It’s generally much quicker and easier to upgrade capacity with JBOD. A traditional RAID setup must be rebuilt completely (all data moved off, all new drives installed) to upgrade storage size. This makes the upgrade process a difficult, time-consuming, and possibly dangerous process. With JBOD, you simply add another drive or replace one drive with another drive of larger capacity.
• It’s the most economical configuration.
• Although RAID is theoretically safer, we find too many people who have lost their entire RAID systems for us to be convinced that it is practically safer, at least with consumer-level RAID.
• The amounts of energy used and heat generated are reduced in a JBOD set-up, since only drives that are being accessed need to “spin up.” Drives housing older material can remain at rest when unneeded.

RAID

RAID is an acronym that stands for Redundant Array of Independent Disks. In a RAID setup, several drives are configured together to act as a single volume. The computer sees multiple drives as a single device, as shown in Figure 4.

FIGURE 4 In a RAID setup, each physical drive also apears as a logical drive.

This can be desirable for reasons of speed and/or redundancy. You can control your RAID setup either by using a hardware controller or by using software. The controller for hardware RAID will either be built in to the drive enclosure, or it could be an add-on PCI card for a tower system.

RAID for redundancy

Some RAID systems write data to multiple drives for added safety. This means that if a drive fails, all the data can still exist on the device. These mirrored or parity RAID setups can be a good choice to store Working files in a mission-critical situation, such as a studio with lots of deadline pressure. And this kind of RAID is essential in a corporate environment where 24/7 uptime is necessary.

RAID for speed

You can also configure RAID to produce significant gains in speed of throughput by reading from or writing to multiple drives simultaneously. This striped RAID is useful in situations where there is a need for lots of throughput, such as in video editing. This can also help a photographer who routinely works with very large image files.

Let’s look at the various flavors of RAID and see what they do.

RAID Level 0 (striped RAID)

RAID 0 treats several drives as a single drive in order to speed up reading from and writing to the drive. Basically, it breaks up a file into several parts and writes those parts to multiple drives simultaneously. While it increases speed, it also increases risk. A failure of one drive destroys all the data. RAID 0 is also used to create a single large volume.

RAID 0 is generally not particularly beneficial for most photography workflows. For video editing, a RAID 0 configuration is very common and is often a performance requirement, particularly for the media files you are using for an editing session. Just be certain that the data is backed up onto an additional drive due to the somewhat volatile nature of RAID 0.

RAID Level 1 (mirrored RAID)

RAID 1 writes the same data to multiple drives to help ensure seamless operation in the event of drive mechanism failure. Mirrored RAID is appropriate for operations that truly need 24/7 functionality, or for files that are works in progress and might represent a whole day’s work. Even with mirrored RAID, you still need to have a set of offline backups because mirroring does not protect against some of the most common causes of data loss (theft, virus, and power surge). It also provides no protection against human error, such as accidentally erasing or downsizing a file.

RAID Level 5

RAID 5 provides a combination of striping and mirroring, resulting in a large single volume with redundant protection. A Level 5 system should be able to survive the loss of any single drive since a parity file (a copy of the data) is written to each of the other drives. Level 5 is probably the most popular current RAID choice for large data storage. The parity file is a compressed copy of the data; it can store the information in a small space, but requires computation to write and read.

RAID Level 6

RAID 6 is an increasingly popular choice for enterprise-level systems since it builds upon the capability of Level 5 by adding the capacity to survive the loss of any two drives in the system. This configuration is typically found in corporate environments.

RAID 0+1 (also known as RAID 10)

This is a configuration with multiple sets of striped drives for speed and redundancy. In a four-drive RAID 0+1, a pair of drives is striped for increased speed. Those drives are then mirrored to the other pair for redundancy. This is faster than RAID 5 or 6 since there is no parity file to be computed or decoded, but it is not quite as efficient.

Which RAID?

Let’s take a look at the considerations to take into account when you set up a RAID system.

Why do you want it?

Only move to RAID because there is a good reason to. Good reasons are that you need the speed, a single large volume, or that you need the 24/7 uptime. “It’s just easier to deal with” is probably not the right reason. Know if you are shopping mainly for speed, size, redundancy or some combination.

All hardware-based drive spanning is proprietary

While there are common terms for RAID levels, it’s important to understand that each RAID 1, 5, 6 or 10 system is a proprietary system. You cannot take drives out of one RAID 5 and install the drives in an enclosure made by another manufacturer, and expect them to work. In each case, a RAID controller is used to split, manage and reconstruct the data pieces back into files.

The RAID system you implement should be one you can get some support for. That may be through the manufacturer, the retailer, the person who sets it up, or through an independent contractor. Before you buy the hardware, know who you will be calling for help if problems should arise.

Software RAID is not so proprietary in that it’s dependent on the computer and OS to do the drive spanning, and the hard drive enclosure is not so important.

Hardware or software RAID?

RAID can be implemented by either your OS, or by a third-party software package, or it can be controlled by a piece of hardware. Each has their advantages.

Software RAID has the advantage of being cheaper, since it can often be configured by simply adding additional internal drives in a tower or using multiple external drives and the system software (or a utility) to set it up. This can be an easy way to provide a speed boost for video editing. If you are using SSDs, software RAID is the preferred method, since it can take advantage of the TRIM control that the OS uses to optimize the drives.

Hardware RAID can be controlled by a PCIe or an ExpressCard expansion card or by a controller that lives inside a RAID enclosure. This allows the use of an external enclosure for added expansion, as well as the use of RAID with computers that don’t have space for additional internal drives. Do some research before you commit to a RAID device. The cheapest one may provide a lot of headaches in the long term.

Many manufacturers, like G-Tech and WiebeTech, sell RAID devices at normal retail outlets. These drives are typically configured in a RAID 0 configuration but can generally be reformatted as a RAID 1 option. These drives do offer a hardware-based solution and can simplify the process of using a RAID.

Which hardware RAID?

There is a difference between hardware controllers. There are cheap ones from companies you have never heard of, enterprise-level devices from well-known brands, and everything in between. WiebeTech is a company that makes, sells, and supports its own line of hardware RAID boxes (shown in Figure 5). Keep in mind that if the RAID card fails, it’s only remotely possible to access the data on the drives if you can get the identical RAID card as a replacement.

FIGURE 5 If you are going to get a RAID setup using conventional drives, consider a hardware RAID, rather than a software RAID.

If this sounds difficult to determine, you might want to think about working through a local reseller who can take responsibility for the purchase. Of all the purchases outlined in this website, this is the one that a photographer or video pro may be least able to make without assistance.

Drobo (BeyondRAID)

In addition to conventional RAID devices as described above, there are also RAID-like solutions available. One of the most popular is the Drobo from Data Robotics. Drobo’s goal is to offer many of the advantages of RAID with some significant improvements, all in a device that is a simple storage appliance.

FIGURE 6 Drobo has made implementation of drive spanning easy for anyone to implement.

At first glance, Drobo acts like a RAID — it spans multiple drives to make one logical drive. Drobo offers units that hold 4, 5, 8 and 12 drives, although you typically only need to load two drives to start. A Drobo device offers the same kind of parity protection you get with RAID 5, where the data can survive the loss of any single drive. Most Drobo units can also be set up to survive the loss of any two drives, like in RAID 6.

However, Drobo offers some things RAID does not. It can intelligently make use of drives of different sizes (unlike RAID, where all drives are chopped down to the size of the smallest drive). When you need to upgrade the capacity of your Drobo system, just take out the smallest drive and replace it with a larger one. The data will be swapped around, making the best use of the new drive space.

Drobo also has some very simple “gas gauge” lights that show you how full it is. As the disks get full, it will show you on the front of the box itself, as well as in the Drobo Dashboard software that comes with the unit. Drobo offers several different connections across its product line including USB2, USB3, FireWire 800, eSATA, and iSCSI. The iSCSI connection can be used to connect the unit to a network or a computer’s gigabit Ethernet port.

Drobo also offers a few products in their File Sharing (FS) line. These can be configured to allow multiple users to connect and share assets. They also allow a single Drobo to be setup for backups directly over a network, without the need for a server.

Drobo also has some intelligent power consumption features. It goes into a very low-power-draw sleep when not connected or not used, then wakes up when it’s needed. And because it is a standardized item, the controller card is likely to be available in the future, in the event that the one in your unit dies and you need to retrieve your data.

FIGURE 7 The DroboPro can span up to 8 drives for up to 16 TB of storage.

NAS

Network Attached Storage (NAS) comes in two flavors: self-contained boxes about the size of an external hard drive, and NAS servers that are effectively small, fully-functional servers and might even have connection ports for monitors, keyboards and external drives. The NAS servers provide an easy way to add a server for a low cost if you don’t already have a computer to turn into a server, but it has limited capability to recover if problems do arise.

Although we generally don’t recommend using NAS as primary storage for professional photographers at the present time, it can be a good solution for someone who wants to keep backup storage online far away from the primary storage. Since you can connect it by Ethernet cable, you have the flexibility of moving it farther away than any other kind of wired storage.

Some NAS products, such as the HP MediaSmart server, run the Windows Home Server OS, and can handle a number of different kinds of files. These devices can serve audio and video files as well as simply storing image files. They can also be used to make files available over the Internet without having to do a lot of custom configuration.

FIGURE 8Stand-alone devices like the HP MediaSmart Server can provide additional storage or backup over a network without having to set up a dedicated server computer.

SAN

A Storage Area Network, or SAN, is an advanced storage configuration that was developed for enterprise-level data storage. In recent years, it has become a valuable storage configuration for video-editing workgroups. In a SAN system, data storage components make a “pool” of data that can be shared by many workstations, appearing to each one as if it were some kind of directly-attached storage, much like an eSATA drive.

A SAN allows for data storage, backup and management to be centralized, even though various different workstations may engage in different parts of the editing process. For example, a producer can review footage, then hand off to an editor who cuts the video. A graphic artist can then access the project to build elements. The whole project can then be opened by a colorist who manipulates color and exposure, and then sent to an audio mixer to enhance the sound. This can all be done without having to duplicate and transfer the project as it moves between tasks.

A SAN system consists of a server, an array of storage devices, and some network connections – typically fast Fibre Channel.

• The server you use must be a real network server, running a server version of Mac, Windows or Linux OS. It can’t be a regular desktop computer with file sharing enabled.
• The storage devices may be a mixed array of hard drive RAIDs along with SSDs, and possibly some flavor of digital tape. This can provide fast storage for commonly-used data, and less expensive tiered storage for less-frequently-used data.
• Fibre Channel offers very fast connections, including the ability for multiple computers to tap into the same storage simultaneously via “Switched Fabric” modes. To set up Fibre Channel storage, dedicated expansion cards are needed, along with special switches and cabling. These cables may be either copper or fiber optic.

Of course this level of speed and flexibility is expensive, and will generally require the services of a dedicated IT contractor to set up and administer.

Special storage considerations for video projects

When working in video, storage can become a much more frequent decision point than with still photography. Video files are much larger than stills, therefore the chance of your needing more storage is much greater. Additionally, tasks like video editing can require specialized solutions in order to maintain the required constant data throughput.

Determining how much storage you need

Knowing exactly how much storage you need is a tough challenge. It is a constant balancing act to make sure you have enough storage for your projects while at the same time not tying up money in drives that sit empty. Storage is a volatile market, one that typically sees prices falling but with unexpected upswings in cost from time to time. You need to balance these when a storage purchase is made.

AJA Data Rate Calculator

The AJA Data Rate Calculator is a useful utility which is available for both desktop and mobile applications. This utility lets you specify several attributes about the video signal, such as frame size, rate, and compression type. It can then provide an estimate for total storage needed based on a time duration. While the software can’t tell you how much footage you’re going to shoot, you can plug in known values for a completed shoot to estimate the cost of transcoding, if that is required. Within a short period of shooting video, you’ll also be able to estimate how much footage you’ll acquire, together with a gross time estimate. 

Let’s consider the following situation:

• For a long format project that you’re working on, you record 10 hours of 1080p 24 fps video (interviews, b-roll, etc). If you used a camera like the Canon 7D (which records video at a rate of around 5 MB per second or about 300 MB per minute) you'd have about 180 GB of raw video.
• For this particular project, the client would like the footage transcoded to work with an older edit system. In this case they’ve asked for Apple ProRes 422. That 180 GB now becomes more than 600 GB of data.
• Of course, in your documentary you’ll probably also want to include photos, music and sound effects, so you’ll need even more space. Rounding that estimate up to account for preview or render files as well as output, and a 1TB drive seems a prudent allocation.

Figure 1 The AJA Data Calculator is a useful tool for accurate assessments of storage needed, based on a time duration.

Using performance RAID drives

When you’re working with photos, you may not notice drive speed as a performance factor. While a faster drive will allow you to load a photo faster, after an image loads, there is not a constant demand placed on the drive. With video editing, however, it’s a very different situation.

When you playback video, you’re essentially asking the drive to load hundreds or thousands of frames sequentially. When this rapid playback is combined with decompressing the various codecs applied to video, you can put an amazing load on a drive. This load is known as data rate. The higher the data rate of a clip, the faster the drive has to be. So if your clip has a data rate of 5 MB/s the drive has to be able to load or playback at least 5 MB/s.

For most DSLR video workflows, you’ll have data rates between 4 MB/s and 30 MB/s. Keep in mind that often times when editing you’ll be working multiple layers of video and audio simultaneously. So, if you have a file with a data rate of 20 MB/s but you are compositing three clips together to create a look, the drive now has to be able to play back 60 MB/s, because the video clips are playing simultaneously. And every cut you make places further strain on the drive(s), as the system has to seek and serve each individual piece of video scattered throughout the drive. (These demands become very visible as you try to edit video from multiple clips together, apply transitions, or use special effects.)

If you are working in an older workflow that requires transcoding to uncompressed video, the problem becomes more acute. The transcoded files are much larger, and will begin to choke your storage throughput pretty quickly if you are using a single drive to serve them.

Most modern hard drives (especially 7200 RPM or faster SATA drives) can handle the load from a few streams of native DSLR video reasonably well, if there are not too many cuts. For the fastest speed, consider investing in a striped RAID solution with high-performance SSDs for your Working files. Of course, RAID 0 is a comparatively delicate storage arrangement, so good automatic backup is an absolute necessity.

Outputting a video project to tape requires extraordinary drives

If you need to output to traditional video tape, the demands for HD video are much greater. In this workflow, you will often have several streams of HD video playing back at once. Additionally, the computer cannot skip or drop frames, otherwise the resulting tape will be jittery and technically unacceptable.

For these reasons, the use of a high-performance storage system, typically a RAID 0, becomes an absolute must. Typically, a user must first render all elements in a timeline to reduce the burden on the drives and system for real-time performance. Additionally, an audio mixdown is often created to simplify the audio burden. Finally, a specialized video card, such as one made by AJA, Matrox or BlackMagic, is likely required to convert the video signal and control the deck’s recording functions.

Of course, it’s possible to do this in two steps. You can first create a finished master video file and then write it out to tape, but this may take additional time that the project does not allow for.