Migration Overview

Digital storage is a process

Digital storage is not a place you put stuff, but rather a process your digital information enters. You can't expect to put a digital archive someplace and come back to it decades later and have access to everything. The media may have become corrupted, or so obsolete that it's hard to find a system that can connect to it. The operating system that was used to configure the storage may have become obsolete. Or the file format may become unreadable by current software. If you want to protect your valuable digital photos, you'll need to perform a migration from time to time.

Keep and eye out for migration triggers

In all likelihood, no one is going to send you a memo telling you it's time to migrate your archive. You'll have to take responsibility for it yourself. There are some telltale signs that it's time to consider a wholesale change. Here are some signs:

• Your storage media starts to have repeated problems. If you have 10 drives, and two failed in the last year, it's time to think about replacing them.
• New storage media has significantly higher capacity and/or speed. If your desk is filled with 120GB drives, and 1.5 TB drives are cheaply available, it's probably time to consolidate the files onto fewer drives.
• You start using software that has new capabilities. If you start using dedicated catalog software, you might find that a migration from proprietary raw to DNG offers you important new functionality.

Figure 1 If your backup drive collection gets too large, you might want to migrate it onto fewer drives of larger capacity.

Data migration steps

Let’s look at some general workflow principles for file migration. As everyone’s situation will be a little different, you may have to customize your procedures; however, these general guidelines should still apply:

• Get comfortable with your new system. It doesn’t make sense to start the process until you know exactly what you are migrating to.
• Create a comprehensive plan. Think about what you are doing (the totality of the task ahead), and break it into manageable chunks. Make sure you have all your materials at hand.
• Track your progress. Make catalog snapshots along the way. Saving a catalog of the material at different steps of the migration process can be very helpful. This will let you reconstruct what you have done if you make any mistakes or omissions.
• Freshen up your backups before migration.
• Start the process with enough storage space at hand.
• Keep a legacy copy for a while. Don’t immediately erase the old configuration until you are sure that everything has migrated as planned.
• Make sure to back up the new configuration. You're not finished until you are backed up.

Get comfortable with your new system

If you are making a large change, you shouldn't worry too much about integrating your legacy images until you have worked the kinks out and gotten very comfortable with it. This will make you more likely to only have to do this work once, and do it efficiently.

The hardest jobs in the DAM universe are ones involving migration. Sorting out a disorganized and duplicated collection can be a brain-busting process as you look over old work and wonder, “What was I thinking when I did this?” Don’t make the hardest thing you do with DAM software also be the first thing you try to do with it.

Depending on the kind of migration you’re trying to do, this may be best accomplished by putting new hardware into service and leaving the legacy system as-is until you’re ready to migrate. Migrating computer and OS, for instance, are often best accomplished by running systems side by side for a period of weeks before trying to make the final switch.

Create a comprehensive plan

Any time you’re going to do a wholesale change to a bunch of data, it’s important to map out the entire process from start to finish. Some changes will be easy enough to implement without a written plan—copying location names from the keywords field to the IPTC Location Fields, for instance, is a self-evident process. Consolidating two dozen small and semiduplicated drives onto new storage, however, may need some charting out. Don’t be afraid to put pencil to paper; sometimes that is the only way to do it reliably, even in the computer age.

Part of developing a comprehensive plan includes deciding which files you need to migrate, and when. Some of the migration tasks that you will be faced with will involve large numbers of files at once. These tasks include storage and OS migration, the type of tasks that affect the entire archive. Other migration tasks will best be done a little at a time. For example, it may only make sense for you to add film images to your digital archive when you actually have a use for those particular images, rather than trying to add all of them at once.

Still other tasks may require you to do part of the work comprehensively, and part of it incrementally. For instance, you may want to bring all of your legacy digital images into any new directory structure at one time. This will reduce the probability of duplication and will encourage thoroughness. However, you may prefer to do the work sorting these files for duplicates, or entering metadata into the files, on an as-needed basis.

Track your progress

One of the most useful tools that cataloging software offers for collection-wide work such as file migration is the ability to create “snapshots” of your collection as you proceed. If you save versioned catalogs at each critical step in the migration process, you will know exactly what you did during the process, and when.

Because the catalog shows where files are, how many images are included, and what metadata is associated with them, you can reconstruct your steps and make sure that everything worked as expected. For instance, you might suspect that you have lost a bunch of files. By looking through the saved catalogs, you can see if there was a problem, where it occurred, and thus have a better chance of correcting your mistakes. If you don’t save versions along the way, you will only know where you ended up.

Always know which is the primary copy

At some point in the migration process, there is a change as the primary version of your files moves from one “place” to another. As you perform any migration, make sure it’s clear to you when the “primary” designation changes. If you are consolidating multiple drives onto new storage, for instance, you should consider the primary copy to be the new copy, as long as there was a validated transfer.

Don’t add new material to the old structure once migration has begun

Once you’ve moved your material, make sure you don’t add to the older version. Doing this adds to uncertainty and increases the possibility of data loss.

Validate

Whenever you are doing a migration, there should be some level of validation before, during and after.

• Before you migrate, you'll want to have a good idea that the files you'll be migrating are in good condition. It would be a shame to migrate your primary storage, and then make new backups from that, only to find out that it was corrupted in the first place. (Even more of a shame if you throw away your old backups before you discover the problem.)
• As you migrate, you'll want to use validation for the migration itself. For storage migration, that can simply be a validated transfer, where you confirm that the copy is an exact duplicate of the original. For format migration, you may also want to make sure that every file came along for the ride.
• And after you finish the migration, you'll want to do some validation of the new data, if possible. Check for completeness and file integrity to the extent you can.

Read more about data validation

Finish the job

And finally, make sure to see the migration all the way through to the end. If you don’t finish, you can make a big mess for yourself.

Freshen up your backups

A migration is one of those tasks where there is often coincidental risk and a chance of cascading errors. You’ll want to make sure your backups are good and fresh. It’s also good to consider validating your backups prior to a migration. We suggest always making a fresh bootable clone before any major OS upgrade and prior to any other software upgrade that might be problematic. We frequently find that when conducting an OS migration, it's also a convenient time to upgrade the hard drive. You can pull the old hard drive and store it for a month or more while you give the new OS a workout.

Start the process with enough storage

One of the most common mistakes is starting a migration without enough storage. This leads to all kind of errors, including accidentally deleting files and deleting old files before the new ones have been fully validated. It’s way too easy to wipe entire drives with a few clicks. Storage is getting really cheap. Buy enough of it.

Keep the legacy system for a while

Whenever you make any major changes to your computer configuration, we suggest you try to keep the legacy system in place for a while. You can then do some real-world testing of the new configuration, while keeping the currently functioning system available in case anything goes wrong.

For instance, when installing a new version of a mission-critical application such as Photoshop, we suggest keeping a fully functioning version of the current configuration at hand as the new one gets set up. That way, if there are any serious problems, you can go back to the known system and get the day’s work done. If you have done the upgrade on your only copy of the system, you have no choice but to work out all the problems before moving forward.

Keeping the parallel structure is also useful if you are transferring images from an old drive to a new, larger one. I suggest that you keep the old drive(s) intact for a while, until you are comfortable that the new storage is functioning correctly.

Back up the new configuration

The end of many migration processes will be the backup of the new system or data. If you are migrating operating systems, you should make a clone of the new OS once you see it works as expected. If you have renamed files, added metadata, or done some other work to your files, make sure you back it up once you validate it.