For any organization with a media library, migrations are an unwelcome but necessary task. They are periodically required to ensure that your content is on modern, supportable storage media, but migrations may also be dictated by corporate decisions to switch between vendors or technologies based on factors other than age, such as performance, cost, or support.
The benefit of an archive management system is that it abstracts the storage media from the users and systems who need to access its content. You get a single point of access to all of your assets, regardless of whether they are stored in the public cloud, the private cloud, or on-premise. Robust storage management systems like Telestream DIVA can ensure not only that you are insulated from issues like hardware outages, but that you can also operate uninterrupted while content moves between the various storage systems that make up your ecosystem.
History of Archive Technology
Since it was introduced in the late 1990s, LTO tape has grown to be the most common archival medium for broadcasters and other media organizations. LTO tape has many benefits, but one of its downsides is the necessity to migrate content to a new generation of media every few years. These migrations are sometimes driven by the lifespan of the tapes, but most often they are due to a desire to replace LTO drives and tapes. The move to newer drives and tapes not only provides greater density and speed, but also makes replacements and additions easier to source, and avoids hardware obsolescence.
Optical disk archive (ODA) is a slightly more recent technology that is an alternative to LTO. Like LTO, ODA uses removable media that are housed in a library device. However, the media in this case are optical disks. Migrations are also periodically required for content on ODA disks since new revisions are released roughly every three years.
In recent years, cloud storage has gained market share as an archive medium. Cloud storage provides a dramatically different business model from on-premise storage systems like LTO libraries, optical disk libraries, and disk-based storage systems. There is no hardware to purchase, so the library storage costs move to the OPEX budget. To take advantage of these benefits, you first need to get the content from your on-premise storage to the cloud. Even when the content has been migrated to cloud storage, there is always the possibility that you will change cloud storage vendors at some point in the future, necessitating another migration.
Questions to ask when migrating
Regardless of the source storage, destination storage, or the reason behind the migration, there are several factors that should be considered:
Which content will be migrated?
Do you want to move all of your content to the cloud, or just assets that fit certain criteria (age, high-usage, low-usage, historical value, etc.)? Do you want to prioritize certain content in the migration? While it can be tempting to prioritize individual assets, it is more efficient to read all of the content off entire tapes/disks, since loading and unloading media can add a lot of time to the process. In addition, using a system that reads the assets in the order they are positioned on the media will eliminate unnecessary drive head movement. Of course, just because you’re reading everything off the existing storage doesn’t mean that it needs to be kept. Your migration can also be an occasion to clean out duplicates, assets whose rights have expired, or any other content that no longer has value.
How will the migrated content be stored?
If you are migrating from one LTO/ODA revision to another, this isn’t really a factor. However, if you are migrating content to cloud or disk storage then you may have the option to choose less expensive storage tiers for some content. These storage tiers typically require a tradeoff in terms of their performance and/or responsiveness, so many customers use them for older and/or less-used content. If you opt for a hybrid approach with content stored both on-premise and in the cloud, then it may be possible to use less responsive cloud storage tiers for all of your content.
What resources are available for the migration?
Reading and writing the migrated files requires the same resources that you use for archiving and restoring assets during normal daily use. Adding resources like LTO/ODA drives and/or data movers enables the migration work to be done in parallel without impacting your everyday archives and restores. Regardless of whether or not it’s possible to add resources, it may be useful to schedule the migration for off-hours when the system is normally less active. The migration will also put a lot of additional wear and tear on your LTO/ODA drives, so depending on the amount of content being migrated you may need to budget for replacement drives in case your existing ones fail.
Do you want to transform the content during the migration?
Over the lifespan of your archive, it’s inevitable that your video format will change, resulting in a mix of different formats within the archive. A media migration is an excellent opportunity to normalize your content, either by transcoding legacy content to your current production or playout format or by transcoding everything to a mezzanine format. Some organizations may even want to consider the opportunity provided by more efficient codecs to decrease the bitrate of their content. While this option may not be suitable for all use cases, for news organizations this may provide an acceptable tradeoff between cost savings and video quality. If you don’t currently have proxies for your content, this can also be done as part of the migration workflow. While proxy workflows can provide value in any configuration, they are especially important when the library is in the cloud, since the egress of the high-res material can incur fees (depending on the cloud storage solution you use).
Do you want to extract metadata from your content during the migration?
Just as the migration provides an opportunity to transform content, it can also be a good point to enrich legacy content. Almost all media companies have assets that are inadequately cataloged. Machine learning, such as speech-to-text, object recognition, celebrity recognition, and sentiment analysis can be used to automatically generate metadata for these assets, making them easier to find. While there is a cost for these services, the addition of metadata can help to uncover “hidden gems” within your archive.
How are you ensuring the integrity of your content?
One of the benefits of storing content on public cloud storage is that you don’t need to manage the migration between physical storage media. However, that doesn’t mean that the content is not moving. Files in public cloud storage may be getting copied between tiers or between regions. Even when the files are “at rest” in a specific tier and region, the cloud storage vendor is periodically migrating the content to new media. Depending on the vendor and the tier that is being used, this media may be disk storage or LTO tape. Regardless of the media type, ”bit rot” is a legitimate concern, so it is beneficial to generate, store and check hashes or checksums to verify that your content retains its integrity.
Summary
Each migration presents you with challenges, but also with opportunities. Putting some forethought into a plan that takes into account your existing technology resources and your long-term archive strategy will result in an archive that is more efficient, responsive, and cost-effective in the long run.
Get in touch today to learn more about how Telestream can help your organization.
To read more about migrations, click here to download the Solutions Brief, or for other interesting blogs, please visit this page.