This is the first in a series of tech articles aimed at helping content producers run the more technical aspects of their studios with greater ease and confidence. As a whole, the series is intended as as a guide to technical subjects for non-technical producers, or content creators, who focus on the creative tools, but not necessarily the engineering behind them.
I received a distressing email from a producer friend of mine a couple of weeks ago. We had talked a while back about some problems he was having with a storage system his studio was using to “back up” the video assets from their web series…We discussed a number of solutions, but the unit wound up failing before they could get around to backing it up or archiving the files—taking a few years worth of production materials with it.
Ouch. That hurts…and I think we’ve all been there. My MacBook hard drive went south on me about five years ago, and it took with it about 10 years of archived correspondence and other stuff that I hadn’t backed up…Which is ironic when you consider that I started out my Hollywood career doing data I/O and storage work for one of the big digital labs in town. Since then I’ve been fairly diligent about backing up to a Cloud server, as well as making periodic local backups to a hard drive.
It only has to happen once, and you learn. Budgets in the web content world are small, and usually spread very thin, and telling a producer that they must spend money on backups system and software, taking take time to create and implement a formal backup plan for all of their assets is a hard sell—Until you lose all of your data, and years of your life (and livelihood.)
Our production assets are the most valuable things we own, yet they often get less care and attention than any of our other production equipment. Depending on the size of the production company, there are some steps that can be taken, at every stage of a company’s growth, to assure that your assets will remain safe for the long haul. But before we start talking about strategies and solutions, let’s talk about a couple of terms: Backups and Archiving.
Data Backups are something that are made periodically, and are intended to get you out of short-term trouble when a file gets deleted, corrupted, or, in the worst case, a storage device fails. Backups can be made a number of ways, from a simple file-copy of a drive to another disk, using backup software like Carbon Copy Cloner, or Apple’s Time Machine, or by using a vendor-based Cloud storage backup solution, such as Carbonite or Crashplan.
Data Archiving is something that is done for preserving your data assets over the long term…Usually 10 years or longer. Tape is currently the best archival medium, with the new LTO-6 tape drives offering media that can hold up to 6.5TB of data and is rated to last 30 years. Archival tape systems come in various sizes from single-drive machines to robotic silos capable of holding hundreds of tapes that can back up many terabytes of data to multi-volume data sets spread over a number of tapes. The latter usually being reserved for large facilities, including the data centers of VFX houses and DI facilities. Archival systems are also used to create data backups.
Regardless of the size of your studio, or content creation operation, finding an appropriate backup and archival process and strategy, one that can scale with your needs as you, hopefully, grow, is vital to making sure that the programs that you produce, and all of the elements that go into them, are protected and available, both now and into the future.
I’m going to outline a few different approaches, based on studio size, but first a word about systems integrators.
At any stage of the game, it’s very beneficial to make friends with your local production/post systems integrator. They have the most experience dealing with the best solutions out there at any point in time, and can suggest solutions that are right for a particular studio size and level of expertise. You may wind up buying a RAID (Redundant Array of Inexpensive Disks) or tape drive from them, and wonder why you didn’t buy it online for 20-30% less (although most integrators are staying price-competitive these days.) The reason is simple: Support.
An integrator will be there for you, providing you with training and ongoing expertise when you have questions. They can recommend a backup strategy for you, and make sure that there is some consistency when support needs arise. They are source for knowledge, and knowledge transfer, and will bridge the gaps when your staff/crewing situation changes. You will never be without a go-to when a crunch situation occurs.
Here are a few strategies for data backup and retention:
Micro Studios (1-3 people)
In these situations you may not have many terabytes of data—maybe a few PCs, perhaps some storage sub-systems for editing. You probably can’t afford the expense of purchasing a dedicated tape drive for archiving, and are using hard drives for backup copies of your assets, and hard drive backups-are okay, as long as you remember to make backups of these drives, and factor in that most hard drives have a life of a couple of years, max.
The simplest solution for the micro studio is to use cloud backup providers, such as Carbonite or CrashPlan. Both companies allow you to sign up for monthly or yearly plans, with a cost-savings associated with committing to a longer term. For home & small business solutions, data is backed-up to the vendor’s cloud, and in the case of CrashPlan, they offer a utility to create backups to a local disk as well…This way you have a local backup, as well as an off-site backup of your data, to protect in case of fire, flood, or zombie apocalypse.
Both Carbonite and CrashPlan offer apps for mobile devices, and web access to your files, as well as options for getting your data shipped back on a hard drive. They both also offer higher-end services for Enterprise clients, and I expect that both companies will continue to expand the variety and scope of their services as time goes on. Either way it’s a solid investment in data protection, starting at about $10 per PC a month. Also, there a numerous other vendors out there providing similar solutions, not just the two companies mentioned here.
One thing to keep in mind, however, is that you will need to have a broadband account that has sufficient monthly bandwidth to keep your data backing up, as well as support your day-to-day Internet usage…I have my machines at home scheduled to back up between midnight and 7am, so that it won’t interfere with my surfing, video streaming, and normal video uploads to YouTube. The more data you have, the more bandwidth you will need, so be sure to figure this in to the overall cost of this type of backup strategy.
Another good practice is to back up all of your production assets to a central storage device…Something like a large network hard drive, an Apple Time Capsule, or a RAID system, like a Drobo. Again, keep in mind, you need to back up your central storage to another device, and the best practice is to keep a copy of your backup data somewhere in secure, off-site, storage.
Small Studios (4-10 People)
At this stage in your development, cloud backup can still be a cost-effective solution, but buying an archival tape unit and backup software starts become a more practical approach. LTO tape units start at about $1500 online for an LTO-5 unit, and go up from there.
There are a number of software solutions for creating and maintaining tape archives, and this is where it really helps to have a relationship with a local systems integrator for choosing the best solution for your needs, as well as getting training and support. The chief feature of tape archiving software is that it will use a database for creating and cataloging datasets, and keeping your data organized for future retrieval.
At this size, it may also be time to start exploring Digital Asset Management, or DAM, solutions. Digital asset managers are workflow-oriented library system that will provide cataloging and extensive meta-data tagging capabilities when storing your data, which will allow you to search and access to all of your studio’s visual assets and production data, from photos and graphics, to VFX elements and video. A DAM can also provide a host of other services, such as proxy creation, file transfer, and even production workflow management and finished output delivery. These systems range from the small, and relatively inexpensive, to large-scale systems, such as the PIX System. Pix is used by many of the major motion picture studios, and is capable of supporting a globally-distributed production team.
One very interesting asset management system on the market right now that is directly applicable to the smaller studio/content creator market is Axle from Axle Video Systems. They have a system that is flexible, scalable, and billed as “Radically Simple” to use. The starting price for the software is about $1300, which provides a very low barrier to entry for even the most frugal studio.
Medium Studios (11-30 People)
At this point the studio has dedicated office space, and there is a network infrastructure in place, even if it’s just an Apple or Windows network. Data storage is shared on NAS (Network-Attached Storage) or a SAN (Storage Area Network.)
Again, cloud storage providers are a possibility, however “Enterprise” solutions are coming into the business conversations a bit more. CrashPlan has some interesting options here, as well as some options that involve use of their dedicated hardware installed locally within your studio. It may be more cost effective to invest in a larger backup and archival system to handle this in-house. The direction at this point will largely be governed by how much of an in-house IT and/or data management staff you have, or whether you outsource your IT to a contractor.
Some type of DAM system becomes more of a must-have at this stage of growth.
At a point where a studio has more than 30 people, office space, including a data center, and at least a solid-core of an IT department. Data management will be primarily handled by your engineering or data I/O department, and you can stop sweating the small stuff, and just wonder why your CTO is spending so much of your company’s budget on things that you used to be able to do yourself. 🙂
I hope that this type of post is useful to the folks in this community, and I would love to hear your feedback, as well as any ideas you might have for future Tech Talk posts.