Instant Replay

"Defeat The Darkness." On promotional copy from Microsoft, this bold mantra appears under the title Nightcaster, a wizards and spells action-adventure game designed for Microsoft's Xbox video game system. It's also an apt description of the fearless attitude that companies, including Microsoft, must take in order to subdue the forces of data loss. When data needs to be rescued, an IT staff member often has to heroically step into the shadows otherwise known as offline storage. If that storage resides on removable media in an outsourced facility, the data may literally have to be pulled from a cavernous netherworld. Nevertheless, as a last line of security, remote data centers and underground storage facilities are essential. However, for moment-by-moment protection against extended periods of downtime, it also helps to have complete copies readily at hand. Even the fearless prefer to handle crises from within the comforting surroundings of their own data centers.

For the IT operations team at Microsoft Gaming Studios (MGS), which develops games for the Xbox system, the weapon of choice for restoring software development data from on-site and off-site copies is surprisingly the same. MGS uses tape - in particular, DLT (digital linear tape) drives and media from Quantum Corp. (Milpitas, CA). Compared to hard disk or optical disc storage - seemingly more obvious choices for fast, on-site retrieval - DLTtape-based storage enables MGS to maintain huge sound- and graphics-intensive files relatively inexpensively. And, a company doesn't have to be a video game developer to understand the demand such files can place on storage capacities and budgets. Organizations in many verticals are increasingly adding audio and video files to their lists of critical digital assets. Of course, at MGS, they are the assets.

Tape Takes On Terabytes
It's easy to see why MGS needs to maintain as much as 1.2 TB of locally stored data, as well as multiple terabytes stored in remote data centers. Six or seven Xbox games are typically in production at any one time, and the production cycle for each game lasts from seven to nine months. For each game, developers periodically compile software code, graphics, and sound files in order to release a "build" for quality assurance testing. Over the course of the production cycle, developers release dozens, if not hundreds, of builds, including several builds each day as a game moves into its final stages before shipping. A typical early-stage build may be 10 GB or larger. A final build for a single game may require 40 GB to 50 GB of space. Since each tester is responsible for detecting bugs and glitches in a particular aspect of the game, each build has to be copied and forwarded to as many as 10 or 15 people.

In addition to the locally stored builds, the uncompressed digital assets and code that make up the games must be archived off-site for disaster recovery purposes. That requires another 30 to 40 GB of storage capacity per game. "We're a software development house, so it's work, work, work/develop, develop, develop," says Shane Nelson, network administrator for MGS. "Once the product passes certain milestones, it goes into testing. From then on, it's test/fix, test/fix, test/fix. During every text/fix cycle, a lot of data is moving back and forth."

Tape Saves Server Space
Its unique production processes have made MGS something of a renegade within Microsoft's overall IT infrastructure. The corporate Information Technology Group (ITG) usually supports every Microsoft business unit. However, unlike most Microsoft units, MGS always has several data-intensive products in development at the same time. "We require more servers, faster networking, and archiving on a larger scale than ITG typically provides," Nelson explains. For instance, ITG doesn't provide backup to desktops or to machines outside of Microsoft's corporate data centers. And, it doesn't offer on-demand archiving. That's why, to supplement the ITG-supported backup to corporate data centers, MGS brought in equipment to handle in-house production archiving.

Now, developers store their daily work on network-attached Compaq ProLiant DL380 servers. Each is equipped with a 300 GB Ultra 160 SCSI (small computer system interface) hard drive array striped in a RAID (random array of independent disks) 5 configuration. For disaster recovery purposes, all of the production data in non-build form is backed up from those servers to DLTtape via a wire connection to a remote Microsoft data center. Nightly in-house backups of all MGS servers, including production servers, file servers, database servers, and Web servers, are handled by a dedicated network backup server to which two ADIC FastStor DLT autoloaders are attached. Each device is equipped with Quantum DLT 8000 tape drives. However, because those backups pull in data from all servers, tapes stored in the autoloaders are relied on for restoring builds only in the event of a major server outage.

More immediate backup and restore needs are served by additional Quantum DLT 8000 drives directly attached to the workstations of the MGS IT staff. From those workstations, staff members use VERITAS Backup Exec software to manage the process of backing up production data for each game's development team. So that IT staff can quickly restore the work of a particular team, a DLTtape cartridge (or set of cartridges, if need be) is reserved for making frequent copies of a particular game as it moves through repeated test/fix cycles. "Low disk space on our servers causes constant battles. The development teams frequently fill their allotted space," Nelson says. "We are constantly encouraging people to request that we archive their materials to tape." To help in that effort, MGS relies on WQuinn's QuotaAdvisor software to limit and monitor the amount of server space each development team can use.

According to Nelson, previous builds have to be restored from tape two or three times per week as developers and testers work to trace and eliminate bugs. In those instances, the data is brought back online using either the DLT drive on an IT staff workstation or a DLTtape station in a CD and DVD burn lab. (Compiled builds are burned to DVD, the medium on which Xbox games are tested, certified, and released for purchase.)

MGS also restores cancelled projects from tape. "All of the data related to a game that gets cancelled in the middle of its production cycle is archived to tape and stored in a file cabinet. That includes bug-tracking databases and all of the builds," Nelson says. "If management decides to reactivate the project, we have to bring that data back online."

Tape Defeats Downtime
With DLTtape locally and remotely backing up production, Nelson and the MGS IT staff can enjoy a fundamental benefit of any backup solution - peace of mind. "Downtime is a primary worry, particularly during the last stages of the game development cycle. Testers can't enter new bugs in the database; developers can't check bugs that have already been recorded," Nelson says. "Four hours of downtime in the final week can affect the shipping date for the product and cost us hundreds of thousands of dollars. That's why we rely heavily on tape."