What If You Could See Your Storage?

Unless you enjoy acute panic attacks, try not to admit to yourself how inflexible your company's storage infrastructure really is. Unless you love insomnia, try counting sheep, not the number of users who will be affected when their application servers run out of storage capacity. If you're lucky enough to work for an organization that has already moved its storage devices to shared, networked environments, it's best not to recall the old days. Lingering still, no doubt, are the nightmarish memories of drained disk resources, server downtime, and harried hardware purchases.

Yet, despite the need for more restful nights, some companies delay the transition to full-blown networked storage. Instead, they nurse the infrastructure along - adding disk arrays, dealing with downtime, plugging in more boxes. While they may bring on the odd NAS (network attached storage) appliance or pilot a single-platform SAN (storage area network), they continue to rely primarily on direct attached storage. The "buy another box" approach induces frequent administrative headaches for an IT team charged with managing an increasingly sprawling infrastructure. Eventually, a migraine-causing systems crash occurs - a very public meltdown that leads everyone, particularly line-of-business managers and enterprise users, to declare, "That was not good. That can never happen again."

A version of this familiar story line recently played itself out at Warner Bros. The entertainment giant had been relying heavily on direct attached storage in its main Burbank, CA data center and across the enterprise. How heavily? According to Anthony Lloyd, Warner Bros.' VP of computer operations, "Buying more hardware was basically a monthly process. It was painful." Not only was the company making frequent purchases to add capacity, it was unintentionally building barriers to efficient storage management. Because many of its several hundred servers had access only to direct attached storage, the IT team could not accurately monitor or predict storage utilization rates across the enterprise. In addition, backup/restore operations between direct attached disks and backup tape devices were lengthy.

When a file-and-print server for one large Warner Bros. division crashed from overutilization, the IT team suddenly had approximately 80 GB, or some 800,000 files worth, of mission-critical transaction data to restore from tape. That job took 6 1/2 days. That's right - not hours, but days. Although the division's operations were not brought to a standstill, the scare underscored the point that enough was clearly enough. Warner Bros.' storage infrastructure needed to become reliable, flexible, and, most importantly, scalable. So, in came the SANs.

Multiple Platforms, Multiple Fabrics
When Warner Bros. decided to build its SANs, it had what Lloyd describes as a "hodgepodge" of servers and storage devices - from multiple vendors, on multiple platforms. "We had something from everyone under the sun, including Sun," says Lloyd. As the SANs do today, the pre-SAN server and storage infrastructure supported the full range of the company's production and business operations. These include applications used for feature film production and the distribution of home video and other Warner Bros.-branded consumer products, as well as apps for human resources, financials, database administration, and so on. The resulting management challenges were particularly demanding in Warner Bros.' main data center in Burbank. The 22,000-square-foot facility was loaded with direct attached server storage. "Because the business was growing rapidly, we didn't have a good way of projecting what the growth model would be for a particular application's storage requirements," Lloyd says. "And, we had no way to standardize the server infrastructure. Everything was a one-off based on departmental or project needs." (Warner Bros. maintains small data centers for its operations in London and New York. It also replicates its SAP environment to a remote, co-located facility primarily used by the AOL side of AOL Time Warner, Warner Bros.' parent company.)

The need to build two SANs in Burbank stemmed from Warner Bros.' decision to keep certain applications in a UNIX environment, with others staying in a Windows NT/Intel environment. (A third UNIX-based SAN is in place at the AOL co-location site.) Mission-critical business applications, as well as some production apps, run on UNIX-based servers from Sun Microsystems. Applications that primarily support specific departments or smaller divisions run on Windows NT-based Intel servers. These include file-and-print apps, database apps, and e-mail apps, as well as the Windows active directory infrastructure.

Except for some older servers and storage devices that can't accommodate Fibre Channel connectivity, the servers on both sides are connected to a SAN via 16-port SilkWorm 2800 fabric switches from Brocade. Providing SAN-based storage for the more than 300 servers in the Windows NT environment are devices from XIOtech. On the UNIX side, where over 100 servers reside, EMC Symmetrix boxes handle the job.

SANs Scale And Protect
With most of its servers already connected to the SANs, Warner Bros. can not only monitor and manage its storage resources more efficiently, but it can also scale them without losing that efficiency. That's important for a company accustomed to business - and data - growth. "We usually add about a TB every quarter," Lloyd says. "We already have 8 TB of data stored on the Intel side and 16 TB on the UNIX side here in Burbank, plus another 14 TB in the EMC SAN that supports SAP in the co-location facility." To keep up with growing storage needs, the SAN fabrics are growing as well. When it initially deployed the SANs, the company installed four of the Brocade 2800s on each of the two main data center SANs. It now has 20 switches providing server to storage connectivity.

Also attached to the SANs are 20-drive StorageTek L700 LTO (linear tape open)-based libraries. Warner Bros. uses IBM's Tivoli Storage Manager to run initial full and ongoing incremental backups. Having the SANs in place also enables Warner Bros. to use Tivoli's archiving functionality to run disk-to-disk snapshots. That capability reduces the window for full backups because the snapshot version of the data can be moved to tape at a more convenient time.

SANs Drive Down Upgrade Costs
According to Lloyd, the SANs have definitely brought the storage management visibility and flexibility Warner Bros. was looking for. "We are now able to proactively find out when applications and users are going to need more disk space," says Lloyd. "And, we can easily reallocate storage resources to other LUNs [logical unit numbers] on the SANs. In the direct attached days, we couldn't do that."

A major benefit of moving to SAN-based storage is a dramatic reduction in costs of Warner Bros.' server purchases. With the SANs taking the bulk of the storage load, application servers no longer need the throughput and capacity required to serve up data from direct attached devices. Thinner servers mean thinner expenditures. "In the past, we would often spend in the neighborhood of $30,000 for a server. Today, it's $13,000 to $14,000 per box because we don't need the additional processors, disk drives, and cabinet size," Lloyd says. "In fact, with many of the servers now as small as 2U or 4U [one unit equals 1.75 inches], we have been able to retrieve 1/3 of our data center floor space."

As he looks to the not-so-distant future, Lloyd anticipates a replacement strategy that will make the server infrastructure even leaner. That strategy involves purchasing blade servers and rebooting their operating systems from the SAN. "Next year, as servers come off lease, we'll start deploying blades. When that happens, a server that used to cost us $13,000 to $14,000 will be, instead, a card," Lloyd says. "We'll go from having one or two dozen servers on a rack to 100 servers on a rack." Talk about being able to see - and manage - everything in your infrastructure. With the SANs in place, the paths are certainly clear at Warner Bros.