A Game Plan For SAN

Like any professional sports team, the Carolina Panthers play to win. On the surface, success is easy to measure. A glance at the scoreboard approximately three hours after kickoff tells the tale. But, the Panthers are not just a team, not just a collection of players and coaches. The Carolina Panthers are a franchise. So, like any competitive enterprise, the Panthers organization also keeps an eye on internal scoreboards. It monitors productivity and efficiency in all lines of business, from sales and marketing to accounting and finance. On any given day, all departments are focused on putting the best possible product in front of the revenue-generating customers who pack Ericsson Stadium or tune in on TV.

No department at the Panthers understands the need for a "play to win" attitude more than the IT department. IT provides front end to back end support for all business, stadium, and football operations. Included in that support is IT's most direct and, therefore, perhaps its most critical contribution to the team's performance. IT maintains a digital archive of footage from NFL games, team practices, and mini-camps. Having already grown to more than 1.5 TB, the archive enables coaches to compile endless combinations of digital video clips as they train players and prepare game plans for specific opponents.

Of course, giving coaches fast access to data-intensive video files is not accomplished simply by invoking Coach Lombardi's famous dictum, "Run to daylight." In a network environment that supports myriad applications and users, communication lanes must be cleared before anyone can run to data. Until recently, a reliance on server attached storage threatened to clog those lanes with too many information tacklers. So, like a left guard pulling to lead a sweep around right end, the Panthers' IT staff decided to migrate data to a SAN (storage area network).

Keep LAN Traffic In Motion
Even if centralizing storage hadn't reduced the LAN burden, the Panthers would have welcomed the addition of a SAN to the IT playbook. If nothing else, the move relieves storage management and file-serving headaches for an IT staff whose five members already wear a lot of helmets. "We are responsible for everything technical within the facility - anything to do with computing and networking," says Roger Goss, the Panthers' director of IT. Goss' team designed and maintains a cabling infrastructure that includes 54 telecommunications closets carrying the LAN around a facility nearly a mile in circumference.

In addition to the standard business applications most IT departments support, the Panthers' IT staff must also handle data requirements unique to sports teams. Maintaining digital video clips for coaches is just one of them. "We spend a lot of energy - on both infrastructure and application integration and support - in areas such as ticketing, scoreboard operations, radio production, player scouting, and injury tracking," Goss explains. As if that weren't enough, Ericsson Stadium houses more than 430 concession areas, all running POS (point of sale) systems. So, whether the data crossing the LAN is about Mel Kiper's picks or pickle replenishment, Goss' team gets it there.

"Because we support so many operations, we wanted to be able to dynamically allocate storage and move it around in a more efficient way. We didn't want our crystal ball to be a big factor," Goss says. "So, we moved all of our application servers and SQL Server databases over to a SAN in our central data center." In each server, a Fibre Channel HBA (host bus adapter) provides the connection to the SAN. Correspondingly, a high-speed Brocade switch fabric connects each server to the LAN. The primary storage unit is a StorageTek (Louisville, CO) 9176 disk subsystem.

Design A New Blocking Scheme
Managing data for such a wide variety of applications - each demanding dramatically different types and sizes of files - is akin to avoiding dissension on a peewee football team. Every kid wants the ball. Knowing it can't fake handoffs to any of its users, the Panthers IT staff uses StorageTek's Object Manager software to slice and dice the disk arrays on the 9176 into volumes of different sizes. Large segments are used for block transfers, such as moving and playing the digital videos. Standard business applications may need to be tuned to move smaller chunks of transaction data. For them, partitioning small segments can increase database performance. "We can define as many disks in an array as we need to. For digital video, we use eight or nine individual drives within one RAID [redundant array of independent disks] 5 array," Goss explains. "All of the storage for the digital video archive is presented to the server as one large drive, even though it's really striped across multiple disks and drives." With the flexibility the system provides, the IT staff can monitor capacity and throughput, reconfigure RAID levels, and reallocate volumes - all without taking the stored data offline.

The IT staff has so far migrated half of the 6 TB stored in the stadium's data center from server attached storage over to arrays on the StorageTek 9176. Of the 3 TB already on the 9176, more than 1.5 TB are dedicated to the digital video archive. "Our storage needs increased fivefold when we started integrating video on a daily basis," says Goss.

Make Video Content Ready For Replay
With only one week between most games, coaches don't have time to waste waiting for video clips to load from storage. So, IT retains up to two years' worth of video online in the disk arrays. Video clips more than two years old are archived to tape.

While online access is crucial for fast retrieval, it constitutes only part of what makes the digital video archive perform so well. Another key is the ease with which coaches can query the database to locate relevant clips. Traditionally, NFL game films have been captured on Sony Betacam tape, the league standard for more than a decade. Teams' video departments made copies, which coaches reviewed in linear playback mode, forwarding and rewinding, forwarding and rewinding. In the last few years, video departments have begun to digitize the analog films in order to create customized, digitally edited tapes, although coaches still tend to view them linearly.

With its online archive, Carolina has taken the use of digital video to the next level. The archive's SQL Server database includes metadata about each video clip, including time codes that identify when each play begins and ends. Certain data (e.g. down, distance, yardage) is imported from league-generated statistical information. The Panthers' quality control coaches watch the games and log additional information. For each play, they chart up to 157 factors, including details about the offensive formation, defensive coverage, personnel on the field, and the outcome of the play. All of this information becomes part of the exhaustive metadata that helps coaches to quickly identify the clips they want to review.

To facilitate access, Goss and his IT team developed a database-querying application. The program's query generator allows coaches to search the metadata based on any factor, and pull up a file emphasizing offense, defense, or special teams. A defensive coach may, for example, want to review all clips of the Panthers' defense against a particular offensive formation in third down, short yardage situations.

The searchable archive of video clips gives the Panthers a competitive edge when it comes to thorough game preparation. It also offers a unique twist on savings. Because coaches can retrieve clips within seconds, they can keep all of their time-outs for the Panthers' final winning drive.