Don't Go To Market Without Storage Software

Imagine having to process online customer transactions at the rate of one every two seconds. If that seems like a gargantuan task ... well, it is. It also happens to be the very task that stock trading and investment firm Scottrade Financial Services (St. Louis) creates for itself every day. Open for customer access 24/7, the company's Web site, Scottrade.com, handles more than 50,000 online transactions per day. During the opening and closing hours of the stock markets, trading volume on Scottrade.com can run as high as 10,000 transactions per hour. Even during non-peak hours, including the middle of the night, the site may still receive as many as 2,000 orders in an hour.

That steady - and, periodically, heavy - traffic puts intense demands on the IT infrastructure supporting the Web site. In terms of hardware alone, Scottrade's IT staff monitors and manages more than 170 servers. Combined, these servers perform the range of functions necessary for tying front end Web applications to back end order processing and database archiving.

Not afraid to add complexity to complexity, Scottrade recently decided to bring its accounting functions in-house rather than continue to outsource them to an off-site service bureau. That move required the integration of back end transaction routing with front end processes that update online customer account status. In moving forward with the project, Scottrade identified one crucial area of back end operations that could not suffer a performance drop-off despite the added burden to the system. It needed to ensure that nightly batch processing of transactions would not impede customer access to online data.

SANs Ease Transaction Processing Burden
To upgrade its storage resources and enhance its storage management capabilities, Scottrade brought in a Symmetrix box from EMC (Hopkinton, MA), as well as EMC data and storage management tools, including TimeFinder volume mirroring software. The Symmetrix unit's 32 SAN (storage area network)-ready ports enable Fibre-based connectivity with Scottrade's application servers. These servers include Windows NT-based Web servers that run the online applications, Sun servers that route transactions to and from the various stock exchanges, and IBM AS/400 servers that maintain updated records of customers' accounts. TimeFinder manages the mirroring of data and operating systems required for efficient batch processing of account information.

Now, when orders come in through the Scottrade.com site, the Sun servers route them to the various stock exchanges via dedicated T1 lines. When orders come back from the exchanges as having been executed, the Sun servers send them to the AS/400s for customer account processing. Then, the AS/400s push completed orders to the Web servers, which populate the Scottrade.com site with updated account information. All the while, the Symmetrix box, located on a SAN segmented from other servers, maintains the SQL Server databases that support the various customer-facing applications.

Mirror Data, Reduce Downtime
According to CIO Jeffrey Polsgrove, Scottrade has realized dramatic enhancements in system performance even with the increase in system demands brought on by bringing the accounting functions in-house. One key scenario that noticeably improved is the soft failover, in which a server or database process has locked up but not actually crashed. Even before Scottrade brought in the EMC tools, all of its servers were clustered in preparation for failover in the event of a hard system crash. If one server went down, another would automatically take over. However, in non-crash situations, when mere lockup was the culprit, the servers wouldn't react. "We had to manually fail them over, and that used to take an hour or two," says Polsgrove. Now, TimeFinder monitors server performance and alerts system administrators when an application doesn't receive a response from a database. "We have cut recovery time from SQL Server database lockups from two hours down to two minutes," says Polsgrove. "A lot of that can also be attributed to the speed of the EMC drives and their Fibre connectivity."

TimeFinder also facilitates the batch processing now handled by the in-house accounting system running on the AS/400s. Scottrade configures two logical AS/400 partitions within a single physical frame. At night, one logical partition is taken offline for nightly batch processing, while the other stays online so customers can access their account data. "We used to have a six-hour outage on the AS/400s to do batch processing," Polsgrove admits. "Using TimeFinder, it now takes only three to four minutes to split the logical partitions and keep the order processing functions available online throughout the night."

Allocate Storage For Archiving
Scottrade is also making use of EMC's SAN management software, ESN Manager. One key application is allocating storage used to archive customer requests for stock quotes. For stock exchange reporting purposes, Scottrade has to keep track of every stock quote that goes out to a customer. "The management tools enable us to dynamically monitor our servers and throw more storage at them when stock quoting increases," Polsgrove explains. "When the market is crazy, people want to see more data. Even on a normal day, however, stock quoting adds a million rows of data to a database."