Remote Monitoring: The Next Generation Of Field Service
By Brian Albright, Field Technologies magazine
An M2M (machine-to-machine)- enabled remote monitoring solution has transformed the way Emerson Network Power provides service.
If you provide field services to Fortune 1000 companies like eBay, Google, and NASDAQ, you can’t afford to fall short when they need you most. Emerson Network Power knows this, and that’s why the company has turned to remote monitoring to ensure it is providing optimal service to these crucial clients. Emerson Network Power manufactures the Liebert brand of precision cooling and power management solutions for data centers. The company provides cooling units, uninterruptible power supplies (UPS) systems, batteries, and other components that help keep data centers from overheating, and ensure that back-up power kicks in if there is any type of electrical interruption.
Large data centers are complex facilities to manage. The heat generated by the racks of servers has to be dissipated using special cooling equipment, and each device requires specialized back-up power equipment to ensure uptime. Emerson manages data centers for large companies like those mentioned before that can’t afford any type of power failure. The back-up systems Emerson provides are critical, because they can keep the equipment powered in case of a system failure or power outage, until the emergency generators can kick in. If the cooling and power systems fail, a data center could potentially lose capacity or even go offline.
The ramifications of an outage for any Fortune 1,000 company can be phenomenally costly. If a regional bank loses its ATM network, it can cost upwards of $50,000 per minute. Because of the high cost of downtime, Emerson’s customers rely on the company to ensure downtime doesn’t occur.
Preventing downtime isn’t always easy. Batteries age and require regular maintenance and replacement. Some batteries may only last three years; high-end units could last up to 15 years. “The batteries are like tires; they start wearing out the day they get installed,” says Gregory Ratcliff, director of professional services at Emerson Network Power’s Liebert Services’ business. “They all age at a different rate at different sites, depending on the temperature, the load drops they have, and other factors.” Precision air units and UPS systems have fairly long lives (10 to 20 years or longer), but specific components like transformers, blowers, or capacitors may need replacement during that time.
Emerson is in charge of maintaining information about the health of these systems for each customer, and deploying maintenance personnel when a failure is detected. The Liebert Services’ business alone has a field service staff of more than 500 technicians in North America (Emerson Network Power employs about 2,000 techs globally). Technicians had to regularly visit each customer site to inspect the power and cooling systems, and make any necessary repairs.
Basic Remote Monitoring Lacks Crucial Functionality
Emerson was actually one of the early pioneers of remote monitoring and has used a version of the technology to monitor temperature, voltage, resistance, and other metrics on its battery equipment. However, the company’s initial remote monitoring solution used a dial-up, modem-based remote communication system to send alerts to technicians in case the equipment experienced an alarm condition (like a battery failure). These systems only sent updates periodically, meaning that alerts were not received in real time.
The solution was also hard to manage from a hardware perspective — thousands of modems were connected at the customer sites. The modems connected to a device called a universal monitor, and one was required for every few pieces of Liebert equipment installed at the customer site (one monitor might be able to connect to three or four precision air cooling units, for example). “The system was cumbersome to maintain,” Ratcliff says. “We had to use dial-up to poll all of those systems, which meant racks and racks of modems. And the solution wasn’t scalable — no one wanted to deploy enough universal monitors to remotely monitor all the equipment they have on site.”
While Emerson’s early remote monitoring system provided some very basic functionality, technicians were still required to visit customer sites to perform regular inspections and maintenance, and to perform basic services like changing setpoints on equipment. This took hours per day for each technician, and generated labor and fuel costs with each truck roll. For example, one technician in Ohio had to service a customer site that was not near his base of operations. He would have to drive two hours to the site, be escorted through the secure facility, inspect and service the location, then drive two hours back. “He expended that effort even to do something simple, like tune the system or adjust a value on the equipment,” Ratcliff says. “It was very inefficient and costly.”
The Shift To M2M-Based Remote Monitoring
Emerson knew it needed to update its remote monitoring solution with one that would provide more detailed information, in real time, and provide the company the ability to perform some tasks remotely. Emerson sought to migrate its dial-up system to an M2M (machine-to-machine)- based remote monitoring solution to provide the added functionality it was looking for. Ultimately, Emerson’s goal was to find a solution to help reduce operational costs from unnecessary trips, reduce equipment failures, and improve uptime at customer facilities.
At the time the company began looking for an alternative, options were limited. “ We considered putting a permanent virtual connection to those systems in place,” says Ratcliff. “However, having a router that could manage 3,000 VPN connections was incredibly expensive.” In this scenario, each site would need a VPN connection so that the devices could communicate from the customer’s LAN to Emerson’s network. “The VPNs would only send data when we poll them, or if there is an alarm. It would have been terribly inefficient and expensive to set up these VPNs that were mostly going to be empty,” explains Ratcliff.
For the system to work, the Liebert power equipment had to be able to securely communicate back to Emerson’s own network without requiring an always-on VPN connection. “We had numerous pieces of equipment we wanted to ‘phone home’ out of a customer’s network, but we don’t want to make the customer do anything,” Ratcliff says. “We wanted to create a tunnel that is encrypted and secure back to the home company, and we wanted to do that in an automated fashion.”
While researching its options, Emerson discovered ILS Technology, a company that provided dynamic remote monitoring technology that could temporarily open a connection between the company and its customers’ networks (to which the equipment is connected), solving the problem of needing an expensive, constant connection while also providing real-time insight. Ratcliff says that when Emerson decided to move forward, ILS Technology had the only viable solution.
Emerson decided to equip its customer sites with an ILS deviceWISE Gateway (which Liebert has branded as its own Ntegrity Gateway) that opens a temporary “pipe” that allows systems at the customer site to communicate directly with Emerson’s internal technicians via the ILS secureWISE connect remote monitoring platform. The customer only has to approve the gateway once, and maintains a “white list” of customer-approved IP addresses that the monitored equipment is allowed to access.
According to Ratcliff, the primary challenge in deploying the solution was the necessity of upgrading the communication infrastructure at each site from dial-up communication to Ethernet and SMTP-based communication. “It took some time to deploy the infrastructure for the solution, but it was by far our best option,” explains Ratcliff. “Taking this approach cost approximately half of what it would have cost to deploy a VPNbased solution, and all of that cost would have been up front. We would have had to build that capability before we were able to scale up the remote monitoring business, whereas with the ILS Gateway, we can expand the solution as the business grows.”
For existing customers, Emerson technicians had to install the deviceWISE Gateway at the site, and then convert any devices that would have to send data. Because the Gateway is a separate device, Emerson has been able to quickly upgrade customer sites, as well as bring new customer sites online. Approximately 90% of Emerson’s new customer implementations are on the remote monitoring system (existing customers are still being upgraded).
Emerson preconfigures each Gateway with parameters from the customer’s security staff, and then ships it to the customer site. The IT staff at the customer location racks the Gateway, and Emerson technicians have immediate access to any of the equipment at that location. “When the devices connect, the ILS technology allows them to behave as if they are on the Emerson corporate LAN even though they aren’t,” Ratcliff says. “This gives us the same functionality that having thousands of VPNs running out to customer sites would have, without actually having a VPN running.”
Liebert’s SiteScan Web centralized monitoring and control software (which resides on a PC-based server at each customer location) manages the precision cooling and power equipment on the site, and is responsible for all telemetry and alarm management. The SiteScan Web system sends all data and alerts back to Emerson via the ILS Gateway and secureWISE connection to database systems on Emerson’s internal network.
The data is then compared to Emerson’s historical equipment database, which contains millions of data points about how the various pieces of equipment perform over time. “Since we are also the manufacturer of the UPS and the precision cooling gear, we know what those early warning setpoints are for imminent problems,” Ratcliff says. “We might see that a capacitor bank on a UPS needs to be changed, so we can replace that and the unit is like new again. If we see the resistance or voltage of a battery is falling out of compliance, we can replace it before it fails.”
If the SiteScan Web system detects an alarm condition, the ILS Gateway initiates a network connection and alerts Emerson’s alarm management center. The company then evaluates the scenario and either contacts the customer or deploys its own technicians. The ILS Gateway itself also sends out a heartbeat poll, so Emerson knows if the system is offline or has gone to sleep. “The system routinely wakes up and asks us if we need anything,” Ratcliff says. “When it does that, we use the remote system to collect operational parameters from the equipment. For a UPS, that might be current, voltage, temperature, alarm status, etc. We put that into a database, and our global team monitors each one of those elements.”
Monitoring that data can help Emerson notify customers of any anomalous conditions, not just power failures or equipment failures. “We had one site that was showing inconsistent readings,” Ratcliff says. “It turns out their generator had kicked on during an outage, and it never went back off. Our technicians noticed the problem and called the customer. They would have run out of fuel if we didn’t have that remote telemetry in place.”
M2M-Based Remote Monitoring Creates New Revenue Stream
With the ILS deviceWISE and secureWISE connect solution in place; Emerson technicians can now remotely access the company’s equipment to perform remote maintenance and upgrades. Emerson is also able to provide new monitoring services for its customers, which has created a new revenue stream for the company. One of the early benefits of the new solution was eliminating the dial-up modem infrastructure, which was expensive, difficult to maintain, and unable to send timely alerts. “Every remote system called us every day, and it was a big burden to have an analog line that cost $50 or more a month for every single device. Our customers had to pay for those lines, and there were thousands of dial-up-connected devices in the field,” Ratcliff says. “We no longer have to have that massive infrastructure in place to accept calls. We also get data much faster. With the old system, we had to determine how long after one day did we wait to see if a system had gone to sleep? Now we know within a few minutes if something is wrong.”
That means Emerson can contact customers immediately if there is an issue, sometimes before the customer is even aware their network is down. That capability is critical for maintaining Emerson’s reputation in the industry, because timely notifications help avoid system outages at customer sites. To date, there have been no system outages due to failed batteries at customer locations where those battery systems are professionally maintained and remotely monitored using the ILS technology.
The ability of Emerson to perform preventive maintenance on these systems also increases system reliability, and the regular alerts generated by the remote monitoring solution help properly direct these efforts. According to Emerson’s calculations, the mean time between failures (MTBF) for UPS systems that receive two preventive service events a year is 23 times higher than a machine that has not been serviced. Before, with the dial-up system, preventive maintenance tasks had to be completed on site, which was time-intensive and costly. “With the old solution, we had very limited data coming in from the equipment, and we had to rely on in-person inspections,” Ratcliff says. “Now we’re able to perform these preventive service tasks remotely.”
Emerson maintains a variety of service-level agreements with each of its customers, ranging from simply providing alerts to the customer’s internal data center staff, all the way up to full service contracts that require Emerson technicians to perform all maintenance and repairs. In some cases, where Emerson technicians had to be on site to make relatively simple equipment adjustments, the remote access is saving the company dozens of truck rolls per year, per customer. A recent statistic released by industry organization TSIA states that the average truck roll costs a service company more than $1,000, so the savings of this benefit add up quickly.
In addition, Emerson can now more easily deploy software and firmware upgrades. “Over dial-up, those transfers were difficult because the files are too large,” Ratcliff says. “And there are services that we offer now that would be impossible without that direct connection to the systems.” For example, Emerson receives alarms for temperature and resistance conditions on its batteries. “We can remotely connect into the battery monitoring system, and change the values and run tests,” Ratcliff says. “The system is really our eyes and ears in the field.” By expanding capabilities like these with remote monitoring, companies like Emerson are often able to charge a premium for the advanced offerings to increase revenues while providing better service to customers.
Customers Begin To Embrace Remote Monitoring
Although the updated remote monitoring solution has proven its worth for Emerson, customer acceptance was not universal when the system was first rolled out, particularly since many of Emerson’s customers have stringent security requirements and had doubts about allowing external access to their network. “If our customer is a bank, there are months of approval we often go through before we get the goahead to install the Gateway,” Ratcliff says. “But you only go through that once, and then the solution is in place and we’re able to reap the benefits. We have an adoption rate of about 90%, but some customers decline — there are just some sites that aren’t going to allow any remote network connection.”
There was also some initial pushback because the idea of remote monitoring was new to many customers. According to Ratcliff, increased adoption of remote monitoring has improved the comfort level a bit among customers. “When we first put the solution in place, our conversations with customers always started with a ‘no’. It was a tough, uphill battle early on,” Ratcliff says. “Over the past two years there’s been a big change. A lot of customers have just come to expect that high-end systems are going to involve remote management. It’s much more accepted now. The market has come around, and there is other equipment out there in other industries that has remote monitoring built right in.”
Emerson’s sales team is trained on the specifics of the remote monitoring solution, and can provide customers with multiple white papers explaining the benefits of the technology. “That’s been really valuable when they have that first discussion with the security professionals,” Ratcliff says.
Emerson is now in the midst of expanding its own remote monitoring and management offerings, and the ILS Gateway continues to play a key role. Emerson recently announced Trellis, a data center infrastructure management platform that will help further expand its remote monitoring functionality. Other Emerson Network Power business units are also deploying the ILS technology.
Next up, Emerson plans to introduce wireless connectivity to the ILS Gateway solution to provide connectivity in cases where customers don’t want the Gateway to connect to their LAN, or if they want a back-up wireless connection in case their LAN goes down. “There’s no cost savings to that for us,” Ratcliff says. “In fact, it’s a bit more expensive, but it increases value to the customer. We’re already piloting wireless at select customer sites.”
For service organizations, remote monitoring and management technology makes it easier to provide proactive service and prevent equipment failures before they happen, while enabling new types of services like remote software upgrades. At Emerson, this type of monitoring is the only way to ensure its power management and precision cooling solutions provide the level of reliability its customers expect. For these data centers, failure is never an option — and 24/7 monitoring via a secure connection has become critical for both Emerson and its customers.
Taking A New Approach To Remote Monitoring
When Emerson Network Power’s Liebert Services’ business wanted to streamline its remote monitoring solution and upgrade to real-time alerts, its options were limited. The company’s existing system used dial-up modems to send alarms and alerts; however, this method lacked reliability since the modem was a single point of failure, and the modems had to dial out each time to communicate alarm data.
The company chose ILS Technology’s deviceWISE Gateway, which is based on a directed circuit approach to monitoring. The Gateway receives data from the devices in the data center, then uses the ILS secureWISE connect platform to broadcast alarms over a secure Internet connection using simple mail transfer protocol. The Gateway only allows the vendor doing the monitoring (in this case, Emerson) to access specific devices on the network. If new equipment is added, the company simply recommissions the Gateway to recognize the new device, and then locks it down again, making the solution easily scalable. Emerson is able to provide real-time alarms to its customers. “We didn’t want to just rely on the Gateway communicating back to us if there was a failure,” says Gregory Ratcliff, director of professional services at Emerson Network Power’s Liebert Services’ business. “We also wanted to help our customers manage our products. They see alarms internally, too.”
While Emerson is using the ILS solution to monitor power equipment, deviceWISE can link any device to any application to enable real-time information transfer, transaction logging, and bidirectional control. Although Emerson chose to deploy the device- WISE Gateway device at its customer sites, device- WISE is also available as enterprise software that can be loaded to a server or an embedded agent that can be loaded directly onto the devices you’re monitoring. ILS also offers flexibility with its secure- WISE connect platform, with a software option that can be installed on customer or ILS-supplied hardware, or as PaaS (Platform as a Service) that is hosted by ILS and offered as a subscription-based service. For more on ILS Technology, visit www.ilstechnology.com.