Traditional enterprise infrastructure management tends to focus on the components of software and hardware systems. It may include Web and application servers, databases and various network infrastructure components. Unfortunately, this siloed approach to measuring availability doesn’t provide a complete view of the user experience. It can indicate database server uptime, or average transactional latency, but it doesn’t tell you about the customer experience.
From Servers to Users
Business Service Reliability (BSR), or service assurance, builds on these key application parameters but also measures the holistic customer experience, ensuring that every customer interaction is successful. I recently asked Tony Davis, VP of Solution Strategy at CA Technologies, about how he approaches BSR.
“Let’s say I’m running an e-commerce site,” he explained, “and a customer makes a request that goes to a cluster of servers. Assume one of those servers is having a slight problem. If I’m monitoring the cluster, it may show 100 percent availability, but to the customer whose request went through the troubled server, service was degraded or even unavailable.” With BSR, Davis said, availability is measured by analyzing each user transaction.
A 3-Part Solution
When tailoring a program for a client, CA Technologies uses a master formula. The first part measures availability based on very granular criteria, as in the example above. The second part measures performance, again with stricter criteria than most companies use today. For instance, if a specific transaction typically takes less than a second to complete, but with an occasional outlier, the entire transaction is marked as defective. The third part of the formula is accuracy. This measurement varies case by case, using key parameters from an application’s workflow that are continuously measured in real time.
Combining these three components results in an application grade -- a percentage that measures the level of service reliability you’re providing customers or end users. That grade is continuously reported to IT so that improvements can be made. More than 60 organizations use CA Technologies’ BSR methods, including Blue Cross Blue Shield, BBVA Compass, Lexmark and EMC.
The Human Factor
Also key, Davis said, “is looking at how a human being experiences an application. We measure every transaction against the three variables of the formula. If they fail on any transactions, we mark the entire customer experience as imperfect.” BSR measures both the frequency of a good user experience with an application as well as whether the user got the desired results.
In other words, metrics deal with more than numbers. “If your application deals with 1 million requests per day and 4 percent were ineffective in some way, that means 40,000 humans you did business with had a bad experience,” he said. This can lead to poor customer relationship and even loss of customers and revenues.
Samir Datt, Managing Director at Protiviti Inc., a global consulting firm that helps manage risk and optimize the technology driving critical business processes, describes his approach to BSR as going beyond server utilization and network latency. “We have helped our clients identify, measure, and mitigate the risk factors that can disrupt revenue, internal processes, and business overall.” (See related Q&A for more details.)
Protiviti, a CA Technologies systems integrator that uses BSR methods, works with its clients to define thresholds and take proactive measures on performance issues before they impact end users. DevOps, which encourages better collaboration between app-dev and production, is part of the solution to “drive improvement upstream and better enable BSR and improved coordination,” Datt said.
Tools and Support
Once the three stages of the master formula are in place, specific tools--such as CA Customer Experience Manager --are introduced to measure end-user experience. CA Application Performance Management may be recommended to gain deep insight into application performance, and CA Application Delivery Analysis for end-to-end application analysis. The CA LISA Suite, which supports a DevOps approach when monitoring with BSR, and custom dashboards to gain instant insight into the user experience, might also be part of the plan.
How much do these tools and practices improve user experience and reliability? “If you’re really good, you would be 99.5 percent perfect,” says Davis. “And then you would have to figure out how to tweak that last .5 percent of defective customer experience,” he says, only half joking.
(Additional case study details can be found at www.ca.com/bsr.)