This nightmare began just before Christmas. A senior executive of a large retail chain called and demanded that his computer supplier remove its computers from his business. Then he canceled his pending orders. Understandably, this really got the supplier's attention, because this customer spent an average of $8 million a year with this supplier.
Large retail chains rely on what is referred to as 7x24 computing during the Christmas season. 7x24 means that the computers run every day of the week, every hour of the day. Most retail chains make their annual profit or loss between Halloween and New Years. When their cash registers (called Point of Sale or POS terminals in the industry) are run from these systems, computers and applications that stay up seven days a week are an absolute requirement. Therefore, retailers purchase this company's computers because they believe they will not go down.
The retail executive's message to the computer company was unmistakable and profane. There were too many failures; computer its latest products were junk; and he was personally sick of fruitless and time-consuming efforts to eliminate the problem.
The supplier was already painfully aware of the fact that its technicians had been working day and night as a result of repeated failures that were damaging this customer's ability to do business. Although its service team has a rigorous path to follow in dealing with this kind of problem, it wasn't working. The problem would not go away and stay away.
The retailer and the supplier were frustrated and unhappy as these failures continued to defy all efforts at preventing them from recurring. Everyone was in figurative--and even real--physical pain. They needed an answer and didn't have time to fail to find it.
At Stake -- A Billion Dollars AND His Job
Of greatest concern was the negative impact on the retailer's Christmas business. The chain could not afford dead registers at this time of year. That would put its entire year and literally billions of dollars in sales at risk. The retailer's executives would lose, not just their jobs, but their careers.
How the computer company dealt with this crisis demonstrates some important concepts for effectively solving business problems. In recounting how this problem was handled, we will analyze what was behind the retailer's phone call and what was done to resolve the problem.
The retail executive who telephoned the computer company was painfully aware of the fact that he was unable to provide reliable service to his customers as the year entered the peak selling period. His staff, who were doing the best they could, assured him that computer failure was the cause of the problem. They reported that there had been repeated efforts to eliminate the problem, but nothing that had been tried had worked.
Up to this point, the executive had been dealing with facts. Then he jumped beyond the facts and assumed that the cause of the failures was the computer itself. As a result, he made the angry phone call previously described.
In seeking to stop these failures, service engineers had repeatedly replaced the same parts. This increased their frustration because it seemed they were dealing with unreliable spares.
This behavior of these people exhibits two common tendencies of both customers and vendors:
- Tendency number 1: blame the product for problems; and
- Tendency number 2: the customer is always right.
However, while defects in your product are often the root cause of a customer's problems, it is dangerous to assume this without any hard evidence. It is especially risky in complex environments like a retail chain's data center. In situations like this, there are so many potential causes that making this assumption is likely to result in seeking the solution to the wrong problem. As a result, your customer will keep on having problems.
It is dangerous for the customer if the vendor assumes that the customer is always right, because in a complex situation the customer cannot always know enough to make the correct judgment. The greater complexity of operations today means that the necessary skills and perspectives can no longer reside in just one expert or even one company.
In this case, a wrong judgment by the retailer's executive is the equivalent of condemning his retail chain to an unprofitable year and him to the loss of his job. Therefore, if the vendor can lend a hand, it would be a grave error for it not to do so.
What We Learned
The path to resolution began with a change in the focus from allocating blame to gathering hard data. Then both the teams assembled to deal with the problem agreed on what to measure. Using neutral facilitation, the two parties found a new way to measure the problem that helped everybody. As a result of applying new measurement tools and, for the first time, pooling resources, information was uncovered that neither company could have obtained independently.
This story shows that frustration and anger can be used to get people to work outside their normal range. We encourage you to replicate the very successful path to resolution that was taken by the computer supplier and its retail customer.
If the computer supplier had continued to assume that the problem was defective products, more resources from both the company and its customer would have been wasted. The customer would have continued to experience failures; losing revenue and becoming increasingly dissatisfied in the process.
Faced with so much pressure and pain, the computer company's account team decided to try a new approach. It opted for a solution that uses hard measurements (metrics) as the basis for discussion and requires the establishment of a relationship between the supplier and the customer which is based on trust and shared responsibility. Achieving this required that everyone agree that neither the customer nor the vendor is always right. This was hard to do.
At this point, the supplier and the customer decided to ask for help. They brought in outside, neutral facilitators who are experts at solving this kind of problem. They quickly decided that more information was needed, as well as a more team oriented approach. The first step they took was to decide what data to gather.
Choosing and Using Metrics
Despite all the work which had been expended to create metrics at both companies, when the people dealing with this problem sat down to discuss solutions, they discovered more work was necessary because they didn't have common ways of describing failures and successes. They could assign blame (at one point, it seemed it was all they could do), but they couldn't decide what was wrong.
Therefore, before they could begin to search for a solution, they had to agree on ways to measure failures. This was not a trivial matter. While it may seem obvious to look at system crashes or "dead" cash registers as the metric, these are unsatisfactory. Needed was a measurement that would make it possible to predict and thus be able avoid blank POS screens. They finally agreed that the appropriate measure is Unscheduled Service Calls (USCs). This choice worked because USCs:
The USC data was pulled. The computer company provided neutral facilitation from headquarters to help the people at the site of the problem work through the issues. The data which was collected confirmed that something unusual was happening at this site.
To everyone's surprise, the data suggested that the problem might be in the environment or was caused by people, rather than the product. Specifically, it revealed that:
The fact that the data surprised everyone indicated that a new solution might be possible. The raw data led them to a new area of discovery. Measuring hard data and getting past the automatic assumption that the customer is right had given them new avenues to pursue in seeking a solution to the problem.
The failures the retailer had experienced could have been caused by the vendor, the customer, or someone else entirely. The computer company and the retailer had tried as two "sides" to figure out how to deal with the problem, but neither of them could nail down its cause. They needed to get away from the blame game and cooperate in finding a solution, because nothing is gained from assigning blame.
Many people are not comfortable with taking this position. Vendors are used to accepting blame, and some customers are used to assigning it. The independence of the facilitator played a key role in getting both sides to realize that it didn't matter who led the them to this point. It only mattered that both sides focus on the problem. This was the only way they could look at possible causes without blinders.
The facilitators asked the participants to form a joint team to analyze the data, identify possible root causes, and develop and carry out a plan to eliminate them. As the team began working, the relationship started to change from an adversarial one to one of true partnership. The team started to focus on solving the problem, rather than assigning blame. Achieving this was difficult, but changing their focus was rewarded by results.
Bingo! The cause of the failures was not the product. It was current circulating through the systems from the local Telephone company's T1 (telecommunications line) installation. While working on site late one night, the specialist noticed that the janitors used polyester dusters. This caused an increase in static electricity build up. Neither the T1 installation nor the dusters alone were enough to independently cause the systems to crash. However, whenever the janitors dusted, the two problems combined and the components in the systems died.
When the supplier and customer had focused on the product, they had been working on the wrong issue! There was no way that the customer, the specialist, or the computer company's technical staff would have discovered the cause of the problem by themselves. If they hadn't worked together, they might still be standing in the computer center--or worse.
The team wasted no time in analysis and reporting. Because everyone who needed to be involved already was, the retailer quickly fixed the problem--with no additional time and without embarrassment or anger. The USC rate dropped and stabilized immediately. This allowed the team to predict what then came true. The systems continued to run.
Forming a customer-vendor team made it possible to look for the root cause of the problem. Looking at independent, hard data supplied the information that was needed. This approach proved its worth after it was discovered that sabotage might be the cause of the problem. As you can imagine, once this became a possibility, progress toward a solution would have been impossible in the absence of a feeling of true teamwork.
Building Dynamic Partnerships
In a successful dynamic partnership, both the customer and vendor take the adult attitude of willingness to accept responsibility for the things in their purview. In a partnership, "the customer is always right" doesn't work because it closes communication and prohibits the partnership from fully investigating a problem and learning the truth about it.
If this type of dynamic relationship had already existed with this customer, the angry phone call probably would never have occurred. However, although belated, they were able to change the relationship--initially through intervention by executives from both companies and the facilitators. As time passed, the first and second level managers from the retail chain and the account team redefined their relationship. They did this by using a clearly defined and simple process based on dealing with facts instead of assumptions.
Conclusion: Surviving Christmas
The retailer is now satisfied with system performance and with the way the relationship works. The retailer made money, and the retail executive kept his job. The computer company is happy with the reduction in frustration, $20 Million in revenue, and saving several hundred thousands of dollars in wasted parts, service calls and executive time. Christmas was indeed merry.
We have shared the lessons we learned from this case, and today we use these tools to redefine relationships with more than 100 companies on three continents. Using them enables us to solve formerly insolvable problems.
-First: Identify the hard measurements that need to be captured and reviewed. (This is discussed above in the in the "Choosing and Using Metrics" section.) Key is to choose data that predicts failure, not data that occurs after failure. You want to avoid the problem, so you need to measure before it happens. Most measures you choose to look at will not reflect problems caused by people. They will reflect a physical state, either good or bad, which is brought about by some process.
Choose metrics that give you an overall view of the systems as well as a detailed view. The USCs made this possible by allowing the team to measure all the systems at once, individual computer rooms, individual computers, and individual subsystems such as disk drives and communication links.
The key to knowing what to measure is knowing what the effect of the results of the measurements will be. If something fails, what will be the effect? What if something else fails? The fact that team members know their businesses is the key to getting this information.
-Second: Ask the parties to form a single, joint team. This team will be the action team to resolve the problem. The charter of the team is important. It should be focused on immediate relief above all else. Cause is a secondary focus. It is important to make sure that the problem does is not repeated, but more important is to make sure that it is solved now.
This team should have two chairs and a number of Subject Matter Experts (SMEs) as well as other representatives of each party. Members or the team are chosen for two reasons. The first is their ability to get things done if action by their department is required. The second is to represent their department's views in discussions. Why do you need this? It is so that after resolution of the problem is under way, there is no chance that you have missed any business perspectives, and there is no chance that a major stakeholder has been ignored.
These are not technical issues. They are business issues. Team members are chosen for their business understanding. SMEs are chosen for technical issues. Team members collectively decide which steps to take and which to fund. For example, in the case described in this article, the team heard recommendations that the grounding expert from Santa Clara be asked to look at the computer room. The team decided and allocated the funds necessary to do this.
Another requirement for team members is that they subordinate their departments' needs to the team's goal.
-Third: Get neutral facilitation. It is impractical to ask team members to represent the perspective of their departments or users at the same time they facilitate. Do not ask someone to both present his or her perspective and to be neutral. While this is theoretically possible, it turns out to be very impractical. Trained neutral facilitators have been a common denominator in the successful problem solving efforts the authors have witnessed.
-Fourth: Ask the team to build a plan to affect the hard measurements. The focus should be on measuring exactly what is needed to get results, and nothing more than that.
-Fifth: Ask the team to review the plan with the stakeholders in each party. This will prevent the team from making mistakes because of something they are unaware of.
-Sixth: - Ask the team to follow the plan, and then insist they measure the results by the same metrics they started with. You cannot know the solution without measuring progress.
If there are two parties (such as the retailer and the supplier) it is important to have two people chair the team who can represent the perspectives of the parties who have equal weight. It makes sense to follow the congressional model. When the Senate and House negotiate bills in conference, they appoint co-chairs for the same reason. Neither party our ranks the other. Each one's perspectives are of equal importance.
Subject Matter Experts are the people who have the technical knowledge needed to achieve a resolution. Often their expertise may be limited to one area, and their business knowledge is narrow. You bring them in for advice, and then let the team make decisions based on this advice. The grounding expert from Santa Clara is an example of an SME. Although he knows electrical issues well, he should not be deciding business issues.
The computer company had a large account team composed of people who serviced the retailer at the data center and at other sites throughout the continent. This team included salespeople, application engineers, service teams, and managers. The account team was led by an account executive who is primarily trained in sales. Many suppliers of capital equipment have account teams with a similar structure.