Mission Critical But Not Impossible
September 15, 2002
Appeared in Facilities Design & Management
As the events of 9/11 unfolded, people were amazed at how many of the firms displaced from the twin towers and vicinity were able to continue with business as usual. Aside from having emergency plans in place, these companies realized the importance of mission critical facilities. These off-site locations were where these companies kept some of their most vital assets—data and information. The companies who remained up and running while the world crashed down around them planned proactively to create and maintain these vital, but seldom used, facilities.
FMs would do best to start by focusing on their worst nightmares, suggests Subodh A. Kumar, principal of the Chartered Facility Management Group, a Pasadena, California-based strategic planning, design, construction, and operations and maintenance consulting group. "Create a list of all the things that could happen," says Kumar. "This list will be different for every industry in every location." And don't forget the regulatory environment; utilities, for instance, are part of the national infrastructure and are held to different requirements in terms of business continuity.
"The role of the FM is to absorb uncertainty," says Kumar. "People want certainty. And we can provide that by looking at all the areas of vulnerability and trying to address them."
"You need a war mentality," says Mark Gibson, president, HLW Resources, a New York-based consulting firm specializing in outsourcing and technology solutions. "You need to do scenario planning. Think about what to do with each individual bit of each facility in every circumstance. If you don't do this, your company will eventually go down."
Kumar agrees. "Eighty percent of businesses without a business continuity plan in place do not come back if they're down for more than one week. We are in a buyers' economy; buyers have choices and will go elsewhere if a company is not open for business."
Potential disasters, he proposes, are usually natural, technical, or human. Natural threats can include internal and external flooding, internal and external fire, seismic activity, high winds, snow and ice storms, volcanic eruptions, tornados, hurricanes, tidal waves, and typhoons. Technical threats include power failure/fluctuation, heating, ventilation or air conditioning failure, malfunction or failure of CPU or application software, power outage, telecommunications failure, gas leaks, or communication failure.
Human threats, which we are more attuned to in the post 9/11 world, could include burglary, bomb threats, embezzlement, extortion, vandalism, chemical spill, civil disorder, work stoppage, computer crime, hazardous waste, sabotage, and, of course, terrorism. But disaster doesn't necessarily mean a major headline-grabber. "It could be some little widget somewhere that no one ever thinks about," says Bill Back, district facilities manager at a mission-critical facility for Houston-based Computer Associates.
Bear in mind, says Kumar, the initial event doesn't necessarily cause the most damage; often the most damage is collateral. On September 11, for instance, the towers didn't fall because of the impact; they fell due to the fires. Indeed, the North Tower was hit first, but the South Tower fell first because its fire protection wasn't as thick. On a more mundane level, when a gas line bursts, the subsequent fires often cause the most disruption.
It is important to think about every aspect of a company's operations as there may be more mission critical activities than come immediately to mind. "This is an extremely detailed process," says Gibson. Critical facilities should take into account everything from the radius of a bomb blast to the more likely scenario of the sprinkler system going off in the computer room.
Think about email. "Most of our communication these days is by email," says Kumar. "As a result, almost no matter what business people are in, if their email is down, they can't work." What began as an alternative means of communicating has now become mission critical.
Another often-forgotten critical function is paying employees. Most businesses focus on their customers, but often employees cannot wait a week to receive their paychecks. Companies must determine where payroll fits into their mission critical plan.
Once you have determined what could happen you need to figure out the relative likelihood of each threat. "Rank the order by the likelihood of each threat happening to your operation in your location," says Kumar.
Now you have to define what functions qualify as "critical" for your organization. If you ask the head of IT, the answer will be data; if you ask the head of HR, he will likely say people; if you put the question to the CFO, you will mostly hear "whoever affects the bottom line."
All of these functions are critical to a company's success. The question is simply how long can a company survive without them fully functioning. But don't just think about the monetary aspect, suggests Kumar. Public perception is key. "If you have a problem and your business goes down," says Kumar, "what does that say about you and your company that you didn't even protect it?" After an emergency, customers want to know two things, the consultant points out: Is the company still in business? And when will it be able to meet people's needs?
A company has to think about what kind of revenue a given function generates, and what will be affected if something gets lost, explains Ann Banning-Wright, senior vice-president, OnlinEnvironments, Syska Hennessy Group, a New York- and Los Angeles-based consulting firm. "If something happens to check processing, you could end up with corrupt data and lost interest. If something happens in your biotechnology research, you could need to redo the whole project," she adds.
Gibson proposes considering several levels of criticality:
Grade 1 is a function that must be operating 24/7 without exception. Think of Amazon.com's web site. "That can't be down for even a second," says Gibson.
Grade 2 is for functions that must be up and running within four to six hours, or half a workday. "This level includes people who can jump in a car and go to another facility, such as the American Express Travel Center's call center," suggests Gibson. Computer Associates' Back recalls when a leased space lost its water. "That meant no restrooms, no water fountains, no coffee. It didn't kill us, but we had to have a plan for it."
Grade 3 is for capacities that can be out of commission for no more than 24 to 36 hours. "This includes functions like the CEO's phone number," says Gibson. "He can go to voicemail for 36 hours, but he'd better be up and running in three days."
Grade 4 is for functions that must be working within one week's time. "The human resources department can go here," suggests Gibson.
Grade 5 is for one month. Salespeople's offices, assuming they have access to the company database and a phone number, can fit into this category.
Grade 6 is for functions such as the secondary warehouse.
Once you have determined the list of possible threats and the order in which company functions must be up and running, the next step is to put a dollar figure on each scenario. "Consider what is the likely loss and what it would cost to keep the function going," says Kumar. How does a company figure that out? "The CFO will probably give you a figure," Gibson says.
Companies should take an integrated approach to planning. "It does no good for IT to have one plan that assumes FM will be able to provide backup power," says Kumar. Better to mesh all the needs—and abilities—into a single plan.
Be careful about over-planning, too. "We often have to talk to companies about whether they are spending too much on preparing for possibilities," says Banning-Wright. "Don't go by the current industry practice; bring disparate parts of your organization together to make decisions. Often, the people who get the most money for mission critical planning are usually the ones with the most persuasive arguments, not the greatest need."
Scrutinize what is important and what is mission critical," she adds. "Companies could spend a lot of money in the wrong spots and not end up with the reliability they need. For instance, what if a brokerage firm has its connection to the stock market intact, but loses its phone list?"
It is also critical to think through all response plans thoroughly. Some companies with more than one call center think that they can serve as each other's backups. If one center goes down, for instance, its employees can simply report to another center quickly. But if the second center is fully staffed, there will be no desks—or, more to the point, no telephones—for those transferred employees to use. Perhaps a better approach would be to change the message on the call center's machine explaining that a disaster has struck and calls will be responded to within three or four minutes instead of the usual one to two minutes, suggests Kumar.
The plan must take into account that once the police and fire departments arrive at the scene of a disaster, they control the situation. "They can tell you whether you can go into your facility," warns Kumar. Companies should determine who will talk with these officials. "And the CEO is probably not a good choice. All the senior staff will be handling unplanned issues," says Kumar. The point person should probably be someone in the facilities management department.
Given today's technology, backing up data isn't so difficult. "It's essentially a no-brainer," says Gibson. What is more complex is redundancy in people. "Data can be reproduced easily enough; cloning isn't such a good sport these days," Gibson adds. "Institutional memory is more important than data. And so is cross-training."
Think, for example, of top execs. ""What about having the CEO and the COO in two different states? In two different continents?" proposes Gibson. "With modern telecommunications, they can operate just as though they are in the same room."
And backup in personnel isn't just for high-level functionaries. "For instance, if you're going to transfer your backup data somewhere," says Kumar, "you need someone to transfer it, someone to make sure the emergency power is on."
Whether it's personnel or data or equipment, one question to consider is the level of redundancy. "Do you need one redundancy? Two? Or even more?" asks Glin W. Jay, managing principal of the Dallas-based In Formation Company, an FM consulting firm. In addition, companies need to plan outside their own four walls. "It's good to know who your neighbors are," says Jay. "If there's a high-energy use facility within your utility grid, for instance, it may not kill the energy supply totally, but it could give you more than your share of brownouts."
Corporations also should take into consideration the vulnerabilities of their suppliers, says Gibson. "If your processing plant in India gets blown up, did they have a disaster recovery plan?" It's useful to know—and prepare for—the answer. "You might want to discuss it with your insurance company, too," says Gibson. Companies also need to be realistic about their demands on their employees, cautions Kumar. "If there's a tornado," he says, "are people going to check on the office first or are they going to check on their families?" Companies need to think about that in setting up their plans."
Managers who determine their companies need to outsource their backup will find several levels of backup, depending on the vendor. "Internet Data Centers, or IDCs, are the highest form of mission critical," says Earnie Leake, regional facility director, Exodus, a cable and wireless service focusing on IDCs. To guarantee service 24/7/365, these centers offer infrastructure (heavily redundant cooling and power as well as advanced security), connectivity (through multiple fiber trunks), and support services.
These facilities can serve as disaster recovery sites, where data is stored until the unplanned-for happens; they can be hot sites, which are always up and running; or companies can actually use, day-to-day, the servers that the IDC stores and maintains. "It can be a huge capital investment to design, build, and maintain this type of facility," says Leake, "so people often outsource the function."
But beware the outsourcers, suggests OnlinEnvironments' Banning-Wright. "Check the outsourcing companies. Often, when they talk about their reliability, they mean their power, cooling, and computer hardware and software. They're not necessarily talking about the staff practices." Outsourcers might not have a good disaster recovery plan of their own. "Scrutinize the way the people keep the data up and running," she says.
"9/11 got everyone thinking about planning for disaster," says Gibson. "It raised the awareness. Unfortunately, it also has everybody focusing on one particular form of terrorism, rather than considering the range of possible scenarios," he adds. "The message of 9/11 is to think about it. Don't just react. Think about it."