Keeping the Seats Warm
May 01, 2002
Appeared in Area Development
BILTMORE HOLDINGS, a Phoenix-based developer, got lucky last year at its El Dorado Tech Center in Gilbert, Ariz., when Bank of America took all 185,000 square feet of an unoccupied building on the campus and built a disaster-recovery center.
“The building was constructed as a speculative development,” explains Brent Richter, an investment analyst with Biltmore Holdings. “It sat there for two years before this tenant came along and bought it.”
While companies have been investing in redundant data operations and disaster-recovery centers for the past decade, the events of September 11 changed everyone’s way of thinking — it moved the issue of redundancy to the forefront for many companies that had only taken sidelong looks at such safeguards in the past.
There have always been numerous options for ensuring operational continuity in the event of weather-related disasters, human-error disasters (such as an inadvertent power outage), and, nowadays, a terrorist attack, but basically it all comes down to two choices: a company builds a disaster-recovery operation for its own use or it leases space in an independently operated disaster-recovery center.
Both are solid options — which a company chooses depends on the job it has to do. Many firms of varying size have opted for independently operated centers — of which there are numerous choices across the country and even overseas. In fact, those choices continue to increase.
BankAtlantic Bancorp in Fort Lauderdale had been using a disaster-recovery center in Atlanta, but learned that a local company, Miami-based LightSpeed Infrastructure, was building one in nearby Coral Gables, Fla. BankAtlantic is now considering making a change. The trouble with the Atlanta location, says Becky Cohen, BankAtlantic’s business- recovery officer, is its distance from the company’s homebase. If a situation similar to September 11 occurred and airports closed, there would be no way to get to Atlanta from South Florida in a hurry.
Accessibility is a feature that the city of Martinsville, Va., is pitching. The small city in south-central Virginia has created a business incubator and one of the first businesses under consideration is a disaster-recovery center.
“We are far enough away from the large population centers to satisfy the requirement of a remote site, but close enough to be accessible,” notes Thomas Harned, director of economic development in Martinsville. Companies have a basic need for disaster-recovery backup, he says, “but, obviously, there has been more attention to this issue because of September 11.”
The term “disaster-recovery center” refers to an offsite, sometimes bunker-like facility with deep technology in place where companies can temporarily relocate staff and operations if a disaster occurs. In the past, companies had been putting disaster plans into effect mostly for natural occurrences such as hurricanes, earthquakes, or floods, and for technology breakdowns such as power outages or hardware failures, notes Judith Eckles, a senior director of marketing and communications for Wayne, Pa.-based SunGard Availability Services.
In the United States, the biggest players in the disaster-recovery business have been SunGard, IBM Corp., and Comdisco Inc., but in November, SunGard acquired Comdisco’s Availability Solutions Group, making the new entity the largest in the field with 48 facilities and 2.5 million square feet of space around the country. In comparison, IBM runs 18 disaster-recovery centers.
Even before the merger, SunGard’s disaster-recovery unit had been experiencing 18-percent annual growth, Eckles says. The company boasts 8,000 clients in the United States and Europe.
Disaster recovery is not just a U.S. phenomenon; centers are being built all over the world. In Europe, UK-based Guardian iT plc is the largest provider of disaster-recovery services. It counts 35 recovery centers in the UK and Europe.
Until recently in the UK, the main reason to use disaster-recovery centers was due to hardware and software failures, explains Piper-Anna Shields, public relations manager for Guardian. However, after September 11 there was a surge in interest. “The scale of events was so huge that obviously it made a number of UK businesses sit up and take notice. The unthinkable happened and people began to take stock of what they were doing” in terms of disaster recovery, she says.
SunGard’s Eckles agrees. In the past, she says, companies planned for situations where they only lost access to information; few considered the fallout of losing their whole facility.
A disaster-recovery center is basically an insurance policy on space. Should a disaster occur, a company can quickly relocate staff and operations to an existing building and be up and running almost immediately because the company’s technology has been established in the disaster-recovery center for just such an emergency use. The technology at the site mirrors the operations at the company’s homebase, so when key personnel sit down before computer screens at the new location everything looks the same as it did in the home office.
For independent disaster-recovery centers, the desks generally are unoccupied until needed and the placements are known as “warm seats.” It is an “available room,” explains Chris Healey, a senior account manager with Computer Alternative Processing Sites, a Shelton, Conn., disaster-recovery company with four sites in the Northeast and Florida. Walk through a disaster-recovery center, Healey says, and it looks like everyone went out for lunch.
Actually, much of the time there is some work going on in disaster-recovery centers, because systems need continual testing. When the sites are not being used for disasters, says Guardian’s Shields, clients are testing their recovery positions. “Businesses are constantly changing and solutions are changing in line with the business requirements. Technology reflects the changes and the recovery centers reflect the technology. So testing is an inherent part of business continuity and being disaster-recovery proactive.”
Independent disaster-recovery centers only build a limited number of seats, but they might “sell” five times that number, as disasters generally don’t happen for everyone at one time. LightSpeed Infrastructure, for example, sells 500 seats for every 100 seats it actually has. If a disaster is widespread, LightSpeed can outsource for additional seats.
Under such a plan, a company pays a monthly fee, like an insurance premium, for a certain amount of warm seats — generally only mission-critical employees go to a disaster-recovery center. Should a disaster occur, the center activates the temporary desks, making the warm seats “hot seats.”
LightSpeed offers two fee structures for such a service, according to Scott Lehman, vice president of the company. In the warm-seat stage there is the monthly fee and a setup fee. When the hot-seat stage kicks in, there’s an activation fee and a fee for each seat actually in use.
Whether a company opts for a dedicated disaster-recovery center or leases space from an independent firm depends on what type of liability the company wants, Lehman says. “If a disaster comes and it affects an entire region, then everyone is fighting over the same seat. Some companies cannot afford that risk. The large banks and trading exchanges will establish a hot seat dedicated to that particular company. There is no oversubscription,” Lehman says. “Other companies can afford to take some semblance of risk and costs are significantly reduced if they opt for a warm-seat scenario.”
Much to Consider
OnlinEnvironments, a unit of Los Angeles-based Syska Hennessy Group, is a consulting and engineering firm that has worked on disaster-recovery centers for the past four years. It is currently working with a client in New Jersey who is intent on building a disaster-recovery facility there. The market is a little confused right now, notes Jim McEnteggart, an associate partner of OnlinEnvironments. “You would think there would be a lot of action with the 9-11 [fallout], but everybody is hemming and hawing, trying to decide what to do.”
While commitment is there, companies are trying to get a handle on needs. “The prime drivers,” McEnteggart says, “are what technology needs to be backed up and proximity to headquarters. Whoever has to work in the backup facility has to be near enough to get there, yet the facility has to be far enough away if there is a natural disaster.” As to redundancy, principal concerns include density of load, number of servers, and flexibility. For most companies, McEnteggart says, it is more economically feasible to make an agreement with an operator of independent disaster-recovery centers, but warns, “you have to be careful to make sure they don’t oversell their space. Also, you have to be sure certain standards of reliability have been built into the system. You don’t want to go to a recovery site and have failure there — it wouldn’t be appreciated by the clientele.”
According to Doug McCoach, head of the Applied Technology Group at RTKL, a design consultancy, there are four possible operational structures for disaster-recovery centers:
Backup: Minimal size; not staffed
Monitoring: Minimal size; staffed to maintain oversight; monitoring hardware only; not necessarily 24/7
Critical Operations: Staff to manage shift of mission-critical operations to new site; monitoring and operations hardware required
Networked: Part of the active network operations of the company and able to switch between facilities at will
Citing a survey of downtime costs conducted by Contingency Planning Research Inc., McCoach explains the necessity of disaster-recovery centers in terms of money lost. For example, he says, the cost of one hour of downtime in the brokerage industry is $6.45 million; for credit-card companies, $2.6 million; for home shopping, $113,000; for package shipping, $28,000; and for ATM fees, $14,500.
At those numbers, customers understandably worry about their systems’ percentage of reliability. “When we talk about percentage of reliability, we are referring to percentage of uptime,” McCoach says. “If you say the percentage of reliability is 99 percent, you may think that is pretty good. But, if you look at a facility that runs 24 hours a day, 365 days a year, and is up just 99 percent of the time, then you are looking at an outage rate of 87.5 hours per year. In the brokerage business, if you multiply 87.5 hours by $6.45 million, that is a lot of money and it is definitely worth buying a remote backup facility.”
In mission-critical facilities, the percentage of reliability needs to be 99.999 percent, McCoach says. A disaster-recovery center, McCoach observes, can be in the same building or on the same site, but most companies prefer it at a separate location, which reduces certain risks but generally increases costs. Still, says McCoach, “there are definite operational advantages to having a facility located outside of the same weather patterns, out of the same geological zone. We deal with a lot of environmental risks — earthquakes, hurricanes, etc. — so if we can get some level of separation between the base facility and the redundant facility, that is a benefit.”
The banking and finance communities have been the drivers of disaster-recovery centers in the past, but after September 11, a host of other types of companies, from manufacturers to call-center operations, have been seeking space. The federal government also has entered the market for disaster-recovery centers. Indeed, it has become an integral part of business strategy, filling the need to keep companies running as they should, regardless of circumstance. As McCoach says, “disaster recovery is like having a second car idling in the driveway.”"