We all have struggled with a lost file, a loss of power, and the dreaded frozen computer screen. In recent years, organizations have combated insidious computer viruses and grappled with major black-outs and extreme weather conditions. Companies have also been tracking the escalating costs of business interruption. By fully assessing facilities and processes, businesses are striving to prevent downtime.
The power outages of August 2003 led to rolling black-outs across the northeastern United States and southeastern Canada. With flickering power frequencies detected as far south as Georgia and as far west as Arkansas, an estimated 50 million people were affected by the power outages.
This fall, a series of large storms ravaged the United States. Hurricanes Ivan and Jeanne left 1.3 million homes and businesses in Florida without power, while tornados touched down in New Jersey, demolishing numerous structures.
Events such as these are causing businesses to consider the true cost of downtime and to act proactively. “We credit the number of large power interruptions that are taking place these days to stimulating a lot of interest,” says Bob Bauer, president at Liebert Corp., Columbus, OH. In the business of mission-critical power and cooling technologies, Liebert Corp. delivers products and services in power protection and environmental controls.
Stepping Back to Move Forward
“When considering business continuity planning, you really need to take a step back and understand the business you are in,” says Bauer. According to Bauer, clients often focus on solving a particular problem. Instead, he encourages building owners to step back and review all of their businesses’ processes that are vital to fulfilling critical products and services. In a bank, for example, this would mean transactions; for a manufacturing plant, a production line; and for a service company, fulfillment services.
The marketplace is evolving, states Bauer, and companies are increasingly pinpointing process flow diagrams of all the essential parts of their organizations to prevent business interruption. He adds: “If you are striving for business continuity goals, make sure you examine all of those areas and have the proper level of protection so that you do not lose part of the process.”
In addition to evaluating data center requirements, facilities professionals are focusing on how to maintain call centers and after-sales support areas in the event of an emergency. “If these support centers are not running, you can have a data center standing by all you want, but you are not actually helping your customers with what they need,” says Bauer.
One of the barriers to creating an effective business continuity plan is the lack of understanding between departments throughout an organization. “IT professionals tend to talk in bits and bytes and take for granted that everyone understands them. They assume the power will be met from the facilities management side,” says Jeff Ames, director, three-phase UPS product management, Eaton Corp.’s Powerware Division, Raleigh, NC. Powerware products are used worldwide in support of healthcare and government applications, enterprise facilities, fixed-line and wireless communication networks, and industrial manufacturing.
In terms of power distribution within buildings, facilities professionals are often more knowledgeable than their IT counterparts. “Whether you are talking about a large data center or a small installation in an office, these departments need to bridge the communication gap,” says Ames.
A lack of understanding about power quality can lead to a critical failure. For a Web-hosting company, retailer, or travel reservation agency, the costs of downtime can be astronomical.
“It is important for facilities managers and IT professionals to consider the benefit of a high-quality power site audit,” says Ames. Every day, office equipment and building components – such as lighting ballasts, HVAC equipment, and copiers – can create harmonic distortion (see Defining Power Problems), which can damage sensitive computer apparatus. A trained power audit expert can detect distinct power problems, determine the culprits causing dirty power, and offer solutions.
The Story of a Data Center
Employing approximately 55,000 people in more than 30 countries, Capgemini, headquartered in Paris, is a leading provider of consulting, technology, and outsourcing services. Uptime is one of the corporation’s top concerns. The Capgemini IT department in Utrecht operates on an outsourcing model, dealing with strategic tactics and servicing networking equipment.
“The basic idea was to harmonize things in such a fashion so that there was a uniform set-up of server placements,” says Cees J.H. de Kuijer, infrastructure manager, Capgemini, Utrecht, Netherlands. Managing three data centers, de Kuijer oversees the northern European and Pacific regions. Originally, Capgemini had a campus set-up with six facilities. Each building had small computer rooms, and computer services were scattered throughout the campus. The IT department adopted InfraStruXure, an integrated UPS and power distribution solution from American Power Conversion Corp. (APC) of West Kingston, RI.
The new 50,000-square-meter facility in Utrecht features a mod-ular system with standardized server racks that can easily be customized. This system also easily accommodates a variety of servers and various cooling and power requirements.
According to the Santa Fe, NM-based Uptime Institute, 54 percent of all reported site infrastructure failures were attributed to human error. By setting up identical server systems in its Utrecht and Amsterdam facilities, the Capgemini IT department is simplifying the training requirements for its staff members and minimizing the chance of human error. “The one thing we have learned from the Utrecht project is to do the same thing twice,” says de Kuijer.
“In the old situation, there was a sticker on each server that said, ‘This is my server,’ with someone’s phone number on it,” says Roeland Dillen, account manager, American Power Conversion, Utrecht, Netherlands. “We would call the old phone numbers and no one would be there. When knowledge leaves you, you hardly know what is there.” In the past, when IT personnel left the organization, irreplaceable and vital knowledge was also lost.
“The nice thing is that, in Amsterdam, we put in another site for Capgemini and the same people are monitoring both sites. It is not a different environment and you only have to be trained once,” says Dillen. With Capgemini’s new data center, an inventory was completed and a more precise administration of servers and related equipment was implemented.
The APC server structure also allows Capgemini to easily expand, downsize, or move data center locations. With blade server technology, de Kuijer believes data center design is moving toward being more modular.
De Kuijer encourages companies to do a reorientation of the marketplace of technology solutions and to not rely on old habits. “You have to be stronger in your position to say, ‘What you deliver must fit in our racks.’ You should be in the lead, not following your supplier,” says de Kuijer. Many times, organizations do not explore new technologies until after a major power outage has occurred.
“The biggest challenge at this moment is for IT to understand how to deal with building issues, and facilities managers do not understand how IT can help them,” says de Kuijer. Facilities departments are facing increasingly complex systems, as security and building automation systems become more complicated. Facilities professionals could rely on their IT departments as a resource.
Many organizations also have redundant systems; for example, separate security ID cards for general access and access to IT areas. By collaborating, IT and facilities management departments can reduce redundancies. At Capgemini in Utrecht and Amsterdam, the IT department collaborated with the building team in the early stages of the construction of the data centers.
“My biggest recommendation is to include IT professionals from the very start of a renovation project or new construction, because they will be a major part of future decisions,” says de Kuijer. In addition to being more effective, coordination of the IT and facilities management departments can prevent a lot of extra costs during the construction process.
Preparing for the Future
According to the 9/11 Commission Report, secure electric power is a vital component of overall security preparedness. The report describes how the private sector and the majority of the public sector are not fully prepared to manage their critical power.
In response to the need to address power quality and business continuity, some companies are collaborating. Recently, representatives of the critical power industry formed the Critical Power Coalition (CPC), a national organization whose mission is to develop common public policy and establish a unified industry voice to ensure the quality, reliability, and continuity of electrical power within critical industries, businesses, and public services.
Headquartered in Washington, D.C., the CPC, comprised of representatives from vendors and end-users, brings together industry leaders to focus on urgent policy, technology, and regulatory issues at the forefront of today’s electrical society. The group’s founding members include representatives from American Power Conversion, Caterpillar, Cummins, Digital Power Group, Eaton Corp.’s Powerware Division, EMCOR, EnerSys, EYP Mission Critical Facilities, Liebert/Emerson, MGE UPS Systems, Siemens, Square D, and Tishman Technologies.
The coalition’s preliminary initiatives include promoting best-practice sharing in critical power management and raising awareness for the importance of critical power. For more information on the Critical Power Coalition, visit (www.criticalpowercoalition.org). A White Paper, entitled “Critical Power,” is also available on the website that discusses protection of electric power infrastructures and addresses key vulnerabilities and considerations for securing critical power.
“While government and public utility companies continue to work on strengthening and improving the grid, companies and government organizations are tasked with designing and managing critical power in their own operations – inside their facilities – and will ultimately need to develop systems and plans to sustain critical operations when the utility grid is unable to provide clean, uninterrupted power,” states Mark A. Ascolese, co-chairman, Critical Power Coalition.
Regina Raiford Babcock (firstname.lastname@example.org) is senior editor at Buildings magazine.
Are Your Tenants Safe?
Identifying the Planning Team
Some issues to be addressed in developing a comprehensive plan are:
Fire department policies and procedures.
Police department policies and procedures.
Local emergency management provisions.
Government requirements and resources.
Structural, fire protection, mechanical, and electrical engineering.
Building security, maintenance, housekeeping, and occupants.
Types of goods or services provided.
Evaluating Security Threats
There are several factors to consider when evaluating security threats, including building types, geographic area, and tenant mix. Potential impact can also be evaluated by financial risk. Review the following criteria as well:
The sophistication level of potential adversaries.
Short-term replacement of material costs, such as renting and installing equipment.
Long-term replacement costs, such as purchasing and installing new equipment.
Amount of revenue lost during downtime.
Cost to reputation of organization, such as damage caused by missed deadlines or lack of customer service.
Excerpted from Are Your Tenants Safe?, a publication from Building Owners and Managers Association (BOMA) International, headquartered in Washington, D.C.
The facilities management department’s first step is to gather information and to become as knowledgeable about its properties as possible. Gaining in-depth knowledge of the property includes taking an inventory of:
Emergency Procedure Requirements. Consider sprinkler and smoke detector laws; city, municipality, and state requirements; and evacuation drill requirements mandated by governmental agencies.
Traffic Patterns. Heavy traffic can impede emergency crews from responding quickly
Crime Patterns. Assess the crime level of the facility’s location and the appropriate security.
Weather Conditions. How do the types of emergency procedures plans match local weather conditions?
Size of Building.
Staffing. Is the building staffed 24 hours a day?
Tenant Populations. Number of tenants? Number of employees? What sorts of equipment and materials do tenants use on-site? Approximate number of visitors to the facility? Number of outside contractors working in the facilities?
Excerpted from Before Disaster Strikes: Developing an Emergency Procedures Manual, a publication from the Institute of Real Estate Management (IREM), Chicago
Defining Power Problems
Total Power Failure: A total loss of utility power– when voltage falls below 80 volts for several cycles or more – can be caused by a number of events, including lightning strikes, downed power lines, transformer malfunctions, over-demands on the power grid, accidents, weather conditions, and natural disasters. In most systems, a power failure can cause any or all of the following conditions: file/data corruption or loss, hardware or firmware damage, and system lock-ups.
Power Sag: This is a drop of at least 10 percent from normal utility line voltage for a half cycle or more. Power sags are triggered by various load and switching mechanisms in the utility grid, or can be caused by the equipment itself. For example, when an electrical motor gears up to 6,400 rpm, it draws up to nine times its normal load from the power grid, making it difficult to keep voltage within tolerances. It’s not uncommon to have 10, 15, or 20 amps be gulped from the available power when a power-hungry piece of equipment starts up. Depending on the loading of the circuit, this can result in a voltage sag that can affect other equipment on the same or adjacent circuit. Even though sags usually last only a few cycles or seconds, the sudden dip in voltage can cause corruption or loss of data, flickering lights, equipment shut-off, or malfunction with automatic shut-off.
Power Surge: The opposite of a power sag, a power surge is a short-term increase in voltage of 110 percent or more above the nominal supply voltage – often caused by large electrical loads turning on and off in the utility grid. The extra heat from high voltage overwhelms and damages circuitry in digital imaging systems. The system may corrupt or lose data, malfunction, or simply shut off. A short, intensive surge is known as a “spike.” Spikes are most commonly associated with lightning strikes, which can send line voltages above 6,000 volts. But spikes can just as easily happen when back-up generators kick in (a 50-volt spike wouldn’t be unusual), and can even be caused by the equipment itself. Spikes are particularly destructive to electronics because circuits cannot absorb the heat of such high-energy content.
Under-Voltage: Also known as a “brown-out,” an under-voltage condition occurs when voltage drops 10 percent below normal voltage, yet remains above 80 volts (the threshold for power failure). Brown-outs may be intentionally induced by the utility for a few minutes or days to conserve power during periods of peak demand. Brown-outs can cause data loss and corruption, and premature hardware failure when the equipment’s internal power supply component draws more current to make up for the drop in voltage. Since electric motors account for 70 percent of industrial electricity use, it makes sense to be proactive in guarding them against the effects of power interruptions and brown-outs.
Over-Voltage: The reverse of under-voltage, over-voltage is increased voltage for a duration of time. Over-voltage doesn’t happen often, but it can accompany rapid reduction in power loads, shut-off of heavy equipment, or utility switching. Over-voltage can cause extensive hardware damage, including burned-out circuit boards, component stress or loss, memory loss, data loss, and data errors.
Line Noise: Electrical line noise is a high-frequency waveform caused by Radio Frequency Interference (RFI) or Electromagnetic Interference (EMI). These common interferences in power can be generated by local or remote influences – such as transmitters, welding devices, printers, lightning, and electrical equipment. Varying degrees of severity can occur from simple keyboard lock-ups to program failures, data crashes, and data corruption.
Frequency Variation: When viewed with an oscilloscope, normal alternating current (AC) produces uniform waveforms. IT systems, laboratory instruments, and production systems typically expect to receive AC power oscillating at 60 Hertz (Hz). Systems can tolerate slight differences from this specification, but a significant deviation, even briefly, can cause problems. The regular sine-wave frequency pattern of “healthy” AC power can be distorted by fast, transient spikes – often from electronic equipment feeding internal noise back into the power line. Power that is delivered with an irregular and abrasive waveform can cause internal system components to degrade and fail. While components are degrading, system technicians may notice a higher incidence of data errors, system and communication lock-ups, and resets in the equipment.
Switching Transients: These extremely brief periods of under- or over-voltage do their damage in mere nanoseconds, far less time than a sag or spike. Damage may be incurred in both hardware and software, resulting in burned circuitry, component stress or failure, and memory and data losses.
Harmonic Distortion: This is the distortion of the normal line waveform. This condition is generally transmitted by non-linear loads, such as from equipment that places on-and-off demands on the power supply. That means harmonic distortions can be caused by everyday equipment found in any commercial setting, such as switch mode power supplies, variable speed motors and drives, pumps, heaters, robotics, copiers, and fax machines – anything with variable power consumption. Harmonic distortions can cause communication errors, overheating, and hardware damage or premature failure.
Jeff Ames, Director, Three-Phase UPS Product Management, Eaton Corp.’s Powerware Division, headquartered in Raleigh, NC