Notes for FSP0020 – Power Loss Checklist
We occasionally lose power at our buildings. Sometimes for hours, sometimes for just a few minutes, and sometimes for less than a second. People are often alarmed or annoyed by this, and when the power comes back on it’s back to business as usual. Or so it appears, but maybe not. There are often things that happen, even in the briefest sub-second power disturbance that won’t be apparent for hours or even days. The idea here is that if we recognize what those things might be, we can check on them when power is restored and prevent the bigger problems.
This is something I started doing a long time ago, and it’s worked out pretty well. Obviously it needs to be adapted to your specific circumstances. I’m just going to kind of dump out my thoughts on the topic and hopefully that will be useful to help you refine your own thinking.
Before the power goes out
First, I think I want to talk about a few things to think about before the power goes out. These are lighting, door security, and ventilation. This isn’t meant to be comprehensive, it’s just my ideas, so again, extrapolate this into your own circumstances.
-
Lighting – Life safety codes and regulations require us to provide adequate lighting of the means of egress (that’s from NFPA 101 in the USA). If your building hasn’t been changed much from its original design and you’ve been keeping up on your battery maintenance, this may be enough. However, there are some areas that aren’t really covered by the egress lighting codes and also there might have been some changes made to the building that didn’t properly take power loss lighting into account. As you walk around your building, think about where people might be when the power goes out and what they might need to do in that kind of situation. Most people will probably be comfortable enough in their office or normal work space. What about restrooms, storage rooms, elevators, walk-in refrigeration, etc. Do these areas have adequate battery-backed lighting. Even where battery-backed lighting isn’t strictly required, it might still serve the building occupants to provide it. With many LED lighting options which require smaller batteries, this is more affordable than it’s ever been.
-
Door security – you have things like magnetic hold open devices or magnetic locking devices or request to exit devices that operate some type of electric locking mechanism or any of a number of other similar types of devices. These devices rely on power for their operation. Just like the lighting, if your building was designed by professional door security designers and you haven’t made any changes, everything probably works as you would want it to. In real life, though, circumstances change and usage patterns reveal issues that cause us to add or remove or change devices without maybe thinking about the unintended consequences of these changes. So, as you are walking around your buildings, think about how the door locking mechanisms will act without power. Will they fail secure or will they fail open and does this match your desired operation.
-
Ventilation – In many cases ventilation is a life safety issue. Sometimes we use ventilation to create negative pressure for containment of potentially hazardous materials or to prevent buildup of toxic or flammable chemicals. Probably if you operate a facility that includes this type of functionality you already have procedures in place to protect personnel in the case of power loss. In any case, it doesn’t hurt to think about this and make sure the procedures are adequate and properly being followed.
While the power is out
-
When the power is out, it usually just becomes a waiting game. Your staff may just be sitting around waiting for the power to return to then react to whatever problems may arise.
-
If you are prepared ahead of time, you can potentially use that unexpected, inconvenient power loss event as a convenient time to do “power loss” testing and maintenance. So when the power goes out, start your stopwatch and get to work.
-
A few things (off the top of my head) that fit in this category:
-
First one: Lighting (mentioned before). According to NFPA 101 You should be testing your emergency lighting for 30 seconds every month and for 90 minutes once a year.. IF that outage falls within your maintenance window, you can potentially use the unplanned downtime to do the planned maintenance of evaluating the condition of the emergency lighting.
-
Second one: In many cases we have electric locks that are supposed to do something in particular when the power is out. Maybe they are supposed to be battery powered and continue to function normally, or maybe they should fail-safe (as in unlocked) or maybe they should fail-secure. I mentioned this a little before. When the power goes out, you can check these things. You can swipe the cards and press the buttons and whatever and make sure everything works as you expect.
-
UPS Systems: Use the time to check whether UPS (uninterruptible power supplies) systems are working as intended. That saves from having to do a simulated power outage to test proper UPS functionality. Sometimes simulating the power loss condition can cause problems you don’t want to have to deal with (especially if the UPS fails), but it the power loss isn’t simulated, there’s no extra loss from evaluating the UPS system under the power loss condition. If something bad was going to happen, it happened. You can then record the result of the “test” and not have to do the simulation later.
-
-
I know I have some systems that I just can’t take down during normal operation. The business wants them running with only very little down time for maintenance (or no downtime for maintenance if they could really have their way). So, in general, look for ways to use these unexpected power loss events as opportunities to do work that’s hard to do under normal circumstances. Now obviously this can be tricky because maybe it’s hard to tell how long the power will be out, but it’s worth thinking about.
After the power comes back on (which is kind of really where I started with this topic)
We wish that everything would automatically go back to normal, sometimes it does, but often it doesn’t.
Sometimes fuses blow, breakers trip, controllers and computers hang or drop their settings or go back to some default, batteries get damaged, timers lose time. Without a plan, we often won’t find out about these things until much later when it’s a bigger problem.
A few examples to get the idea across.
-
Controller weirdness: RTU that would sometimes come back up in unoccupied mode. The power would go out, and when it came back on the unit wasn’t working properly. Not always, not often, but sometimes. The first time this happened, we get a call hours after the power event informing us that the space is too warm. Everything on the unit looks weird, all the setpoints are wrong. It wasn’t immediately apparent what was happening because the building is occupied 24/7 so we never runt that unit in unoccupied mode. After that, we made sure to check that issue each time we had a power event until the problem was ultimately resolved, I think with an update to its program.
-
Three phase things: Sometimes, on a three-phase system, you only lose one phase. In this case a three phase motor or pump will try to pull more current from the other 2 phases which might cause an over-current condition resulting in blown fuses or tripped breakers. Dropping a phase for even a second while a motor is running heavily loaded can cause this kind of thing to happen which will shut down the motor. I can give a few examples of this:
-
I have a reverse osmosis water purification system feeding a deionized water system for a lab. The reverse osmosis process has a three phase motor. If that motor isn’t able to run, there will eventually be an expensive and potentially mission-interrupting cascade of failures. “Eventually” could mean many hours later at obviously the most inconvenient time. In my case, there was no monitoring on that particular motor, so a simple power event that caused no apparent problem at the time, but since the power event occurred while the pump was running, the overload device feeding the pump tripped which ended up causing a big problem later. That particular problem didn’t happen again, because we made sure to check on the pump everything time there was a power event until we were able to set up proper monitoring for the pump.
-
Another example was with a VFD on a roof top air conditioning unit. A phase loss caused an overload condition in the VFD so the VFD disconnected the blower motor. This VFD is not original to the unit, it’s a retrofit, so it’s not fully integrated into the unit’s control system. So, the unit kept running without complaining even though no air was moving through it. Hours later of course, we get the call, discover that problem and add it to the list.
-
Pretty much every device running on three-phase power is susceptible to this kind of thing. Think about HVAC devices, or water pump or lift station pumps and make sure to check on them after a power event.
-
-
Mechanical timers – Have you ever gotten a call an hour after dark that the lights in the parking lot aren’t on yet and Nancy from the call center hurt her knee because she tripped on a speed bump and this is a safety issue and why are you so bad at your job? And then you remember that the power went out for 2 hours this morning and those lights are on an electro-mechanical timer that only advances when it is powered…and is now 2 hours behind. I have. And of course I could have known that was going to happen if I had thought about it. And now that we have thought about it and we know, we can prompt ourselves to pay attention to that sort of thing when we lose power.
Implementation of this idea:
-
Make the lists. What are the things you should check on? What things have you had trouble with in the past? What 3-phase devices do you have? What is designed to run on batteries? What things need power to keep proper time? What things might cause big problems “some time later” but nobody will notice any problem in the short term?
-
Create maintenance procedures for evaluating the lists.
-
Create work orders to evaluate the lists when the power goes out.
-
For bonus points, use the power loss event notification from your building monitoring system as a trigger to automatically create the work order. Also, for the maintenance or evaluation items you can’t really do for whatever reason under normal circumstance but you might be able to exploit a power loss event to get done, you can potentially have those triggered by the power loss event also, with completion obviously subject to the exact circumstance of the event
-
Finally, for repeat offenders, what can you do to eliminate the problem? Maybe you can update the controls, or optimize monitoring or replace with a more robust system. Labor and down time are expensive. Updated controls or monitoring devices may be cheaper in comparison.