When the Coronavirus Enters Your Control Room 

In this report, we follow the naming for the virus as SARS-CoV-2, or coronavirus, and the disease as COVID-19.  When control rooms for critical infrastructure were being designed, there were lots of considerations for redundancy, cyber security, and human factors.  Some of the human factors were things like selecting operators based on educational background, security checks, training, and drug testing.  Testing operators for COVID-19 has never been a control room design requirement, and there is a distinct lack of plans in place to handle the situation where a control room operator tests positive for a hazardous contagion.  The implication of a control room infection is that all operators in that control room will need to be quarantined as they were working nearby and share kitchen and bathroom facilities.  Considerable effort is being put into place to respond to this increasing likely scenario.  The specific group of personnel in the central control room is now being kept to minimum essential staff, and how operators are teamed up is being re-examined to insure that when any particular group of operators is moved to quarantine the plant will still have adequate staff to continue operation.  In this article, we review some of the new procedures that are being implement in refinery and power plant control rooms.  Basically, you never want the SARS-CoV-2 virus to enter the control room, and if it did enter, it needs to be quickly detected, and the person bringing the virus in would need to be isolated and all human contact traced to track down other people that have been exposed.  This is the same “Whack-a-Mole” strategy that has proven effective in the general population of South Korea, Singapore, and Hong Kong, but there are many other things that are being done to ensure critical infrastructure remains operational.

Let’s look at some of the control room situations that operate critical infrastructure where the concern of operator health has taken on a new importance.  Consider nuclear power plants, fossil power plants, refineries, chemical plants, grid operations, air traffic control, aerospace mission control, food plants, pharmaceuticals plants, and pulp and paper plants that may have such large control rooms.  Losing these plants could certainly impact the delivery of energy, transportation, and disrupt the supply chain for essential items.

What are these facilities doing to prepare for SARS-CoV-2 virus entering their control rooms?

ARC interviewed several end-users that are operating these plants and asked about what they had planned.  Let’s look at some control room configurations and how some plants are coping.

Several types of control rooms are involved.

Old style panel boards are essentially hardwired with buttons, lights, and panel mounted controllers. These facilities need operators in the control room for all activities and cannot be remotely operated by computer networking.  Sometime, certain functions can be done from the I/O rack room, but plant operation requires operators in the main control room.

Modern Digital Control room using Windows or Unix workstations for the DCS offer some advantages for putting more space between operators.  Many plants have control rooms with backup architecture, like local control rooms with all the functions of the central control room.  Large refineries and chemical plants may have a central control room for the refinery with maybe 6-12 local control rooms, one for each unit operation.  Field operators tend to hang out in these local control rooms, and console and field operators often know each other’s jobs, which provides additional personnel backup.  Most nuclear plants have back up emergency control rooms that could be used to operate the plant.  NUREG-660 requires nuclear plants to have 3 systems: Technical support center (TSC), On-site operational support center (OSC), and Emergency operations facility (EOF).

Although nuclear plants do not allow it for security reasons, the operator control stations can support remote desktop or X-window applications allowing extra seats at remote locations.  Relatively easy to add, a few plants are adding that capability now.  DCS systems can add new operating stations and locate them in an isolated room.  While adding remote stations may be easy, such a sudden change in architecture is not without its problems.  Console operators use portable walkie-talkies, but some functions, like custom alarm annunciator panels, panel mounted lights, and the big red shutdown switches that are hardwired to the console or panel in the central control room, cannot be easily moved into an isolated room.  Nuclear plants do not allow remote desktops and modifying the control system architecture in a running plant can present new risks, so most plants are not relying on this solution.  It could be that such control rooms in the future move away from the large open space design, or maybe they improve spacing and add new purified air flow curtains for each operator.

New and evolving procedures have been created

The federal government considers power plants part of the nation’s critical infrastructure.  The US Department of Homeland Security (DHS) issued guidelines to government and business officials on March 19 about activating measures to protect critical staff members from the coronavirus.

While there is no uniform response at all facilities, here is what we have seen at some sites.  Central control rooms are being locked down.  This means only the console operators are allowed, with no visitors, supervisors, engineers, or maintenance staff.  At many sites, operators work with 4 crews with 12-hour shifts in a rotation.  Nuclear plants have 5 crews with the odd crew in training.  What has changed is there is more resistance for operators to mingle between crews, and in some cases, crews now work for 14-day stints with their normal 12-hour shifts, but still going home each shift.  The control system support personnel are working on 7-day splits with A/B groups alternating weeks in the plant and working from home.  The control group is the only other personnel allowed in the central control room.  Every 12-hour shift change is a new cleaning ritual where keyboard, mice, tables and other equipment and surfaces are sprayed with alcohol solutions.

These operators and the control staff do go home every day and have been given special briefings about how to avoid getting the SARS-CoV-2 virus.  This arrangement is not without some risks.  For example, it is not uncommon for family member to have boyfriend-girlfriend relationships that are hard to keep isolated, thus linking the operator’s family with another family should either get the virus.  In some cases, the operator’s spouse is working in a hospital with virus exposure or maybe the spouse or other family member is not careful.  In one refinery we contacted, contingency training had recently been completed as part of union negotiations in case of a strike.  This means that process and control system engineers recently refreshed their operational training in the event they need to step into the operating role should operators get sick and go into quarantine. ARC has always been a big advocate of using simulators to quickly and effective train operators, although it does take a long time for operators to be ready to run a high consequence process.

Maintenance staff at these refineries are still working full time, but social distancing and new evolving work procedures have been developed for each craft.  Optional field work that requires groups has been delayed.  Essential maintenance is continued.

Another big change that we have all experienced is the huge expansion of web meetings, and all operators and engineers are getting updated on multiple web meeting applications to insure a high level of communications.  The big morning meeting previously held in the central control room is now virtual.  Economics and planning groups and any other staff, that can, are working remotely.   

Several power plants and grid operation centers have been stockpiling supplies, including portable beds and food, to enable staff to remain onsite for weeks if not months.

Grid operators like ISO New England routinely practice abandoning their main control room and operating the grid from a remote-control room, but now those groups do not mingle.  Some of the latest actions they have taken include:

• Limiting control room access only to control room staff
• Splitting system operations shifts between the main control center and the back-up center, minimizing the potential of cross contamination between crews and allowing for more frequent control room cleaning
• Coordinating with transmission companies, as well as other ISOs and RTOs to share best practices and situational awareness
• Communicating with resource owners to understand any challenges to their staffing or business operations

The natural gas industry has particular circumstances to consider, such as the need to enter a home to shut off the gas supply in the event of a leak.  Karen Harbert, president and CEO of the American Gas Association, which represents more than 200 local energy companies that deliver natural gas across the US, in a March 17 news release said, “Natural gas utilities are consistently focused on delivering energy safely and reliably.  Natural gas utilities practice heightened levels of hygiene and protection for employees who may need to enter a customer’s home.”

NASA mission control would have unique problems as there is no backup set of operators during a launch. There are many different control room situations and procedures being tailored to the local conditions, operating modes, and criticality of the facility.

ARC has been working hard to update all market studies to consider the impacts from the coronavirus and has provided webinars and numerous reports on the impacts to various market segments.  See how Smart City technology can help protect people from getting COVID-19 and the impact on global supply chains and securing remote workers in the time of the coronavirus, which are just three examples.