Training the Field Operator for High Consequence Processes

Operator training for high consequence processes, like those found in nuclear, refining, and chemical industries, has always focused on training the console operators with less emphasis on training the field operators.  These processes can have severe consequences and are controlled using DCS and PLC type controls that are arranged for central and/or local control rooms.  These modern control systems regulate the process and provide the control room operator with key information they need to make sure the process is operated safely, efficiently and according to production plans. Calibrated and sometimes redundant instruments measure the process, and the fault tolerant control systems act on valves, solenoids, pumps and motors often with little human intervention.  

ARC studies show that such highly automated control systems are tremendously effective at increasing the safety and performance of the operating process.  The control system can include sophisticated advanced regulatory controls, alarming systems, supervisory optimization, and when things go wrong ESD (Emergency Shutdown Systems) stop, or at least constrain, the process to insure people, equipment, and product quality are protected.  One might think that such highly automated processes hardly need field operators, only requiring maintenance personnel to keep the process and the automation systems in good working order.  All such high consequence processes utilize field operators, but what do they need to do?  And, how should field be operators trained?

The Fukushima Daiichi tsunami accident was a unique station blackout event presenting unprecedented challenges for the field operators. The control systems at 11 of 55 nuclear plants throughout Japan quickly sensed the earthquake and shutdown their reactors. All the Fukushima Daiichi reactors shut down and waste heat from the decomposing spent fuel in the reactors was removed by electrically powered pumps until the tsunami reached the plant about 41 minutes later. The tsunami flooded the electrical distribution system killing all power to the operating reactor pumps in units 1, 2, and 3, leading to a dreaded station blackout, as battery backup power eventually was used up.

This was a situation where the control system was dark, and the console and field operators teamed up with supervisors, engineers, and managers in a valiant but failed attempt to save the reactors. Post mortem reactor modeling indicted core damage at reactor unit #1 occurred only four hours after the earthquake. This left very little time for the field operators to act. Lessons learned from the Fukushima accident have resulted in modifications to nuclear reactors and operator training worldwide. Field operators at Fukushima were never trained for this type of station blackout situation and the plant infrastructure was not arranged to deal with this situation. Innovative efforts to use firetruck pumps to cool the reactors proved problematic, resulting in disastrous core meltdowns in all three operating reactors.

What role can modern VR and AR technology play in training the field operator?

ARC believes new virtual reality (VR) and augmented reality (AR) technology can play a role in field operator training and operating effectiveness. A combination of factors holds new VR and AR technology back. These barriers are related to cost, effectiveness, and industry requirements. The Nuclear industry is required by ANSI 3.5 standard in the US and similar standards in other countries to provide console operator training using simulators, but there is no such simulation standard for training field operators.  Major refiners use console operator training simulators, but some refineries do not use any simulators for training.

Cost and staff to support VR and AR technologies are major considerations, but there are other factors too. Field operators are expected to intimately know the process equipment and the current situation in the field, and many of these dynamic conditions are not well presented in a virtual 3D simulation like those found in video games. There are sounds vibrations, smells, colors, and a visual memory that gives a human field operator the ability to recognize abnormal conditions that require attention.

New technologies can empower the field operator

If you are going to provide training simulators for field operators it is a good idea to understand what might be asked of them in worst case scenarios and to provide them with expert guidance in highly unusually situations that are beyond those expectations. Simulation-based training with the right abnormal situation scenarios will help the field operator to understand his/her equipment. Experts on process and control systems need to impart understanding of the chemistry, physics, and control system behavior that govern the process operation. This first principle understanding allows operators to extrapolate their training beyond those in the book of SOPs (Standard Operating Procedures).  

In the future ARC sees new and better VR and AR simulation-based training systems for field operators and improved communications utilizing AR enabled interactive headsets replacing the simple push to talk walkie-talkies in common use today.  AR enabled headsets with high resolution camera’s, heads up AR displays, with wireless audio communications can put a real human expert on the shoulders of the field operator. The ability of the console operator to see what the field operator is seeing can result in more effective collaboration. AR enabled field operators may be able to lock out equipment, sign hot work permits for welding, view and interact with control system displays, pull up the spec sheets for instruments and valves, while carrying on a 3-way conversation with local maintenance personnel and the control room operator. It will be interesting to watch which new technologies and products become successful at increasing the effectiveness of field operators. See previous articles on this topic by this author.