Tuesday, March 10, 2009

Video Process Monitoring

Video Process Monitoring

By Steve Rubin, President & CEO
Longwatch, Inc.

There are three primary ways to enjoy a baseball game: view it in person, watch it on TV or listen to it on the radio. We’ve probably all used each of these methods at one time or another; some fans even combine a couple of methods such as bringing a radio to the game to hear the announcers describe what they are seeing. Maybe they don’t trust their own interpretation of what they see, and need to have it explained. That’s like watching readouts on an HMI screen—what’s really happening out there? Why not cut to the chase and see what’s going on?

Today, there are two primary ways to monitor a process: in person, by walking around the plant, or from a control room via an HMI screen. But there is a third way: Watching the process via camera monitors that put images directly on the HMI screen (Figure 1) or onto a cell phone or PDA. That way, you don’t have to wonder what’s happening at the process unit. You can see it.

Camera monitoring of process control is relatively rare. Granted, there are limited applications, such as a flare stack camera or a camera in the shipping area. Why hasn’t video been applied more widely in process monitoring applications and what additional value could be delivered?

A camera image on an HMI screen can be used to verify that certain operations are actually being performed—such as an operator adding ingredients to a batch reactor. The video “snapshot” can be stored on disk, along with batch production data, as a visual record . Like a baseball game, it’s an instant replay. Unlike a baseball game, you can watch it as many times as you want.

In water and wastewater treatment plants, where tanks, lagoons, pumping stations and other equipment is spread over a wide area, camera images can make sure that valves open and close, lagoons are at the right level , and a child hasn’t fallen into the lagoon. The same stream of video has multiple purposes, much like a multi-variable sensor. Video not only provides process information, it can mitigate liability and provide security. With video, you know whether to take a shotgun or a wrench to the field to fix a stuck valve.

A camera image can be used to help diagnose problems in the field. Wouldn’t it be nice to be able to actually see what the process is doing without having to walk out to a distillation column in a Texas summer? I’ve been to plenty of plants where the sensors have exhibited problems which confounded the operators. Open circuits on thermocouples, valves that stick, pressure transmitters that fail, and so on. In some instances, the engineers solve this problem by putting extra code in the control system, or install extra sensors, so that problems can be diagnosed remotely. Imagine how easy it would be to simply look at a camera image of the valve while it was operating, to see if it was sticking.

For example, I’ve seen a situation where instrumentation told the operator that pumps were off-loading oil from a barge, only to find out later that the coupling was off and the oil didn’t make it into the plant…it made it into the river! Another situation involved a cryogenic pump that froze up on a humid day and didn’t work properly. These are situations that an operator could see and understand what was happening, if he or she had video.

Watching Instead of Visualizing
Whenever I’m visiting a plant, I’m usually given a tour of the facility, shown the process equipment and the instrumentation, and then brought to the control room. There, the operators show me the HMI displays that are designed to mimic the layout, behavior and status of the equipment. It’s up to the operator to imagine, in his or her mind’s eye, what’s actually happening in the plant, given the indications from the instrumentation.

This is easier with today’s HMIs than with the traditional dial, chart recorder and annunciators of 10-20 years ago, but this method relies on the accuracy and timeliness of the instrumentation readings, as well as the fidelity of the HMI’s mimic display.

Some process measurements are easier to see than they are to instrument (the flare camera comes to mind…a quick look tells you that it’s lit and if it is smoking). And a camera image can tell you if a process vessel is overflowing, a line is leaking, or if steam is escaping, Figure 2. After all, “a picture’s worth a thousand words.”

“Attention by exception” is really how most process plants operate. We periodically observe indications, but only take action when an alarm condition occurs. The alarm may be an “alert” about an event, or it may be an actual transition requiring remedial action.

If a video image of the process unit appeared on the screen when an exception occurred, the operator would be able to immediately see what was happening. This method of operation is much better than the standard “surveillance” approach of watching closed-circuit TV monitors, waiting to pick up visual cues to changes. Studies have shown that humans will lose interest and be unable to detect changes shown on screens if they are forced to stare for more than 20 minutes.

How many times have you needed to “babysit” an intermittent problem in the field or the plant? Video is a tool that can significantly boost efficiency. A video system can be configured to watch equipment and take video snapshots when certain conditions occur—such as that intermittent problem.

With proper configuration, a “before” and “after” clip can be generated to help troubleshoot the cause of the problem. In other words, the camera watches the process continuously, and stores video images internally; when an alarm condition occurs, it can transmit a video snapshot of the unit for, say, five minutes before the alarm occurred, and then continue transmitting real-time images for as long as necessary. All the images can be stored for playback and analyzed as many times as necessary.

Transmitting Data to the Control System
If video can help us discern more about the real conditions, help us see into remote areas, and bring us more information faster…then why has video been so slow to be adopted in process applications? Until recently, all camera technology was analog, and thus it required its own network for operation. Many traditional video systems were originally designed for casinos, parking garages and shopping malls. It was extremely difficult to connect these “closed circuit TVs” to an HIM screen.

But many plants have a much more suitable network, one that’s almost hack-proof, paid for, and reliable: the instrumentation network for the SCADA/HMI system, Figure 3. This network was designed to handle digital communications between controllers and computers, and it can easily accommodate video images, too.

Some users are reluctant to use the SCADA (or “level one”) network for anything other than process control communications. In some respects, that makes sense: when there’s a plant upset, you want to make sure that control messages (for example: turn off a motor) are delivered on-time and reliably. But there are ways to embed chunks of video clips into “envelopes” that simply pass through the SCADA network. These messages travel at a lower priority than the control messages, so that it might take longer for the entire video clip to show up at the HMI. But it’s better than seeing nothing, and better than driving to a remote process unit.

New camera systems use the bandwidth available in most industrial networks to report video information to the host system, handle real-time diagnostics and the like. It’s easy to “drop” cameras on these networks, especially if two networks are being used in the plant: one for instrumentation/control and the other for operations data/information .

While digital cameras will quickly saturate most networks, proper software provides a means of controlling the network traffic, in an orderly fashion, so that video and control messages can co-exist.

Adding a video camera to most industrial networks simply requires a local “Ethernet port” or a drop-in communications module, similar to adding another smart sensor. The camera plugs in and can be configured as just another sensor. However, configuration and the integration with a SCADA/HMI system can be tricky if you don’t have good application software.

Dealing With Data
Digital cameras can become the electronic equivalent of purple loosestrife (prevalent in the Charles River basin around Boston): Nice to look at, but overloads the environment quickly. In fact, just four cameras transmitting images could use 55MB of bandwidth every minute! Storing video from twelve megapixel cameras for 10 days would take 2.4 terabytes of storage.

How can we deal with the issues of:

  • Mitigating impact and traffic on the network,
  • Deterministic response of the network,
  • Reasonable storage requirements and fast access to video-of-interest, and
  • Integrating easily with the operator’s standard console

    in a way that’s consistent with good engineering and operating practices for industries?

    The solution lies with software specifically designed for video applications in factory automation. Essentially, the software asks for video images to be transmitted over the factory network only when it needs the data—such as when an alarm occurs or a step occurs in a batch procedure.

    The software converts a computer into a comprehensive digital video recorder (DVR) that:
  • Continuously collects and archives high-resolution video from a multitude of cameras
  • Automatically edits the video stream into before-and-after “clips” when an “event” occurs
  • Sends that video clip, along with an alarm message, to the HMI or SCADA system for further handling by the user
  • Automatically latches the alarm so that the user is always notified (and must acknowledge) and no alarms are lost
  • Automatically stores and forwards messages if the network is temporarily unavailable
  • Communicates efficiently over almost any network provided, from gigabit fiber networks, to mid-speed wireless and wired systems, all the way down to 9600 baud wireless and telephone lines.

    Newer industrial video monitoring systems have all these features. Installation is simple, requiring only plug-in connections to the existing network. The software to process video data can be embedded in most commercial HMI/SCADA software systems, or interfaced via OPC. And the commands to take video snapshots can be defined by ISA-88 recipes, or sent by any major distributed control system, at various stages in a batch or continuous process, or in response to an alarm.

    A key need is to edit the videos down to the clips that are important, and store those compressed clips in a way that makes them easy to retrieve. A relational database serves this need well, especially in manufacturing and process control applications.

    How do you interface to the HMI? Fortunately, the industry has developed standards such as OPC, HTTP and even Modbus. These enable application programs to share data structures and pass commands between applications. In the case of Longwatch, we use OPC so that the operator can, from an HMI (like InTouch or iFIX) send a command to acknowledge a video alarm, create an event clip “on the fly,” or even go into “live streaming” mode to take an immediate look into the field.

    Video has the opportunity to be applied to process control applications in ways that were just not possible a few years ago. Better yet, there is technology that enables video to be transmitted over very long distances, on existing networks, so that “blind spots” in the operation are eliminated at very low cost.

    Video images can also be transmitted to cell phones or PDAs, so that engineers can diagnose problems from home in the middle of the night. Or an engineer can stand next to the process unit, watch the recorded video snapshot of a problem, and try to figure out what really happened . Because the video data is available as an historical record, images and process data can be sent to an outside expert for an analysis.

    Having video available on HMI screens helps operators see what’s going on at a process unit, verifies and provides a record that events occurred, and helps operators and engineers diagnose problems from afar.


    Figure 1: Video images on the HMI screen at the Littleton water department in Littleton, MA, allow operators to see what’s happening in the well house and surrounding area.

    Figure 2: An operator at the water plant in Madison, MA, can quickly glance at video images from important areas of the process.

    Figure 3: Diagram shows how cameras transmit data over standard industrial networks to a video processor. The video processor stores the data in a relational database and makes images available to an HMI/SCADA system, DCS, PLC, process historians, web browsers, cellphones and PDAs.

    About Longwatch
    Longwatch, Inc. was founded by industrial automation and software veterans with the goal of simplifying video delivery over existing SCADA, HMI and distributed control networks. The result is the Longwatch Surveillance System™, a portfolio of products that enables SCADA system users to view events and easily verify alarms at local and remote sites using both legacy and new networking infrastructures. The system integrates video and system alarms on the same display for fast, reliable operation and decision-making. Visit the website