Controller Lost Communication, redundant was unavailable, plant upset five minutes after a landing module DO download

We had an upset in the plant. Here is the sequence:

1. Controller redundancy was between "available" and "unavailable" every 10/15 secs or so (we did not know that until after the incident).

2. We did deltaV landing module download with only 1 or 2 DO points in the module while kept all devices in the card in "manual" mode to avoid issue.

3. Download was successful without giving any error message. DV Operate looked good, no issue.

4. 3/5 mins later operators complains the entire operate is showing "@@@@@" in all places.

5. Technician goes into filed and he finds that controller was blinking red.  He reset the controller.

6. DV Operate comes back to live data but already upset the plant.

7. Event logger shows that after the download the VIM card switch over happened for some reason and the controller lost communication to every peer. and the redundancy did not work.

Any explanation?  Any suggestion will help.

3 Replies

  • Hello,
     
    Please specify below. We had similar issue in past; we found that the older VIM firmware was causing issue with the redundancy switchover after downloads. Issue was fixed after installation of latest VIM firmware.
     

    <![if !supportLists]>1)      <![endif]>Are you using VIM or VIM2 on your system.

    <![if !supportLists]>2)      <![endif]>Do you have redundant VIM setup?

    <![if !supportLists]>3)      <![endif]>What devices are interfacing with the VIM?

    <![if !supportLists]>4)      <![endif]>What is the firmware revision on the VIM cards?

     
    Regards,
     
    Girish Mahajan,
     
  • In reply to girishbm1:

    I think it is VIM2. I am not sure about the firmware version but I am checking on it. We have redundant VIM. The devices are PLCs (Allen Bradley, Modicon, Analyzers, etc.). I will check these. Also, we have VIM nuisance communication and get flurries of alarms for comm failure.
  • If the redundant VIM is normally unavailable it is probably because the Standby VIM ping is failing. This is witnessed by seeing the Active and Standby VIMs switching back and forth. You will see this behavior if a device is powered down or the device doesn't understand the ping request. I know Rockwell PLCs understand default VIM pings and have no issue but sometimes other equipment doesn't understand the MODBUS ping. To test this, try a custom ping. You have to add 10 to the PLC Datatype in the first input dataset of the suspect device. For example if the PLC Datatype is 1, then change it to 10+1=11 or if it is a 3 then 10+3=13. This forces the redundant VIM to issue a custom ping, which is a Read command. So basically the standby VIM tries to read the data and if successful it knows the device is listening. The data itself is then discarded.
    There may be other config issues with the way the devices are redundant.