Emerson Exchange 365

Loss of Communication between Active Redundant DeltaV OPC server and Triconex

Dear All

Hope you all are doing well.

We have DeltaV redundant OPC servers which communicate via OPC with Triconex Safety System.

We reboot these servers on regular intervals.

We faced a situation where we were re starting our Secondary/Standby DeltaV Redundant OPC server and the current active server communicating with Triconex lost all communication with triconex for about 50 seconds

Has anyone experienced this before?

Any insight on what might have caused it, if someone faced this before?

dikshitsh

• 26 Feb 2021 2:52 PM
• Cancel

10 Replies

• AdrianOffield

  

  • 26 Feb 2021 5:41 PM

  DeltaV OPC is only one side of the communication channel, with OPC Mirror sitting ontop, what are you using to interface to Triconex controllers? Both OPC servers (active and Standby) will need the triconex interface installed. How have you configured that for redundancy purposes?

  • Cancel
• dikshitsh

  

  • 27 Feb 2021 1:22 PM

  In reply to AdrianOffield:

  Hello Adrian, Hope you are doing well. We use matrikon as an Interface, and that is installed on both the primary and the standby servers, and also the other side. We regularly restart the servers with no issues, Only this time while restarting the secondary, even the primary lost comms for about a minute and comms were re established before even the standby came up. Experienced this the first time, so totally clueless on the reasons.

  • Cancel
• AdrianOffield

  

  • 27 Feb 2021 1:51 PM

  In reply to dikshitsh:

  Hi, the redundancy is handled by Mirror so did you see anything in Event log for OPC Mirror redundancy? Mirror has to establish a partnership with the other server so there could be some issue as why the primary dropped.
  
  When you say lost comms, how did that manifest? Pipes go bad, diagnostics indicate integrity error or values go BAD status??? More details about the issue you faced would help to pinpoint things to look at.

  • Cancel
• dikshitsh

  

  • 27 Feb 2021 2:46 PM

  In reply to AdrianOffield:

  There is nothing in the event log of the Primary server, I was not able to check diagnostics or the pipes diagnostics as it was for only about a minute, Have put up a call with Guardian and waiting for the outcome of the investigation, As we got a watchdog so I assume the pipes went bad, though not sure and we also got the input/output transfer errors in all the OPC Modules for about a minute.

  • Cancel
• LaurentB

  

  • 1 Mar 2021 10:15 AM

  In reply to dikshitsh:

  Hi
  I was on the FPSO where you are, and probably I'm the guy who put in place this redundant OPC COMMs :-(
  As you know you have to reset OPC Mirror servers to avoid memory leaks....
  In some case I also observed a freeze of OPC comm , until watchdog and we never found the root cause. That the reasons why we  increased watchdog timer to 60sec ( especially on HULL server) In fact this delay could be increased a little bit more ( I can't explain you why in this topic it's too long)
  I know I don't give you a final solution, but in 8 years past on FPSO, we never found the root cause
  I think it's an issue of comm overloard between Triconex CPU and their OPC Card ( May be during auto diag sequence).
  The triconex OPC Card are not able to refresh data with CPU, that's probably the reason why we see nothing into all DeltaV Logs . If you want to go ahead don't forget to investigate also at Triconex side.
  
  Good day to you and Dario ;-)

  • Cancel
• dikshitsh

  

  • 1 Mar 2021 10:56 AM

  In reply to LaurentB:

  Hello Laurent
  You are right, its the same FPSO, nice connecting with you. I recognize your name now as I came across a few mails of yours previously. This comms issue has been a problem and now become a big issue as the comms failed this time when we restarting the secondary server which was not even active at that time. I also went through previous failure reports and they all said the same thing, the cause of comms fail is unknown. Lets see what comes out this time.

  • Cancel
• LaurentB

  

  • 1 Mar 2021 11:16 AM

  In reply to dikshitsh:

  It's always unconfortable to manage this kind of issue with the production . I just hope to you that it was not an ESD 0.
  Obejctively I think the issue is from Triconex Side, but as it's a "blind" application at firmware level, it's quite impossible to investigate deeper to highlight it....
  Good luck ;-)

  • Cancel
• dikshitsh

  

  • 1 Mar 2021 11:29 AM

  In reply to LaurentB:

  No it was ESD-2 but whole plant tripped as diesel pumps tripped due to some other issues, One other thing I don't understand is why did we use OPC points in Trips and Logic & PID Control. Never seen OPC points being used for things other than monitoring. Anyways thanks for the inputs but even I suspect we stay may not be able to find the root cause but lets see

  • Cancel
• LaurentB

  

  • 1 Mar 2021 12:06 PM

  In reply to dikshitsh:

  I didn't participate to the hazops and project at upstream phases, but it was probably considered as not safe to operate the plant from DCS without capability of monitoring Safety system ( there was no powerfull hmi to minitor safety part) . This is why i consider we can increase the timer little bit (This is only my self point of view, may be I don't have all data in my hands)
  As always the project Engineers had to manage the paradoxal situation between Safety and availability and they go to the best way : Safety.

  • Cancel
• Tadeu Batista

  

  • 3 Mar 2021 5:33 PM

  In reply to LaurentB:

  dikshitsh and LaurentB , I was reviewing the back and forth conversation, it strikes me odd that somebody would start a trip based on HMI loss of comms alone, you still have the DCS variables and I believe the SIS Engineering station as a backup, if an unsafe situation is identified, a physical Shutdown button should be available. As we know, a plant is stead state is much safer than a plant in transition state.
  
  The other point is the OPC trip signals, arguably, they would be hard-wired.

  • Cancel