Emerson Exchange 365

• Not Answered

Redundant controller and switchover

We converted some simplex MX controllers to redundant MX controller. After that we did a little test and forced a switchover of the active controller. We did not have any issues with the associated modules inside this switching controller, but we having an interruption in historical data archiving. There was a gap of 3-4 seconds in our trends (no data / comm fail). Our Historian is an OSI Pi Enterprise server talking to an DV application station acting as OPC server. So it seems that during switchover the OPC server can't talk to the effected controller and his modules and is reporting the values as bad.
Emerson is saying in their whitepaper '..the process continues as though nothing had happened..'
What does this issue mean to e.g.
1. Modules with controller to controller communication during a switchover. - Other controller Module is read or writing to the switching controller.
2. Control Loops / valves / drives inside this redundant and switching controller.
3. Running Batches (Phases - with the default failure monitor SWITCHED_OVER parameter used.)

• Phase Failure Monitor:  In all of our Failure Monitors the SWITCHED_OVER Parameter used to send the Phase to Hold. I am thinking about to suggest taking out this  SWITCHED_OVER condition from all Failure Monitors, because the expectation with Redundant controllers was that an Phase (and Batch) continues as though nothing had happened in case of a controller failure and switchover.  But even an standard Emerson Phase Failure Monitor is coming by default with the SWITCHED_OVER (and watchdog) condition for Hold.

- Any experience/recomodations or thoughts about Phases and redundant Controllers in case of switchover?

 

Regards Martin

• Redundant controller switchover

Martin Mueller

• 29 Jul 2016 10:59 AM
• Cancel

10 Replies

• Tinh Phan

  

  • 29 Jul 2016 2:47 PM

  Have you flash update the MX controller firmware match and licensed the controllers to be redundant? What version of DeltaV are you running on? I have ran into this situation before where I received a new controller with the factory fw running at a new version than my DeltaV. I had to downgrade the new controller to the DeltaV version I had for it to work properly or vic versa. This fixed my issue. You may want to check to see if your DeltaV system is missing any hotfix patches related to switchover. Keep us in the loop what you find.

  • Cancel
• Mike Link

  

  • 29 Jul 2016 3:18 PM

  Books online has a great article about this, search for the article named "Controller Redundancy"
  
  Logic contained within that controller will switchover without issue. Hart values will be lost for ~6 seconds while the "new" controller reinitialize the Hart modems. External communication eg. external references, historian etc. can take a few seconds for everything to resubscribe for updates. During this time the status goes to a bad no comm, and the DeltaV status handling takes care of it gracefully, depending on your configuration, but usually just holds last value until communication is reestablished. The HMI / historian are lower priority tasks for the controller so it is expected that they could lose a few seconds of data on a heavily loaded controller. We do switchovers every now and then in the continuous plants without issue, usually for controller replacement (SD+ to SX), controller hotfix installation, and DeltaV major version upgrades.
  
  Phases and batch logic used to have some issues during a switchover (pre DeltaV version 8 if I recall) so going to hold ensured that an operator could verify everything had switched over ok. In the recent versions this is more of a DeltaV doing the safest thing possible. It is very unlikely something would go wrong during a switchover but best not to tempt the fates. Emerson usually recommends that if you want phases to automatically continue, have your holding logic set AUTO_RESTART to TRUE when a switchover is detected, this way your phase will hold, then automatically start the restart logic without operator intervention.

  • Cancel
• Mike Link

  

  • 29 Jul 2016 3:25 PM

  In reply to Mike Link:

  Also, recommended that you run the DeltaV Controller Redundancy Analyzer which can help identify configuration that could cause issues during a switchover. it is available on DeltaV Disk 2/MigrationUItilities/UpgradeWizard/ ACRViewer.exe, ACRViewer.chm (the help file), and ACRUtil.exe. This is a standalone utility that just requires an FHX. More information in Emerson KBA AP-0800-0141

  • Cancel
• MichaelP

  

  • 16 Aug 2016 12:22 PM

  In reply to Mike Link:

  When a controller switches over a gap is visible in recorded data.
  Posted at the same time to the event log is an Event stating "Switchover occured" . The event is posted with Parameter = REDU from Area_A.
  Capturing the event and configuring an alarm for the relevant "area" (e.g. BUFFER) would, at least, alert the BUFFER team to a gap in records.
  There are several possible suspects mentioned in Books online (e.g. PExist, PAvail, RedEnb) but these read different to a "SwitchOver occured".
  Keeping the controller in AREA_A is reasonable for other reasons.
  The trick here is to find the the source flag / event "Switchover occured".
  
  Any folks here know how to capture the "Switchover occured" ?

  • Cancel
• MichaelP

  

  • 16 Aug 2016 12:24 PM

  In reply to MichaelP:

  

  Screen capture of "Switchover occured" event.

  • Cancel
• Andre Dicaire

  

  • 16 Aug 2016 4:43 PM

  In reply to MichaelP:

  I'm not seeing the value of generating an alarm. There is no Operator Action or time to respond that makes this worthy of an alarm. Since you indicate the Gap in data is to alert the BUFFER Team, and again, there is nothing they can do, the alarm would appear as an Event and since you already have the "Switchover Occurred" even logged, why do you need to log an alarm?
  
  First, switch overs are few and far between, and there is no user action to take with respect to historical data. A report based on the Event Chronicle can provide insight to any such events, which the BUFFER Team might use. Given the data gap is several seconds, it would be interesting to know exactly what this team is expected to do with an alert to a switchover. I'm curious.
  
  Please let us know what version of DeltaV. There have been enhancements in v12 to the behavior of remote parameters on a switchover.

  Andre Dicaire

  • Cancel
• MichaelP

  

  • 18 Aug 2016 8:39 AM

  In reply to Andre Dicaire:

  DeltaV 12.3
  
  It is necessary to bring the occurance of a gap in records (due to a switchover) to the attention of the owners of the area.
  Alarms are monitored by the relevant area teams. Events not so much (in comparison).
  By bringing the switchover "event" to an "alarm" and posting this to the area of concern the area team are aware. This is important.
  The area team are obligated to follow SOP and reconcile any gap in records to the actual.
  There is no "reponse" to the switchover itself.
  The response is to the gap in records due to a switchover.

  • Cancel
• MichaelP

  

  • 18 Aug 2016 11:27 AM

  In reply to MichaelP:

  solution : sys stat function to detect switch over

  • Cancel
• Summer

  

  • 4 May 2020 6:59 PM

  In reply to MichaelP:

  Thank you ...the above information was helpful for me to find when the controller "Switchover" happened.

  • Cancel
• Summer

  

  • 4 May 2020 7:00 PM

  In reply to MichaelP:

  Thank you...this screen capture helped in narrowing my search for controller "Switchover".

  • Cancel