Emerson Exchange 365

CIOC Redundancy not ensured

Hello! 

I hope I am in the correct thread for this question. I have recently realized that I have a question mark in the Redundancy category for our CIOC-1. I have tried swapping out the "standby" CIOC to prevent process interruptions with another known working one. I continue to receive the same issue, and i'm scratching my head as to what the problem may be.

I am presented the following in Delta V Diagnostics.

Description: Standby Value: Bad

Description: Overall Integrity Value: Bad

Description: Software Revision Value: 11.3.1.4256.xr

Description: Partner Exists Value: Yes

Description: Partner is Available Value: NO

Description: Redundancy Enabled Value: YES

Description: Status Value: Standby not Available

Dodgethis

• 18 Jan 2017 12:27 PM
• Cancel

8 Replies

• Rachelle McWright

  

  • 18 Jan 2017 2:39 PM

  Hello Dodgethis, I am going to move this to the DeltaV group where you will likely get a quicker response. :)

  Best Regards,

  Rachelle McWright: Business Development Manager, Dynamic Simulation: U.S. Gulf Coast

  • Cancel
• Jason.Brumfield

  

  • 31 Jan 2017 12:39 PM

  When you say you swapped out the stand by CIOC, I am assuming you mean with another CIOC card. Did you commission the new card? I would start by re-installing the original CIOC card. Normally I can get the redundancy to reset simply by re-seating it. Do you have any lights flashing on the CIOC cards? If you changed the CIOC card out with a new one, do you have any hardware showing up to be decommissioned?

  • Cancel
• Dodgethis

  

  • 8 Feb 2017 11:33 AM

  In reply to Jason.Brumfield:

  Jason,
  
  Thank you for your reply. At the time when I was troubleshooting, I took another CIOC, and swapped it with the stand by CIOC. I didn't see anything show up in Delta V Explorer that mentioned "Commission CIOC". I did put the original stand by CIOC back in its original spot. I have tried reseating the stand by CIOC in hopes it would fix the problem. When I did reseat the original standby CIOC the lights did show their boot up cycle, but redundancy never came back.

  • Cancel
• Jason.Brumfield

  

  • 8 Feb 2017 3:54 PM

  In reply to Dodgethis:

  So now do you have red flashing lights on the standby CIOC card?

  Are the CIOC cards the same software revision?

  Are they the same Hardware revision?  Not sure if this matters but worth a look.

  Do you have Node is redundant checked on the properties for the CIOC card?

  When you drill down in diagnostics can you telnet the standby CIOC?

  Just going thru my troubleshooting thought process to see if we can identify any indicators.

  • Cancel
• scdrummer

  

  • 8 Feb 2017 5:04 PM

  
  

  
  

  Change the two cards from left slot to the right and from the right slot to the left one. So they become new IP-adresses and you should find both.
  
  Von meinem iPad gesendet

  
  Am 08.02.2017 um 17:57 schrieb Jason.Brumfield <bounce-JasonBrumfield@emersonexchange365.com>:
  
  

  Update from Emerson Exchange 365 Jason.Brumfield So now do you have red flashing lights on the standby CIOC card? Are the CIOC cards the same software revision? Are they the same Hardware revision?  Not sure if this matters but worth a look. Do you have Node is redundant checked on the properties for the CIOC card? When you drill down in diagnostics can you telnet the standby CIOC? Just going thru my troubleshooting thought process to see if we can identify any indicators. View online   You received this notification because you subscribed to the forum.  To unsubscribe from only this thread, go here. Flag this post as spam/abuse.

  • Cancel
• Dodgethis

  

  • 9 Feb 2017 12:20 PM

  In reply to Jason.Brumfield:

  Jason, Thanks again for your thoughts to consider. I'll Report back when I have the answers to your questions.
  
  Scdrummer, That is a thought to consider, however as you might imagine we are a 24/7 running plant. I would only be able to do that during a down day.

  • Cancel
• Peterlx01

  

  • 14 Feb 2017 3:37 PM

  I don't know if this is related but we had problems with our SX controllers losing their redundancy.
  Same diagnostic messages as your cioc's.
  Only running on one and not being able to download anything anymore created some production/safety issues as you can imagine.
  
  We ended up finding multiple problems that needed fixing including:
  
  Removing empty strings from the config file <- big issue
  Removing logic from interlocks writing to external references <- was more an effect than a cause
  Mailing the controller to Emerson which lead to loading hotfixes on all other controllers <- big issue

  • Cancel
• Andre Dicaire

  

  • 14 Feb 2017 6:42 PM

  In reply to Peterlx01:

  The right thread for this type of question is to log a call with the GSC. They will take you through an orderly process to identify the issue, and can cross reference your situation with known issues. IF this is a new issue, they need to be involved so that it can be properly escalated.
  
  Why you shouldn't use this forum? You get unrelated suggestions that are well meaning, but not productive, and get you chasing ghosts. v11.3.1 has a myriad of hotfixes for the controllers and CIOC's, so the first thing that needs to be defined is what versions of Software do you have running on the controllers and CIOC's. and specifically, are the Primary and Secondary cards at the same revision, and is this the latest revision.
  
  As for commissioning, when you replace the standby card in a redundant pair of controllers or IO cards, the active card informs the pro plus that the standby card has been changed, and the card is automatically commissioned to the standby address. It is normal not to see the new card as a decommissioned node. During initial boot, sometimes you might see a simplex card from a redundant pair show up briefly in decommissioned Nodes when you insert the first card and then the second. But in this case, the active card remains on, and handles the commissioning of the new standby.
  
  The reason the card is Not Available, is that the Standby is not being configured, or the download is somehow being rejected. Without a configuration the card cannot go Active. That is all we know. If the active card has a newer firmware rev, sometimes the standby is not allowed to go active until it at least matches the active card. Hence, we need to make sure we check firmware rev on all cards involved, and that it is the latest.
  
  What if there is an intermittent issue on the network that allows normal traffic to generally flow, but the download file is being corrupted? Or does the standby CIOC always get configured from the Active, which means the network is not in play? I'm not sure about that. Does the Standby CIOC Memory diagnostic or CPU diagnostic show any sign that the configuration is being loaded. The Standby Free Memory should be close to the Active Free memory. If you look at a decommissioned CIOC you'll see the max Free Memory. Is it moving, like it reduces then jumps back to max. You can trend this. This would indicate it is processing the download, then discarding and not completing. That might help isolate the problem, but unless someone on this thread has experience with that symptom, it won't solve your issue.
  
  Are there any communication faults seen in diagnostics? you can compare good CIOC nodes with this one to see if there are any error counters showing activity (changing). Note that a counter may have a value, but if it is not changing, it is not indicating a current issue. On a healthy connection, the error counters are not incrementing. The Multicast and Broadcast counts may increase, but slowly. But the best thing is to discuss this with the GSC so you don't misinterpret their meaning.
  
  I could be a hardware issue. Might be the Active card has an issue with the redundancy link. Is the CIOC overloaded? not likely but easy enough to check. If you have 96 HART enabled charms with CIOC at 50 ms update, is it possible it doesn't have enough CPU to establish the redundancy link? If CPU is down to 0% free, change the CIOC update rate and increase this to next level. I've not seen this happen, but it is easy enough to check.
  
  But again the best place to chase this down to a resolution is with the GSC. Because if this is a new issue, that is the path to escalate it to the next level and get the attention it deserves. If they already know what this is, I've wasted more of your time with this speculation. :-)

  Andre Dicaire

  • Cancel