Recent Control & Safety Systems Discussions
Similar Posts
Redundant controller and switchover
DeltaV Batch Executive Redundancy Switchover
Force Redundancy Switchover Command within DeltaV Operate
CIOC Redundancy not ensured
Importance of deltav patches, Redundancy problems for controllers, cioc and Batch Executive
Share

Redundant CIOC and EIOC Behavior on Switchover

DeltaV S Series SXPVX Controller Model,V13.3.1- Modules installed in redundant carrier.  CIOC points are only produced and consumed with one controller.

Can someone describe how the CIOC and EIOC redundant systems behave upon a switchover triggered by EIOC/CIOC fault?   I suspect there are two questions here possibly with different answers.

Do I lose communication with my I/O network and/or devices during the switch and while the backup comes online? 

Is it bump less like a controller switchover or is there a gap in comms?

I believe I have recently seen an EIOC switch. During that switch I lost comms with my network which caused control modules to go bad, since it couldn't connect to PLC I/O, and eventually took down my process.  I think that is what I saw in the event historian but wanted to confirm.

It triggered a thought about the CIOCs also since we are currently installing several(>10) redundant CIOC setups on a current project.

Thanks!

Scott Brown

s_brwn

8 Replies

  • Andre Dicaire
    First, the CIOC and EIOC are completely different devices. The CIOC is a native IO subsystem to the controller and communicates on the DeltaV network with a proprietary protocol that manages the health of both network connections as well as the CIOC itself. Control modules are in controllers and not in the IO Node.

    The EIOC is positioned as an Ethernet IO device, but it is more like a controller. It hosts Control modules that consume the Ethernet IO Signals configured in the PDT/LDT communication data structures. This IO communicates over a 3rd party network and is subject to the behavior of these devices.

    So the EIOC is more like the Controller than it is the CIOC. When a redundant controller or EIOC are running, the standby processor is updated by the active processor so that the configured blocks and parameters hold the same data as the Active.

    The EIOC supports redundant communications, with Primary network out the left Ethernet IO Port, and Secondary out the Right EIP. Both processors are connected in this way. When an EIOC looses communication with all the connected Ethernet devices, it will trigger a switchover. You could call this a hail Mary in that doing nothing means you have no communication, so why not switch over in case the reason for this total loss of communication is internal to the active Processor. On switchover, the modules all have the last known values with Bad status from the previously active card, and communication starts up. The standby uses the next IP address from the one configured on the Primary (if primary is 192.168.20.10, then the secondary uses 192.168.20.11). If the new active EIOC is unable to connect to the devices, then all the signals are marked BAD and values go to ???? in modules.

    Losing all devices on both segments (Primary/Secondary) should be next to impossible. Unless you have a single point of failure such as using only one network (Primary), or using a common power source on all switches. Most end devices are simplex in nature and do not support redundancy. However, you can set up the Primary network as an "A Bus" and the Secondary network as a "B Bus". Configure some devices on A and others on B. This would then create two separate busses using both EIP's on the EIOC, separate switches and network cables. With at least one device communicating, the total Loss of Communication scenario is all but eliminated. Unless of course you have common power as a single point of failure. Or you used a single switch with VLANS.

    If you have all your devices on only the Primary network, then you have a single cable to connect to a single switch through the primary EIP. Losing one of these crucial links in your system means a total loss of communication, leading to a switchover, resulting in failure to re-establish communication and the LDT Signals all showing ???? in your Modules.

    Take away: Split devices between both busses even if they are simplex end devices.

    On the CIOC, behavior is different. The Controller (SX/SQ in v13.3.1, but SX/SQ/MX/MQ and PK series in v14.LTS) maintains Primary and Secondary communication to both the Primary and Secondary CIOC processors. A switch over of the controller re-establishes the connection to the CIOC's to start receiving IO data and holds IO values and Status so control modules do not change their inputs as modules start to execute.

    When it comes to the EIOC in v13.3.1, there have been several Hotfixes to address some issues with Redundant network behavior for PDT's configured as redundant devices using the Ethernet IP protocol. You should update to the latest Hotfix as soon as possible. You should also log a call with the GSC if you experience unexpected behavior of the EIOC communications.

    Andre Dicaire

  • s_brwn

    In reply to Andre Dicaire:

    Thanks for the detailed explanation. I need to do some research as to how our networks and configs are setup.
    Just so I understand- The CIOCs should not cause I/O to drop/modules to go bad if a single CIOC fails?
    I hear you saying-On an EIOC it depends on network and power design as well as config of the ports(p01,etc) on the EIOC but, when it switches, the modules referencing data on a port/device using the network or eioc that failed will show bad during switch?

    Thanks again. I will likely have some follow-up questions once I do some digging on my end.
  • pmhamilton

    In reply to Andre Dicaire:

    @ Andre Dicaire: "Most end devices are simplex in nature and do not support redundancy. However, you can set up the Primary network as an "A Bus" and the Secondary network as a "B Bus". Configure some devices on A and others on B. This would then create two separate busses using both EIP's on the EIOC, separate switches and network cables. With at least one device communicating, the total Loss of Communication scenario is all but eliminated."

    Doesn't redundancy imply the primary and the secondary IO scanner have the same scanlist? How can you have some devices on the Primary and some on the Secondary. Maybe both IO scanners are dually attached to "A Bus" and "B Bus"?

    PRP, Parallel Redundancy Protocol, happens at layer 0, and layer 1... in the hardware interface. This requires two RJ45 on the interface, e.g. LAN A and LAN B. Each 1/2 of the EIOC only has one RJ45. PRP would require two RJ45. So maybe the to IO scanners in the EIOC are both sharing the same interface which is a DAN, Dually Attached Node. The left IO scanner is using both LAN A and LAN B. The right IO scanner is also using both LAN A and LAN B. If the left is 192.168.1.10 and the right it 192.168.1.11 then both addresses are assigned to the left RJ45 and both are assigned to the right port... thus the left is TX/RX on both LAN A and LAN B via 192.168.1.10 and the right is TX/RX on both LAN A and LAN B via 192.168.1.11

    On the controller side of the EIOC, are those RJ45 interfaces also shared in the same way where the left 1/2 of the EIOC is available to the controller network via both LAN A and LAN B and the same case with the right 1/2. Each RJ45 port on the top of the EIOC would represent both halves of the EIOC via the left IP address and the right IP address.
  • Andre Dicaire

    In reply to s_brwn:

    Correct, The CIOC is designed to continue serving data when one CIOC processor fails or is taken out. Similarly for the EIP modules. If an active CIOC fails, the secondary takes over and there is still full redundancy on all communications (to controllers and to CHARMS).

    On the EIOC, the IO will indicate BAD status on loss of communication, but the last usable value still appears in the modules. This persists through the switchover, but if communication to a device is not restored after switchover, the data will go to ????. A switch over due to no communication happens when all device communication is lost. Which would imply an internal fault or catastrophic failure of the two networks.

    Andre Dicaire

  • Andre Dicaire

    In reply to pmhamilton:

    pmhamilton, In v14 the EIOC is better positioned for integrating subsystems, like process skid PLC's, Analyzers, SCADA networks, IEDs. PRP was supported for connection to EIC61850 IEDs, but can be used for any protocol increased network availability to simplex devices. In v13, it was released as a gateway to all your Ethernet devices, up to 256 devices on one EIOC. However, motors and drives are better integrated to the controllers that host their hardwired signals. With the release of the PK controllers, the Ethernet Protocols are now natively in the controllers.

    When integrating subsystems to the EIOC, you have a choice to deploy the redundant network architecture, or the PRP network architecture. The Redundant architecture uses the two Network Interfaces of the EIOC as two separate subnets, and thus we can refer to them as Primary and Secondary, but also as LAN A and LAN B. The EIOC does allow simplex devices to be connected to either of these, while a redundant comms device would be connected to both of these.

    PRP is supported, which means the two Network Interfaces of the EIOC share the same IP address, and there is only one subnet. Packets are sent on both ports at the same time, with a PRP preamble indicating they are paired. When received by a PRP device, the first packet to arrive is used and the partner is discarded as the data has already arrived. Since the packets traverse two separate networks, one will typically arrive ahead of the other, or they may alternate, but it does not matter. The packets come together in the end device where they are either accepted or discarded.

    So yes, the EIOC has two Ethernet ports, and the are connected to the two RJ45 ports on the primary and secondary EIPs. There is no LAN A/LAN B when PRP is enabled. There is only LAN A. The EIOC is assigned its address (192.168.1.10) and the Secondary card is assigned the Plus 1 IP address (192.168.1.11). All devices are on the same subnet.

    For PLC's with separate primary and secondary NIC's and addresses, the redundant network architecture would be used. For Analyzers and monitoring type subsystems, these may be simplex devices and their availability requirements do not justify a PRP network, you can connect these to either bus, and share redundant and Simplex devices distributed across both networks. The PRP option is was implemented for EIC61850 networks, but is available for any of the communication protocols, if you wish to use that architecture.

    The PK controller does not currently support PRP, but is should be there. Most Motor and Drive devices do not support redundant networks. Some support Device Level Ring or Rapid Spanning Tree for network fault tolerance. But if a device is taken off line for maintenance, the fault tolerance is removed, and a failure in another device could lead to loss of communication to multiple devices. Most users deploy star topologies which are simpler to maintain and isolate each device. For the PK controller, you can deploy the LAN A/LAN B approach and segregate those devices to mitigate the impact of losing an upper level switch.

    So yes, PRP works on the EIOC, and it does have two RJ45 ports for the two separate network paths.

    Andre Dicaire

  • ha nguyen

    In reply to Andre Dicaire:

    Hello Mr. Dicaire, I cant tell you how happy I found this thread. I am having problems setting up a redundant EIOC system that will communicate with a plant Modicon Quantum 434 PLC that has two NICs (an "A" NIC and a "B" NIC) with different addresses. Currently the DeltaV is communicating with the PLC but failover is not working. From what I can tell the DeltaV is talking with the PLC through the left-hand EIOC module only.

    When the left EIOC is Active (right EIOC in Standby) and there is a loss of communication (from the PLC) to the left EIOC, the right EIOC takes on the Active role but all information are still BAD.

    When the right EIOC is Active (left EIOC in Standby) and there is a loss of communication (from the PLC) to the right EIOC, nothing happens and all information remains GOOD.

    We have the two EIOC modules, version 14.3, installed in a dual horizontal carrier, with redundant 24VDC power sources.

    My setup is as followed:

    Primary Deltav control network <-----> top left ethernet port of EIOC carrier
    Secondary DeltaV control network <-----> top right ethernet port of EIOC carrier
    "A" NIC card on PLC <---------> bottom left ethernet port of EIOC carrier (through "A" switch)
    "B" NIC card on PLC <---------> bottom right ethernet port of EIOC carrier (through "B" switch)

    My "A" NIC is 192.168.2.26 and the "B" NIC is 192.168.1.24 (separate subnet).

    First question (for now):

    _ In your previous writeup, you mentioned "The EIOC is assigned its address (192.168.1.10) and the Secondary card is assigned the Plus 1 IP address (192.168.1.11). All devices are on the same subnet". How would this work with the setup we have (the NICs have separete subnet)?.

    I will probably have more questions. Hope you don't mind. Thanks so much.

    Ha Nguyen
  • Andre Dicaire

    In reply to ha nguyen:

    Glad the information has been useful.

    You should first work to ensure the PLC is behaving properly on both the Primary and Secondary network. My suggestion is to set this up on a simplex EIOC, and confirm communication is maintained on loss of primary network. The redundant communication is independent of the redundant processors on the EIOC. Once you have solved the secondary network communication issues, you can add the secondary EIOC and observe that loss of primary network no longer causes EIOC to switchover.

    I would use matching addresses on both subnets to avoid confusion. I would also start with subnet 255.255.255.0, and narrow this later if you want a smaller subnet address space. Remember that the Primary EIOC will use the explicitly configured addresses on its connections to both networks. The Secondary EIOC will use the implied addresses which are +1 of the explicitly defined addresses. In total there are four IP addresses used by the redundant EIOC. Make sure these are not in conflict with the addresses of any devices.
    .

    Andre Dicaire

  • ha nguyen

    In reply to Andre Dicaire:

    Thank you for your response and suggestions. I am happy to say after a couple more days of working on this, it looks as though we now have a functional redundant EIOC system communicating with a redundant network PLC. Your suggestion pointed me to the realization that a single EIOC module has Primary & Secondary communication links to the DeltaV Control Network as well as Primary & Secondary communication links to the Ethernet I/O devices network. As you said, the left EIOC module has 4 IP addresses that must be specified, then the right EIOC (if used as a redundant EIOC) will take on the implied or auto-configured IP addresses.

    I think my confusion came when my original assumption was that the left EIOC module takes care of all Primary communication and the right EIOC takes care of all Secondary communication. In fact, each EIOC can handle Pri and Sec communication. So, even if one uses a simplex EIOC setup, it can handle a redundant network Ethernet device (like a PLC with two NICs). I think it is helpful to think of the redundant EIOC modules as LEFT & RIGHT or A & B, not Primary and Secondary as it will get one confused with the Pri & Sec communication links.

    Some suggestions on terminology and other lesson-learned based on my recent experience. Hope users will benefit (or at least pull less hair out in the process):
    1. Use Left and Right (or A & B) for the EIOC pair
    2. Use ACTIVE and STANDBY as roles each Left or Right EIOC can take on.
    3. Each EIOC can handle both Pri and Sec communication to a Ethernet device (like your PLC)
    4. The two pages on Book Online on EIOC redundancy are a must-read.
    5. Set up a simplex EIOC system first (wonderful advice from Andre) and check redundant communication functionality first.
    6. There are three sets of Pri & Sec IP addresses to keep in mind. These must be specified in the DeltaV system. One set is for the EIOC as a DeltaV control network node. One set is for EIOC as a Ethernet IO network node. The last set is for the Ethernet device itself (the two NICs for the PLC).

    Thank you Andre. Best regards.

This is the official online community site of the Emerson Global Users Exchange, a forum for the free exchange of non-proprietary information among the global user community of all Emerson Automation Solution's products and services. Our goal is to improve the efficiency and use of automation systems and solutions employed at members’ facilities by sharing our knowledge, experiences, and application information.

User Groups | World Areas | Community Guidelines | Legal Information | Privacy Policy | Contact Community Manager

Website translation provided by Website Translator

© 2015-2022 Emerson Global Users Exchange. All rights reserved.