Emerson Exchange 365

ACN switch over events are caused by a node experiencing issues with its ability to communicate to other nodes on its primary Network. All nodes will use the Primary network when available and switch to the secondary when message confirmation time out. When this happens, all nodes that communicate to this device will indicate they have switched to the secondary for this node. The loss of message confirmation can be due to an issue with the node, its primary Ethernet cable, the switch port it is connected to. If the problem is with the Switch itself, or the uplink port, the issue would affect all devices on that switch.

If all the messages involve one common device, Then the problem must be local to that device, i.e. its switch port, cable or the device itself, or a media converter connecting the device to the switch. In this scenario, the affected device will be referenced in every message as either the source or destination.

If the messages involve various devices, you can sort to determine which group of devices are effected, and this would point you to a common point in the network, such as the uplink of a switch that has these devices connected to it.

Once a device switches to the secondary network, it continues to monitor the primary with diagnostic messages to the devices it is connected to. Removing power from a switch would affect all devices that connect and share data through that switch. Rebooting a console could cause a flurry of messages and once rebooted more messages as comms return to primary network.

Start by identifying affected controllers and determining where in the network they are connected to narrow down the common connection path.

Andre Dicaire

In reply to Andre Dicaire:

Thank you Andre for all the valuable information.

In reply to doug bray:

Thank you Doug .

In reply to doug bray:

ACN switch overs happen for a reason. Restarting a workstation or the DeltaV services or other activities that affect a node's ability to communicate will cause them. These would be "normal" as they are an expected result to a specific action. I've see such responses in call logs. If someone disconnects a cable temporarily, you get ACN errors, and they stop.

The network connections "heal" themselves and it is not likely worth the effort to investigate every occurrence of ACN switchovers that come and go. But generally, there should be no ACN switchovers that just randomly happen with no explanation.

the ACN switch over from primary to secondary happens when a node fails to receive confirmation of packets sent to another node. Data is communicated in Uni-cast packets, meaning that each node sends data in separate packets to different nodes. Each packet is confirmed received. If this confirmation is not received on 3 consecutive sends to that node, an ACN switchover happens and the data is sent to the non-responding node over the secondary.

There must be multiple failures in a row for the switchover to trigger. The issue is either the packet did not get through on the send so the destination did not see a need to respond, or the packet gets through but the confirmation does not, or the node is not responding. If a workstation is somehow unable to process communications, ACN switchover would be triggered. In this case, there is no network issue, but the lack of confirmation messages will trigger the sending nodes to try on the secondary. They only know that no response was received. They don't know why.

So there may be conditions that result in ACN switchovers that are created under normal activities, like maybe a total download to a node, and maybe this is considered as "some" ACN switchovers are expected. But there is a reason behind them and they are not random or phantom occurrences.

Andre Dicaire

In reply to Andre Dicaire:

Lol... "some" in our facility is eight during the last hour. I know they are caused by something and not phantom but with no clear pattern is is very difficult to find the culprit. They are sporadic, coming from different devices talking to different devices. Since each device has connection list to each other device, an ACN switchover often comes from a device that really does not need to talk to. For example, a controller in one part of the facility may have a quick issue talking to a CIOC which the controller does not have I/O on and really does not need information from.

Maybe they are caused by a large motor starting somewhere and maybe there are Deltav cables that run too close to high voltage motor feeders? I don't know. I just have learned to accept ACN switchovers as thing and I try to keep an eye out for excessive amounts.

I have been successful a few times in finding the cause of ACN switchovers, but those were when we had serious issues that showed up via the pivot table look. The various issues we identified before were things such as bad cables, bad connectors, and an operator station sporadically broadcasting bad packets. I hope our upcoming upgrade from version 11 to 14, with the addition of smart switches will give us a larger arsenal to use to identify the causes.

Andre,

I would love o hear of other ways find the culprits. We look at switch logs and use pivot tables. Those are my two best tools to look.

In reply to doug bray:

Are you up-to-date on hotfixes and firmware? We were likely the initiating site that drove a resolution on this issue. We've not seen these recurring problems since applying hotfixes last year. Our issue was due to several non-critical engineering workstations which do not have both ACN connections.