• Not Answered

VIM Communication Errors, Ethernet Frame Check Sequence Failure

Thanks for taking your time to review my VIM Communication issue.

 

Five years ago we installed and commissioned the components for our Ethernet IP network.  This consisted of 10 Allen Bradley 755 VFDs, and adding a VIM card to our controller rack.  I created the interfaces on the VIM card, I also built the DeltaV serial links, programmed the modules, and developed the graphics for the control and operation of the VFD motor controllers.  There are 10 VFDs in our process.  They have all communicated with the DeltaV network reliably for the past five years.

 

Two weeks ago a VFD failed.  Two of our junior I&C technicians were assigned to remove and replace the VFD.  I had completed the same job a couple of times before with each technician and it was decided to let them attempt the work without my supervision.  During the installation and downloading of the setup data to the VFD, it was discovered the new VFD would not communicate with the other devices on the network.  I went to assist the two technicians and discovered the VFD was installed with the wrong IP address configured.  We changed the IP address on the VFD, re-downloaded the drive, and the VFD began to communicate on the network and was returned to normal operation.

 

This past Monday it was noticed, by the plant Operations personnel, they would occasionally receive a “Communications Alarm” for the recently replaced VFD.  I began to troubleshoot the cause of the communications alarm.  

I first noted on the Process History View page, several informational alarms coming from two VFD control modules.  These informational alarms are “Input Transfer Failure” & “Output Transfer Failure.”   

I went to the DeltaV Diagnostics page and found no active diagnostic alarms for the system, occasionally communications with the replaced drive would be lost then return a second later

I next went to control studio to inspect and verify the operation of the actual modules receiving data from the VFDs.  While watching the modules I noticed that occasionally the incoming data registers “Register N_45_##” would become a RED X to indicate a loss of data communications, then they would return to normal operation.  Near the same time the data registers would become a RED X and the Diagnostic loss of communications would appear, the Process History View would indicate a new “Input Transfer Failure” & “Output Transfer Failure” alarm. 

I then went to the DeltaV controller rack where the VIM card is installed.  I looked at the card status indication lights and all showed normal operating condition.  I logged into the VIM controller and reviewed the operational status of the VFDs installed.  The controller appeared to be operating normally and I did not find any diagnostic alarms for the controller or its configured interfaces to DeltaV or the 10 VFDs.  While inspecting the controller cabinet and associated hardware I noticed that occasionally the Ethernet communication port, connected to the VIM controller, on the Cisco 2960 network switch, would flash yellow for about 5-6 seconds then return to blinking green.  This same event would occur every 1-2 minutes. 

My next step was to log into the Cisco 2960 switch and review the operating status of the switch and associated ports for the network.  By logging into the switch service port, I was able to determine that data collisions were occurring on the VIM network port in the same frequency as the alarms I was getting on the DeltaV system.

 

I then assumed there was a problem with VIM controller and/or the configuration had been corrupted.  I re-downloaded the VIM controller and rebooted.  This had no effect on the operation of the network. 

As a hopeful guess in correcting the issue, I then re-downloaded the VFD in hopes that it might correct the issue.  This also had no effect on the network operations.

After much digging and checking into the network operations I connected a third party PC to the VIM/VFD network and monitored the Ethernet communications with the Wireshark software application.  The main error I kept seeing repeated on the network communications was “Ethernet Frame Check Sequence Failure” for the VFDs IP address.  I changed the replaced drive's IP address to a new number, thinking there was a conflict with another device that had been corrupted with the wrong IP address.  This also had no effect on the communications error.  The next step we performed was to power down each individual device on the network to see if the communications errors would clear.  The only device that had any effect on the communications error alarm was when the recently replaced drive was powered down.

We then replaced the VFD with another drive from the warehouse.  Once the new VFD was restored to the network after configuration and downloading the communications errors returned, more frequently, than before with the first VFD.  The Wireshark application also displayed a large increase in the number of network communications messages “Ethernet Frame Check Sequence Failure” for the VFDs IP address.

My next thought is to replace the VIM card.  Every other device has been rebooted, reconfigured, or replaced that I thought possible to be causing this error.

 

Have other users experienced a similar issue and what means did you use to isolate your problem?

I hate to continue to throw parts at a problem, but it has become a major issue to resolve.

 

Thanks for your help.

 

4 Replies

  • Have you considered the Cisco switch as an issue when dealing with different MAC addresses on the same port? Could there be rules that cause the switch port to block/reset? You say you've replaced the VFD with a new one, therefore the MAC address is going to be different.
  • Please, read KBA, AP-0800-0051, to see if your switches are compatible.
  • In reply to AdrianOffield:

    Yes, I have considered the switch as a possible source of the issue. However, this system has been installed for five years, and multiple VFDs have been replaced during that time, without issue. The communications error became active at the exact minute of the incorrect download of the replaced VFD. The Rockwell Automation specification for configuring the VFDs specifies the use of a RA 1783-ETAP for each VFD, in a ring configuration, with another ETAP configured as the Ring Manager for communications to the VIM card. I believe the Ring Manager is tasked with controlling the communications to and from the VIM card through the Cisco switch. I rebooted all network devices on the VIM network, e.g. VIM card, Cisco Switch, 10 VFDs, 11 RA 1783-ETAPs, with no change in the type or quantity of communication alarms generated.
  • In reply to Hiliard :

    Thank you for the suggestion. I quickly reviewed the KBA and it appears to be written around the installation of the Cisco 2960 inside of the DeltaV network. My 2960 is installed in the VIM network. I will reread the KBA and attempt to apply any applicable recommendations.