Emerson Exchange 365

MQ controller DeltaV V14.3.1 loses communications with an entire 8-Wide carrier

Ran into a weird one this AM which I'd never seen until we upgraded our software and controllers to V14.3.1 and type MQ respectively. One of my controllers lost communications with an intermediate 8-Wide carrier housing 8 120 VAC Isolated input cards. Looked in diagnostics and all eight cards were showing as CFG Present but no card. My first instinct was to change out the carrier, which I did but which didn't rectify the problem. After checking a bunch of other stuff to no avail I decided to download the entire controller, which still didn't rectify the problem. I finally decided to pull the controller off the bus and put it back in. This worked but I'm thinking that the download should have taken care of the problem, or conversely, that it should never have occurred. Has anyone else run into this?

• controller
• DeltaV #DCS
• 14.3.1

harnettw

• 14 Feb 2021 1:35 PM
• Cancel

20 Replies

• LaurentB

  

  • 14 Feb 2021 3:14 PM

  Probably no relationship with your failure and DV version.
  The issue you describe is related to IOs address bus ( short circuit or open circuit ) . It’s often caused by cables connectors between carriers or at the last pîn connector which should be protected with plastic cover again dust or metal waste.
  Sometimes in corrosive environments the backplane it self can be dammage, but necessary the backplane where you get issues

  also check pin connector behind controller if no one is a lttle bit twisted

  • Cancel
• harnettw

  

  • 14 Feb 2021 3:29 PM

  Thanks Laurent. What was puzzling was that there were two more 8-Wide carriers further down the line connected via cable to this one and they were okay. The controllers were replaced in October during the system upgrade (MD to MQ) so the physical connection to the controller carrier sounds plausible. At any rate, removing the controller and re-seating it seemingly solved the problem.

  • Cancel
• patil_chandrashekhar

  

  • 14 Feb 2021 3:42 PM

  Agree with below points. This issue known as Thin Whisker. I think there is KBA for this issue. This issue may caused due to old type of 1 wide IO extender made with tin plated housing or data pin short circuit with housing. Due to tin whisker , this tin plated housing develope Thin whisker phenomenon and  get short circuit with signal pin which can cause complete IO communication issue.Everytime you will observe different impact on system( all IO's not communicating for controller, from redundant card only standby card no comments, some random card shows cfg not present status, controllers switchover continuously etc..). You can check Run Time recorder of affected controller and confirm if it is DIN rail communication issue. If you know serial no of  installed 1 wide carrier then confirm with Emerson and change affected 1 wide extender with latest which is with stainless steel housing. 

  
  

  Regards,

  
  

  Chandrashekhar Patil
  
  

  On Sun, 14 Feb 2021, 18:18 LaurentB, <bounce-LaurentB@emersonexchange365.com> wrote:
  

  Update from Emerson Exchange 365 LaurentB Probably no relationship with your failure and DV version. The issue you describe is related to IOs address bus ( short circuit or open circuit ) . It’s often caused by cables connectors between carriers or at the last pîn connector which should be protected with plastic cover again dust or metal waste. Sometimes in corrosive environments the backplane it self can be dammage, but necessary the backplane where you get issues also check pin connector behind controller if no one is a lttle bit twisted View online   You received this notification because you subscribed to the forum.  To unsubscribe from only this thread, go here. Flag this post as spam/abuse.

  • Cancel
• harnettw

  

  • 15 Feb 2021 11:34 AM

  In reply to patil_chandrashekhar:

  Thanks Patil. The only thing that detracts from the possibility of it being related to the 1-Wide connection issue is that there are two more 8-Wide carriers further down the line which are joined to the carrier that faulted via a 1-Wide and cable connection. These were fine, as were all preceding 8-Wide carriers. The carrier I had the problem with is the mid carrier in a 6 x 8-Wide I/O/Controller cabinet.

  • Cancel
• LaurentB

  

  • 15 Feb 2021 11:55 AM

  In reply to harnettw:

  What you have to know that this is a adress buss , consisting on 8 "lines" , for example if the controller want to speak with IO card number 26 , the following bit lines will be put out of high impendance  "0001 1010"  but for any reason one of bit line is wrong , the card could not be addressed

  What I want to say , is that the error observed in one carrier can be generated by failure  positionned anywhere on this Address Bus , replacing carrier is not enough, the continuity must be checked from the controller pin mass  until the right end of the latest 8wides carrier, including junction cables, inter connexion between rack on din rail

  And if the physical issue is on the second rack, this second rack can be OK, the 3rd Not Ok and the following 4, 5 6 ... OK,  depending on which address line the faut is

  if you tell me the position of you faulty rack I will be able to tell you which bus line is wrong, , but It will not possible to tell you the exact location of the issue. 

  As previously said the most often case is the latest free Pin connector of the last rack, or on cables connectors or PIN behind the controllers, some times connexion between racks on din rail ( this last case is the worst because impossible to correct it on running plant. This the reason why recommend to check this point on every preventive maintenane done during plant shutdown.

  • Cancel
• harnettw

  

  • 15 Feb 2021 12:01 PM

  In reply to LaurentB:

  Thanks Laurent. I'll have to dig into this and try and figure out what constitutes the address bus in terms of which pins. I suspect I may have to purchase new 1-Wide carriers to replace what I have here.

  • Cancel
• Andre Dicaire

  

  • 15 Feb 2021 4:46 PM

  In reply to LaurentB:

  The card selection is made up of two 8 line busses. One selects the carrier(Bank Select) and the other is Slot selection.
  
  If there is a fault on a line it would affect an entire carrier or all cards in the same slot position on each carrier.
  
  The tin whisker issue is worth investigating, but it might not be that. If a bank line or a slot line were shorted, multiple cards would be affected.
  
  The controller connects to all 16 select lines and will only activate one bank and one slot at a time, this only one card at a time. But how does a card know when it is selected?
  
  Each carrier is identical. the 8 slit lines enter to the left. slot one is connected to line one, line 2 to slot 2 and so one. these lines also pass through the carrier to the right to the next carrier.
  
  The bank select lines enter on the left and line one is connected to all eight slots. the magic happens on the right hand side where the bank select lines are shifted up. line becomes line one, three becomes two, and so on and line 1 becomes line 8.
  
  In this way, line 2 frome the controller becomes line 1 on the second carrier, line three selects the third carrier and so on.
  
  So if a bank line is shorted, all cards on that carrier would suffer same fat. If a slot line is shorted, then all cards in the same relative slots would be affected.

  Andre Dicaire

  • Cancel
• harnettw

  

  • 16 Feb 2021 9:49 AM

  In reply to Andre Dicaire:

  Thanks Andre. I had no idea how that worked. Since I lost all cards in one 8-Wide carrier, I'm going to assume that we experienced a bank line failure. Since I changed both the right-hand carrier that we had the problem with and the left-hand carrier which was plugged directly into it I'm going to keep my fingers crossed that it fixed the problem.

  • Cancel
• Andre Dicaire

  

  • 16 Feb 2021 5:28 PM

  In reply to harnettw:

  The KBA dealing with Tin Whiskers mentioned above is a good one to review. It discusses versions of 1 wire extenders that were more susceptible to this, so you should confirm if your system has these and consider replacing them at your next opportunity. Resetting connections and securing cables can address a faulty connection.
  
  1. Check the entire LocalBus to ensure all carriers and extenders are securely connected and have not separated due to vibration over time. Check DIN rail stops to ensure they are present and securely fastened to prevent separation. Check all extender cables are securely fastened. Confirm the unconnected connectors are covered/protected.
  2. Remove each affected IO card one at a time and observe if other affected cards begin operating. after removing the last card, reinstall the first card. If it is still not working, the issue is not an IO card.
  
  If the issue is not identified by the above, then we have to consider other failure points, such as the Tin Whiskers issue, which involves specific builds of Extenders and the environment. If the conditions for this known issue are present, all your extenders in the same location are susceptible and you should mitigate this as indicated in the KBA. I would recommend you understand if this needs attention on your part.
  
  When it comes to the 1-wide extenders, the later versions released with parallel connectors, allowing for dual cabling. If the second connector is not used, it should be covered/protected. Tin Whiskers are microscopic so you cannot simply look to see if the issue exists. You could replace on connector and send it to Emerson for evaluation, assuming the version you have falls into the components identified in the KBA.
  
  Since your system is back up, the issue appears to be in the carriers and extenders/cables.

  Andre Dicaire

  • Cancel
• Lun.Raznik

  

  • 16 Feb 2021 5:47 PM

  In reply to Andre Dicaire:

  I was reading an article that due to the lead-free initiative thin whiskers are back. I would consult with GSC/PE if they have mitigated the side effect of lead-free initiative causing thin whiskers.

  • Cancel
• LaurentB

  

  • 16 Feb 2021 5:57 PM

  In reply to Lun.Raznik:

  I not only read the article i already experienced this issue 2 times

  • Cancel
• Andre Dicaire

  

  • 16 Feb 2021 6:09 PM

  In reply to LaurentB:

  Care to share any details, such as what components were affected? Does your site have a corrosive environment? Did you replace some or all of the affected products?
  
  I recall the shift to lead free solder and the resulting Tin Whisker issues, and that Emerson had investigated solutions to address this in all its products. Since DeltaV Systems typically run for decades, I'm hoping you can provide some context to your experiences as opposed to just leaving an open statement.

  Andre Dicaire

  • Cancel
• doug bray

  

  • 16 Feb 2021 6:28 PM

  In reply to Andre Dicaire:

  Does Emerson recommend periodic removal of end covers , the covers recommended by the KBA, and inspecting the connectors for whiskers?

  • Cancel
• Andre Dicaire

  

  • 16 Feb 2021 7:56 PM

  In reply to doug bray:

  the KBA AP-0700-046 discusses this Tin Whisker issue in DeltaV that impacts the Dual Extenders on the M-series Railbus and the KBA calls for mandatory installation of the specified covers, which shroud each pin and fills the D-shell cavity of the connector.
  
  It also states that when installing these, do so without power on the IO subsystem (remove the system power supplies or power them down) and to use compressed are to clean out the connector and ensure there is no debris. The KBA does not explain why you should power down, but my guess is that if there is debris in the connector, blowing it up and around could cause a short.
  
  Periodic verification is suggested, which is to make sure the connector covers are in place. The recommendation does not call for removing and inspecting the connector. So no, no recommendations to remove the covers.

  Andre Dicaire

  • Cancel
• doug bray

  

  • 16 Feb 2021 8:33 PM

  In reply to Andre Dicaire:

  Thanks Andre. I do look at them periodically to be sure they are in place.

  • Cancel