Emerson Exchange 365

EIOC AWSnt Number of asynchronous buffered writes >10

13.3.1, PSCD 13- I've recently had some intermittent comms with only one of several PLC devices on my system.  PLC brand is Rockwell. Using ENBT or similar comm card in PC rack.

It happened when we installed a "passive" network sniffer on the PLC network.  Supposedly it wasn't sniffing or causing any traffic but when we removed it, my problems went away.

The symptom was one of my plcs was dropping out a watchdog register.  This particular PLC has 22 LDTs and is passing quite a bit of data.  

After some research I've found my AWSnt number at 54.  I increased my RPI on all the LDTs from 1000 to 1500 this morning and didn't see a difference on the AWSnt.

Does anyone know what drives this AWSnt number?  If I understood better what affects it, it would help with troubleshooting and correcting it.

Thanks

• EIOC
• Buffered Writes
• AWSnt

s_brwn

• 19 Sep 2024 1:29 PM
• Cancel

5 Replies

• Matt Forbis

  

  • 19 Sep 2024 1:47 PM

  AWSnt is the Asynchronus Writes on the DeltaV network. So, that is DeltaV module to module communication between controllers. The RPI would not have an impact on this, but the module scan rates and configuration of the modules in the EIOC would drive this number. Have less write external references and more reads from the other side will reduce this number for example.

  • Cancel
• Andre Dicaire

  

  • 19 Sep 2024 8:58 PM

  In reply to Matt Forbis:

  As Matt said, the AWSent value is triggered by a Module parameter write from a local module to a remote Module located on a different controller or EIOC or App station.
  
  Such writes should be by exception. THey can be an assign statement in an expression, or a wired connection to an Output Parameter configured as an external reference. A wired connection would result in an AWSnt every time the module executes.
  
  The main use case for a write is when you have a sequence in one controller that is managing modules in other controllers. As the SFC executes, it writes to the MODE and or SP or other parameters to coordinate their behavior, much like an Operator would do following a procedure. The SFC Actions are structured with confirm statements that can be used to verify that the AWSnt was successful, by reading the MOD/PARAM.AWST status of the parameter written to.
  
  What happens when an AWSnt is created? The module executes the write which is buffered in the controller comm layer, allowing the module to finish execution. The buffered writes are gathered up and transmitted every 200 or 250 ms (can't remember the period for this), at which point values are sent. The receiving controller processes these writes at a higher priority, writing the value into the destination module, and confirming the value is acceptable (in range, block mode allows write, etc). If the value is rejected, the response is returned, even before the module executes. The AWST value will reflect the success of the write by going to 0.
  
  BOL indicates that you should have a maximum of 20 AWSnt values per second. In reality, the extra load is on the destination end. If you had 50 AWSnt's but distributed evenly across 5 other controllers, that would not be outrageous. But it can get out of hand if you are not aware and are writing in CALC blocks continuously or wired to Output parameters.
  
  When you perform an assign statement (Parameter := Something) to a remote module, the writing controller will setup what is called a ProxyClient of that remote module, and add the parameter in question so that it receives the value of that parameter as unsolicited data with exception reporting. The parameter will have its .CV, .ST, .CST and .AWST fields. The CST is the Connection Status, which is typically 0 unless there is a network issue or module or parameter has been deleted. You can use these fields in the confirm expression of the SFC Action to know the write was successful, and subsequently handle any errors.
  
  For continuous data exchange, always use a read, which is accomplished at the destination end via an External Reference on an Input parameter, or in an expression. On download and first execution, the destination module causes the same Proxy Client object to be created for the source module and the parameter is added to the Proxy server and updated by exception at the frequency of the requesting module. The data is sent and waiting in the PRoxy Client so that the destination module can read it in real time.
  
  Another benefit of reading the data is that the value is sent once by exception to the Proxy Client and can be read by multiple modules in the destination controller. Using Writes to send a value to multiple modules in the same controller requires separate individual writes and multiple reciprocal sends of the parameters written too.
  
  Writes create more comm load and control processing that Reads.
  
  But these would have nothing to do with RPI issues at the PLC comm layer. That activity is abstracted by the EIOC control modules, which must execute to read or write to the PLC via the P01 port/protocol. AWSnt are on the DeltaV peer to peer layer and independent of P01 comms.

  Andre Dicaire

  • Cancel
• Andre Dicaire

  

  • 19 Sep 2024 9:33 PM

  Some learnings I've had with EIOC (and PK) Ethernet drivers. With Rockwell you are likely using EthernetIP UCMM for logix tags. In this set up you create Array tags in the PLC which map into EIOC LDT's. Each array can be up to 100 values/registers, which are mapped into the LDT signals. You set the LDT's RPI or requested packet interval to a desired update and as long as the data is received within this time interval, all is good.
  
  I think there is a white paper or KBA that explains a bit more how this works. BOL approaches the configuration from a small device perspective like a drive or MCC.
  
  Here we go. Each Physical device will schedule an LDT every 20 ms. This allows many, upto 256 devices to interleave their respective LDT's and provides a lot of time for the Ethernet packets to be sent and received. I think the response packet is returned within a few milliseconds.
  
  So how many LDT's can you have in a PDT? Well that depends on your RPI. If you set RPI to 500 ms, and the PDT schedules its LDT's every 20 ms, if you have more than 25 LDT's, you will exceed 500 ms before the first LDT is sent again, and that will get you an RPI Error.
  
  In addition to the schedule LDT's, Output LDT's can be triggered to be sent when a module writes a changed value to a signal. The EIOC will flag that LDT to go out immediately as the next schedule LDT, delaying the regular schedule. The KBA recommends keeping the number of LDT's to 80% of the calculated number of LDT's for a given RPI.
  
  What if you have some fast (500 RPI) and Slow(2000 RPI) LDT's in the same PDT? You will have fewer packets exchanged vs having all those LDTs at 500 RPI, but the scheduler does not balance the longer RPI LDTs in any way. During the first cycle, all LDT's will be scheduled and if you have more than 20-25 of them, your 500 RPI LDT's will experience an RPI fault. If things catch up by the fourth cycle, the 2000RPI LDT's would remain happy, but you would trigger another RPI error on the 500 RPI LDTs.
  
  An RPI error does not mean the data is bad. Just that the LDT response was longer than requested. When the data arrives, it is made available to modules. The DeltaV Diagnostics on the LDT's (SHOW Values) will show values as good status. but the LDT status will be bad.
  
  For your watchdog to trip, the Response time needs to be long enough to affect your WDT timer setting.
  
  On the other side of this equation is the PLC servicing the data. The UCMM communications is a lower priority than the PLC IO data. It is possible that the PLC bandwidth is throttled to ensure IO is updated. I am not that familiar with the communciation mechanisms of the PLC so I'm not saying that is the problem. But if you wireshark the network and look at the LDT and response traffic, you would see the periodic polls from the EIOC and response data. If the Response are delayed, and this intern causes the LDT's schedule to be delayed, then your RPI times can be exceeded. and if this is long enough to trip the WDT, things are overloaded.
  
  What we did was create multiple PDT's to spread out the LDT's into multiple schedules. Yes you have to license the PDT, even though you still only have one PLC device. With UCMM, this resolved our RPI errors. We decided to group all LDTs by RPI and kept the number of LDT's well below the 80% mark. i.e. for a 1000ms RPI, we have about 30 to 35 LDT's or 60-70%. For a 2000ms RPI, we can have 60 to 70 LDT's. We ended up with multiple 500 RPI PDTs.
  
  My theory, is that the Rockwell PLC also places limits on how much data can pass through one connection. The EIOC does establish a connection, but there are individual CIP connections for each PDT and then the LDT's within them. We successfully managed thousands of values out of the PLC's. Tuning parameters, alarm limits and such are generally static. These were set slower. Output with readback allows DeltaV to see the value change if tuned via a PLC interface directly, or to set them from DeltaV. Same for SP and OUT values of a PID. but these were 1 second. or maybe 500 ms in some cases. PV's generally 1 second.
  
  Anybody want to offer some insight on the PLC End of this connection? What impacts RPI from the PLC Side and how would one address it?

  Andre Dicaire

  • Cancel
• Andre Dicaire

  

  • 19 Sep 2024 9:53 PM

  One more dissertation.
  
  The new Control Tag Integration driver in v14 provides a direct interface to Rockwell control tags and forgoes the need for intermediate Arrays to map data. This adds a substructure to the LDT's. There are up to 10 control tags of up to 10 signals per LDT. Each control Tag links to a UDT or control tag in the PLC by name and can directly reference up to 10 values of any type.
  
  Licensing is per control tag, so that can be up to 10 DST's per LDT's. The added licensing is offset by eliminating arrays and data mapping. On an OEM PLC where modifying the logic to add intermediate arrays would somehow not be welcomed, Control Tag Integration could solve that issue.
  
  CTI changes the underlying PDT/LDT schedule. The driver separates Output Float signals from the others an signals from multiple LDT's are grouped in the messaging structure. We ran into RPI errors with this protocol as well, passing upwards of 5000 signals. Adding more PDT's as we did with UCMM helped. But the CIP structure was different and we could not reliably get to 0 RPI errors. As I said, the data still comes across with good status, but RPI errors were disconcerting and not acceptable.
  
  Eventually, the engineering team decided to add an additional EIOC and spread the PDT's across both EIOC's. With the same number of PDT's and LDT's, all RPI errors were resolved. In the Wireshark capture, we could see how each EIOC created a master connection to the PLC, and each PDT created a CIP connection under this main connection. With Two EIOC's, there were two separate IP level connections.
  
  This leads me to believe the CIP layer applies some limits to each client node. This makes sense as you wouldn't want one node to disrupt all communication to other nodes.
  
  Once again, I'm not familiar enough with the workings of the CIP protocol in the Rockwell PLC to know how this might be worked around. One idea I had was to add a second ENTB card to the PLC and address half the PDT's in the same EIOC to the second Card. Would this have the same result as two EIOC's? There would have to be two IP connections in the EIOC, one to each ENTB card, and the CIP communications in each card would be separate. The limitation was not in the physical capacity of the single ENTB card as it passed all data to two EIOC's without issue. But a second ENTB card is much less costly than a second EIOC.
  
  One final note. BOL performance information for the EIOC (and PK) in terms of number of signals is based on 256 separate physical devices. As you add devices, you add signals. This is why we added additional PDT's with UCMM and solved the RPI errors. CTI is a different beast. We were not able to get to a solid understanding of what drove the RPI's.

  Andre Dicaire

  • Cancel
• s_brwn

  

  • 25 Sep 2024 12:01 PM

  In reply to Andre Dicaire:

  Thank you very much for the information. It helps a lot. I have started to drill down into the system to see what I can find.

  • Cancel