External Reference | Online Value and Status is "???"

DeltaV 13.3.1

I've got an issue resolving external references but only between a specific controller and application station.

I've got an MD controller running a control module with external references to modules in an application station.  Currently, all of those external references are showing an online value of ??? and an online status of ???.
The issue seems to be isolated to the controller reading from the application station.

Additional info:
Other controllers can read these same values from the application station
The affected application station can READ values from the affected controller.
On the affected controller, if i go into diagnostics and view the communications connection list, the app station shows up normally and in a ready state.

Gonna be digging around but was hoping you guys could help me save some time. Slight smile

  • First, I would do a changed setup download to both of the nodes. This could cover if the module table has been updated and not downloaded.

    That seems to be the most common cause, but there's more we can look at if that doesn't solve it.
  • In reply to Matt Forbis:

    I have done that on the Application Station and Pro Plus machines. Are you saying to do that on the controller also?
    Does the setup data to the controller include module level data?
    I thought if there was an external reference, the controller would query the pro plus to get remote module information and then use that to establish comms.

    Not questioning, just asking questions. ha
  • In reply to Invalid String:

    Every node has it's own copy of the the module table that's downloaded. This is what allows the distributed nature of DeltaV and allows it to work between any nodes, with no requirement for any server like the Pro+. The module table is basically a list of modules and where to find them so that each node knows where to go to resolve paths. It does not include individual parameters down inside of a module.

    Another example of things covered by setup data is named sets. All named sets have to be downloaded to every node that needs them so the nodes can convert numbers to strings.

    So, when in doubt, downloading "Changed Setup Data" is always a good first troubleshooting step. Please note, I mentioned "Changed". In general, downloading "Setup data" should not cause issues, but the number 1 rule of downloading I follow is the minimum amount of downloads the better. By choosing Changed Setup Data, you will only download the pieces that need updating.
  • In reply to Matt Forbis:

    Thanks for the explanation.
    I'll have an opportunity to download changed data this weekend and see if it fixes the issue.
    For now i have reassigned the module to a different controller and it is working as expected.
  • In reply to Matt Forbis:

    A little clarification on module table. Controllers and EIOC's do not receive a complete module table that is sent to all workstations. The controllers know of their local modules on download. External references to modules in other nodes require the controller to request that information from the "Directory Serivces" in the form of a multicast request. Normally, the Pro Plus handles this request and sends the controller the host node for the module. If the Pro Plus is off line, the lowest IP address workstation will take over servicing the requests. The Pro Plus will take that back once it is on line. Early versions of DeltaV downloaded the entire module table to the controllers, but as the system grew, the size of the file and content became problematic and the request method came into being.

    Once a controller receives the location information, it creates a Proxy Client object in the communication layer to manage data exchange with the remote module. The Remote node creates a Proxy server object to do the unsolicited communications of data to the requesting node, at the frequency specified by the requesting module's scan rate. i.e. a one second execution scan rate on the requesting module sets up the proxy server to check for changed data every second. The Source module must execute to flag any data as changed. If the source module runs slower than the requesting module, then the data will change lower than the Proxy server checks for change, and will only send data after the module runs. So the communication will be the slower of the two module scan rates.

    In order for the proxy server and proxy client objects to be created, there must be available Object ID's in each node. In the Application station, if there are no Free Oids, the proxy server cannot be created. If the application station is running History or serving OPC data to an external consumer, every module sending data will have a Proxy client defined on the app station, and each consumes and OID. similarly, each module assigned to the station consumes and OID, and any Proxy server objects created use an OID. Since a Proxy server is created for each node that requests data from a module assigned to the station, then it is possible that one node successfully created the Proxy Client/Server objects, using the last OID, and subsequent attempts fail. Check diagnostics on the App Station for Free Oids. Likely not the issue, but then again, could be. Also check the controller in question.

    The Workstations today have 32,000 Free oids and controllers I think have 16000. They are consumed by modules, IO, Proxy Clients, Proxy servers etc. Very rarely are these an issue. But it is a finite resource. On application stations with History, every module that contains a history tag will have a proxy client on the app station. If you historized one tag per module, and historized 30,000 tags, there would be 30,000 oids used. Normally, history collection averages 2 or more tags per module and the actual OID usage is typically below 15000.

    Note that once a module has been setup with a Proxy Client in a node, all data from that module flows to that proxy, and all consumers of data from that module do so from the same Proxy Client. If History and OPC Server use data from the same module, even if it is different data, only one Proxy Client is created and one OID consumed. The Proxy Server will send the data at the fastest rate requested. If history is set to 10 second, and OPC Server is asking for 1 second, the Module proxy server will send everything at 1 second.

    Enough about that. check for Free Oids.

    Andre Dicaire

  • In reply to Invalid String:

    Which module did you reassign. The remote one you are getting data from?

    Andre Dicaire

  • In reply to Andre Dicaire:

    I reassigned the module that was running in the controller. The data source is the application station module and the external references are defined in the module running on the controller.(now in a different controller)

    Free OID is 8485 on the original controller(MD), 31698 on the new controller (MX), and 31441 on the Applicaton Station
  • In reply to Invalid String:

    Just to follow up, downloading the changed setup data fixed the issue.
    Thanks guys