Is there an issue with dynamic references , inter controller communication and reboot of the ProPlus?

Hi Folks,

We had an issue last where the ProPlus need to reboot while our process was running.

The project uses extensive the dynamic references as a very powerful solution in our batch.

During the reboot we figured out some problems with phase pulsed actions to control modules in different controllers - and yes, we check the CST before using the reference.

Which Node is responsible to resolve these references, how does it roughly work DeltaV intern  and is there any kind of redundancy for that?  

Thank You in advance for answers.

Best Regards, Michael

  • Michael,
    Are you using the ProPlus for any tasks in addition to configuration tasks with the configuration database? For example, do you have an OPC server, the Batch Executive or any other application running on the ProPlus?

    What specific kind of problems did you observe with the pulsed actions in the phases in different controllers after the reboot of the ProPlus?

    J.D.
  • In reply to JDWheelis:

    Hi JD,
    we observe not strange things AFTER but DURING the reboot, while the process is still running.
    The ProPlus works as a Proplus and only as a ProPlus and only as a ProPlus! Batch-Server and everything else is on another node.
  • In reply to Michael Krispin:

    What version of DeltaV? I seem to recall that there was a version of DeltaV that would need ProPlus for locating the modules for dynamic references.
  • In reply to Michael Krispin:

    Hi, Michael,
    OK, I agree it is best to have the Batch Executive and everything else on other node(s). So what kind of strange things did you observe during the reboot? And did those things stop occurring once the reboot of the ProPlus completed?
    J.D.
  • In reply to JDWheelis:

    Hi JD,
    so far as I understand from the maintenance group there where that the references did not resolve and cause a kind of hanging.
  • In reply to Lun.Raznik:

    Version is DeltaV 12
  • In reply to Michael Krispin:

    This was an issue with v12, can you please check with GSC if this was fixed.?

    Also, can you please try to apply all the DeltaV hotfixes?
  • Hi Michael, were also at version 12 and have had similar issues. Our Pro+ DVDBserver hangs on occasion. DeltaV diagnostics indicates a Red X for the Pro+ but the Pro+ is still but the database is hung. I believe that if the Pro+ had a fault and powered down that another operate station would provide the path to inter-controller communications calls like in a phase. We worked with Andre Dicaire to implement a work around. We built modules in each controller that ran batch points with ref's to the other controllers. The modules just contain one reference to a parameter in the module so that the inter-controller communications are setup using continuous modules rather than called when running a phase. Since the path to the module is in a continuous module the phase can reference any parameter within the module.

    leo

    Leo

  • In reply to Leo Paradis:

    Hi Leo,
    I read about other issues regarding the dynamic references and that a fixed external reference to such a module helped the proxy server to resolve it. I step out of understanding it if someone come with a proxy server etc., but believe It does not suite our configuration.
    I know that even in very earlier versions some strange behavior occur around that dynamic reference, that one of our service guys told better not to use them. That I denied that time and told him to deliver a solution, not rumors, due to the function of dynamic reference.
    So please come back to my initial question:
    >Which Node is responsible to resolve these references, how does it roughly work DeltaV intern and is there any kind of redundancy for that?<
  • In reply to Michael Krispin:

    This could be wrong but I thought the fix was that it is now being handled by who is the YellowPages master. In DeltaV if ProPLUS goes down another workstation will become new YellowPages master. If that new YP master goes down, a new one will be automatically promoted (I think based on node ID).

    In v14.3 (PK release), this should be even better as PK can work without any workstation and dynamic reference is working. It might be that PK can act as YP master. I don't have PK at the moment so I can't verify.

  • In reply to Lun.Raznik:

    Hi Lun, is that valid also for DeltaV 12?
  • In reply to Lun.Raznik:

    Lun, The PK controller can be standalone or native to a DeltaV system. When it is standalone, there is only one PK controller and so remote references cannot exist. References to the workstation (i.e. HORN parameter) are device references and not module references.

    As Leo explained, the Pro Plus was observed to get into a state where it was still connected to the system but was not servicing requests for Module Location to enable remote references. Normally, the Pro Plus services these requests, which re made via broadcast so all workstations can see them. Only the computer with the active role will respond, typically the Pro Plus. If the Pro Plus is taken off line, disconnected or rebooted, the next workstation by Node ID/IP address will take on the role. This is an election based on loss of connection to the Pro Plus. When the Pro Plus comes back, it will take back the role.

    This has been the case for a long time, even pre-Yellow pages. Not sure. I think before v7. It used to be that controllers were passed the complete list of modules in the system, and their host controller, allowing a remote reference to be established without help. As the system grew, the size of this module table became too large and the task of determining the module location was taken on by this service. On download, the Pro plus is necessarily available and fixed references are established soon after download. The controller sees the reference is remote and broadcasts for the location of the module in question. The active role node (pro plus typically) responds and the controller can build the Proxy client/server connection for the module. Any subsequent parameter reference to this module is added to this existing connection so subsequent dynamic references or PHASE Logic references that come when a phase is instantiated will make use of the existing Proxy.

    The Proxy connections persist in the controller until either a total download or a switchover. When a switchover occurs, any existing active remote references will require the proxy to be be rebuilt and the broadcasts will be made accordingly. To affect remote references, both the module location service must be unavailable and a switchover must happen on the node making the reference. By creating some Artificial References to modules that will be needed by Phases or dynamic references at some later time, the Proxy Client is created and avoids the need to make the location request. The only time we need the service is when we make a change on a download or following a switchover. This shrinks the window where an issue with the Pro Plus will impact remote references.

    So even though the Pro Plus is shutdown or disconnected, module location requests will still be serviced by the next workstation. If the Pro plus is in a limbo state where it is still on the system but not servicing requires, then the election to a different computer does not happen. This is being investigated.

    Andre Dicaire

  • In reply to Andre Dicaire:

    Hi Andre,
    thank you very much for that detailed explanation.
    I ask the operator/maintenance guys again for more detail.
    The ProPlus have a daily scheduled reboot after midnight. Usually there is no issue with that
    In our special case happen that an EQM- command given by a phase via a dyn-reference were not transferred on what ever reason, so that the batch hang at the same time when the ProPlus reboot. Unit Module and EQM are in different controller.
    We found the following KBA NK-1200-0157 for DV10 and 11. We run on version 12.
    Could it be that we just had bad luck as one time event that writing/resolving just happen when the ProPlus start to reboot?
    I mean, it take a very short time to switch to the next node.
    Would have eventually the write queue of the inter controller communication also have an impact here?
    Remain only the point that the CST was 0 before, what I understand that the dyn-reference is resolved.
    Using of the dyn-reference to write commands depend on the successful confirmed status in our phase actions.
    On the other side it is confirmed that the dynamic references also can resolve without the ProPlus due to the mechanism which you described above. So, no general design problem here.
    BR , Michael