Emerson Exchange 365

• Not Answered

SD Plus controller upgrade to SX controller online (DV12.3.1)

Hi

We have a client who is running on 12 redundant SD Plus controllers where 3 of them are overloaded meaning the FREETIME is less than 28% because we are using CIOC's. One controller is at 17%. We are experiencing inter controller communication failure for some parameters when we have to download some changes on other controllers. The Client is prepared to upgrade the SD Plus controllers to SX controllers but I can not find information if it is possible without causing a plant disruption.

How much FREETIME must the Active SD Plus Controller (Redundant) have to upgrade the standby to an SX controller online without causing a plant upset in DeltaV12.3.1 ?

I searched on SMS website, Books Online and Datasheets but could find no information about the minimum FREETIME that is required.

I assume that it must be above 28% meaning that an online switchover is not possible.

Kind Regards

Evert

evertz

• 27 Sep 2016 7:25 AM
• Cancel

3 Replies

• Neil Castro

  

  • 2 Oct 2016 9:59 AM

  Hello Evert,
  
  Following is what I would do.
  1. Contact Global Service Center (GSC) and ask for advise/help. Make sure you have at least Foundation support or if it is a newer system a Guardian support before calling. GSC will be able to give you good information on FREMEM and FREETIME. As far as I remember the recommended limit changes when the controller is using CHARMS I/O Card (CIOC) and whether it is redundant or simplex.
  2. Wait until the customer is in maintenance (scheduled or the unscheduled) when they are ready to power off the controllers to be upgraded. Do the upgrade from SD to SX at this time. At least when it fail you are safe because the controllers are not controlling anything.
  3. I am always uncomfortable when doing an online controller upgrade when controller parameters are near recommended limits.
  4. I do not upgrade a controller when it has integrity error or error on one of its components (i.e. I/O, CIOC, WIOC, etc.). Doing the upgrade might mean my job.
  
  Hope this helps.
  
  Cheers,
  Neil Castro

  • Cancel
• Andre Dicaire

  

  • 3 Oct 2016 12:03 AM

  Evert. The recommendation for minimum free time is based on multiple factors that affect the operation of the controllers. The Free time minimum for conventional IO (local cards) is 20%, and is really based on how DeltaV calculates free time for control. This calculation did not change with the release of CHARMs in version 11 and for the SDPlus in version 12. Since CHARMS uses the Ethernet connection to the controller, it added a communication load to the controller, and this could impact the CPU available for control. So by increasing the recommended minimum free time to 28%, this provides a guideline to users for their SDPlus controllers.
  
  Basically, the CPU Free time is calculated based on the actual CPU usage of the control modules and the allocated CPU amount for control. I think it is 70% of the CPU is reserved for control modules. As you add IO, especially CHARMS, you start to use some of this 30% CPU for communications. As you add control modules, you start reducing the CPU FRETIME. Other Tasks like Redundancy update and some overhead tasks also consume some of this 30%. As you add more control modules, you add more IO and redundancy updates. Ideally, you meet in the middle and the controller is full.
  
  In reality, the CPU usage fluctuates and if we calculated the FRETIME based off the overall free time, the number would jump around a lot. So FRETIME is calculated based on the expected available maximum CPU for control (or 70%). When the CPU FRETIME reaches 20%, the calculation looks at actual available free time and if the other tasks have exceeded 30%, the CPU FRETIM will start to reflect actual free time and may drop in a non-linear fashion toward 0. Therefore, the recommended minimum free time of 20% should be used to indicate the controller is "full", and no further control modules should be assigned. With CHARMS, there is an increased amount of communication and the recommended minimum free time was changed to 28%.
  
  What happens if the CPU Freetime falls below 28, or below 20%? We should look some additional parameters. First, are you seeing any CPU Slippage? The On Time Percent indicators for low, medium and High priority modules should be evaluated. If you see slippage, this means the modules are unable to execute at the scheduled, configured time. If there is no slippage, control is performing as expected.
  
  What if there is slippage? That will be addressed when you install the SX Controllers. The Slippage confirms that the controller is starved for CPU and is overloaded. All you can do to address this is to slow down module execution, remove modules or change them to less complicated. Since you are replacing the controller, no need to do anything in this area.
  
  Next you should look at the Time Utilization Chart for these controllers. This is a trend like window that will plot the CPU loading threads, along with the overall CPU Idle time. First thing you are looking for is whether there is Idle time at all. Since you indicate a CPU FRETIME of 17%, I would expect there is some idle time to work with.
  
  IF the On time Percent indicators are at 100% (above 95%), and you see some idle time, and you see CPU FRETIME above 0%, you should expect to be able to replace the standby controller with the SX controller. The behavior you should expect is the same as with the SDPlus controllers.
  
  You indicated that
  " We are experiencing inter controller communication failure for some parameters when we have to download some changes on other controllers."
  Downloading a module requires the controller to accept the download, map data from the old module to the new and confirm all memory references are updated, and this added processing can disrupt other tasks. Unsolicited reporting of parameters to other controllers falls at a lower priority than the download update task. There may be other things happening and we don't know what type of parameters, how many and whether they are related to the modules downloaded. However, a switchover is different than processing a download.
  
  Do you know how the system behaves if a switchover were to occur between two SDPlus controllers? The switchover to a new SX controller would be no worse. And given the addition CPU in the SX, it will resolve any current loading related issues.
  
  What happens when a switchover occurs with the SDPlus controllers? You may see some remote parameters with a bad status until the new active controller is able to rebind and send/receive data from those references. The behavior with the SX will be similar, but may be more transparent as it completes the initialization quicker.
  
  The DeltaV controllers are very robust in their operation, and adapt to overload conditions so that control modules continue to execute, rather than stop. The SX controller synchronizes with the SDPlus active controller to receive redundancy link data at the speed the SDPlus sends it. once installed, the active SDPlus controller will detect the new standby and tell the Pro Plus to commission this new partner, and the pro plus will send a Cold Start download to the standby to configure it. Once configured, the SX will begin to receive the redundancy update data for all the modules, and once this is completed it will set the partner available diagnostic.
  
  This will have the same impact on the active SDPlus controller as a normal switchover.
  
  So the question you have to answer is how does this controller behave on a switchover with two SDPlus controllers. If that works without disrupting the control system operations, then replacing the SDPlus standby with an SX should yield the same results. The redundancy switch over will work even if CPU FRETIME is below 20%.
  
  No one is in a position to guarantee that the switchover will not cause a disruption, but if there is a disruption, it is not because you've added an SX controller as the standby, but rather because of how the system is configured to respond to a switchover.
  
  One more comment on redundancy. When a controller's redundant partner is available, the active controller does two things. First, each module performs some additional processing to create the update information for the Standby controller. Then, the redundancy task sends this over the link to the standby. The redundancy task is high priority and is triggered at the end of each module execution. If you compare the CPU FRETIME of a simplex controller before and after a redundant partner is installed, you will see the CPU FRETIME is lower with the redundant controller. This is due to the added processing each module does. If you look at the Redundancy task in the Time Utilization Chart, you will see this task uses CPU and the more modules you add, the more CPU this task takes, as it has to process more data.
  
  In your case, you already have the controllers running as redundant pairs. Replacing the Standby with an SX will not change the loading on the SDPlus. Once the standby is available, you should be able to switch over, and the SX will begin module execution and take over all communication. Note that the SX uses different diagnostics for loading and the CPU FRETIME value is no longer valid. Use the Time Utilization Chart menu choice and this will launch the new Index view. There is a tab that shows you CPU, Memory and Communication % capacity. The CPU % Capacity is a different calculation and provides a better indication of available capacity, and is "good" all the way to 0%.
  
  Good luck.

  Andre Dicaire

  • Cancel
• Mike Link

  

  • 3 Oct 2016 7:46 PM

  In reply to Andre Dicaire:

  As Andre said the upgrade is the same system impact as a normal controller switchover.
  
  Emerson has a KBA that describes what you are trying to do NA-0300-0055: "Procedures for Adding Simplex or Redundant Controllers and Changing Out Redundant Controllers and CIOCs" (Section 4: Procedure for Changing Out a Redundant Pair of Controllers)
  It walks you through the process, and has a few extra checks to do along the way, such as recording the standby controller's MAC address and verifying it gets updated to that of the replacement controller's.
  
  You may also want to run the controller redundancy analyzer which will help you identify some common configurations that will cause problems during the switchovers required. AP-0800-0141: "Information and Download Location of the Improved Controller Redundancy Analyzer Utility." It checks for things such as using Hart variables for control etc.

  • Cancel