Recent Control & Safety Systems Discussions
Similar Posts
Corrupted database recovery
Unable to restore Pro plus database from Restore option available in DV database Administration
PROPLUS CRASH AND DATABASE RESTORE
DeltaV database administration backup and restore
Delta V restore database issue/question
Share
  • Not Answered

Corrupted database restore on a running process

Has anyone attempted to restore a backup database on a live (running) process?

Running a virtual system on DeltaV 13.3.1.

We have a corrupt database and are in need of restoring the database from a backup.   We can download some of the controllers, but others will require coordination with operations and/or wait for unit outages.  We want to know if anyone has attempted this process and what the outcome was.  

Leon Gonzales

17 Replies

  • Bruno DB
    if i remember correct, you don't need to download the controller, important are the checkpoint folders ( i forgot the name ) on the proplus
  • Lun.Raznik

    In reply to Bruno DB:

    On a running system, DeltaV configuration database is not generally involved except for few applicatoins - OPC client as an examle.

    It is hard to say that no side effect but generally you should be fine.
  • Matt Stoner
    Please log a call with the GSC for this issue to ensure you get the system back properly
  • Andre Dicaire

    In reply to Lun.Raznik:

    Agree with Matt and log a call.

    the Powerup folder should be preserved. it holds the download scripts that serve to configure a stand by controller if a switchover occurs.

    OPC clients do not use the database. they connect to the OPC server on the app station. Not sure if the Browse function needs the database, but but configured clients will not need the browser.

    you need to log a call. This not a problem that you solve in a community forum. Many good intentioned suggestions can lead you to a worse situation.

    Andre Dicaire

  • Leon Gonzales

    In reply to Andre Dicaire:

    I do have a call in the system. My LBP is working with us as well as Guardian support. I'm just trying to find out if a restore on a running process has been done before. We were told a total download of the controllers will be required in order to be able to make module changes again. This is the difficult part. Some controllers can be downloaded now, but some are going to have to wait until operations schedule an opportunity. Are there any known issues with running in this state?
  • Bruno DB

    In reply to Leon Gonzales:

    if they are redundant, you can switch the controller and they are loaded properly, but the powerup folder should not be corrupted
  • Andre Dicaire

    In reply to Bruno DB:

    the database generates the Powerup files during a download and these are what is running in the controller. when you recover the database, it likely has some modules that were changed and as such some of your running modules will be different. These changes have to be redone in the recovered database to align. If you also have exports of the changed modules you can import these to bring the database . you have to know this change history.

    If you perform total downloads you will align the recovered database, the Powerup scripts and the controller configuration. when you do this total download is up to you. by recovering and redoing the changes first your total download will have minimal disruption. You can run indefinitely without downloading controllers. And with the Powerup files matching the runtime, controllers will switchover and standby will go configured. Download your controllers when you are ready and able to.

    And yes, recovering a database on a running plant has been done, successfully. The outcome is mostly dependent on the completeness of the Backups and the MOC process, like having an export of each module that is changed.

    Andre Dicaire

  • Leon Gonzales

    In reply to Andre Dicaire:

    Thank you Andre for your reply. Two more items i would like feed back on.
    1. If we have controller to controller cross communication, would there be any issues with downloading one controller and waiting on the second for a later opportunity?
    2. Is there any way to have outputs hold their position during a complete controller download?
  • Bruno DB

    In reply to Leon Gonzales:

    1. Depends on the control strategy, but most of the time you have to manipulate the strategies in the other controllers
    2. This is always the grey zone in DeltaV, all depends how you made the CMs, but it will go to fail state
  • Andre Dicaire

    In reply to Leon Gonzales:

    1. A total download will wipe the existing proxy clients/Servers as well as initialize modules in the downloaded. The Download Verify will show database issues, but not ones that will result from misalignment of runtime modules with the newly downloaded onces. Assuming you can update your restored database so that the requested parameters exist in runtime, these references should resolve after total download.

    Use Diagnostic Explorer to monitor Unresolved references per node, before and after the download. If there are none after, then the references have bound successfully, even Dynamic references.

    2. By design, the Output channels hold their value on a total download. They do not go to fault state and value unless the detect loss of communication to the controller. When modules begin executing, Output blocks will sync to their channel value in Initializing Manual and transition to MAN, and then to AUTO or CAS.

    Cascaded blocks will propagate upstream to initialize to the Output channel value so avoid bumping the IO. However, once blocks are initialized, the output value is determined by the module logic. If Interlocks are configured and they drive action on the output signals, the result will be as per the design of the module.

    With Bus cards and serial data, output references are often configured with External References. These do not have an initialization sequence and the Output values will assume the result of the module logic on the first scan, and write to the Card signal or register.

    On a Database Restoration in a running plant, the concern is how recent is the backup and do we know what is different between the running controllers and the restored database. If you have a strong MOC and know exactly what modules have been changed and what those changes are, you can proceed to redo these changes prior to downloading the controllers. This will result in the least disturbance to the running system. It will also make the delay of some controller downloads a non-issue.

    If we are not sure of which modules underwent change from downloads or uploads, you can use the Powerup directories and sort by date/time to find the modules with dates later than when the Backup was done. Evaluate both the .CM file and the upload override file, both have the Module name as the file name.

    But if you at least know what items were downloaded after your backup, you at least know what items need more attention.

    If you have exports of changed modules that you could recover, you should do this before you download. I suggest you attempt to bring your recovered database as close as you can to what is in runtime before you begin downloads. Armed with a list of modules that are affected by lost changes, you can evaluate those modules and monitor them post download.


    To summarize:
    1. Preserve the Powerup folder as it was when the restore database was started (hopefully you have that)
    2. Identify all modules, Composites and upload data files in the POwerup that are post backup date/Time
    3. Update all modules to the best of your abilities prior to downloading controllers
    4. Make a list of the most critical modules that underwent rework to redo changes and verify these are working as required after the download.
    5. Do a Verify with out download and review the Warnings for anything new or unexpected. Note however that this will not pickup a new reference that is in the runtime, but invalid if you download one controller with the missing parameter.

    6. After Download, use Diagnostics to verify the module Error and Status values prior to download and after. Do you have Unresolved references before and/or after. Investigate any changes in these status values. New Unresolved References likely indicate missing parameter.

    Note that after recovery of the database, the total download is recommended to ensure the database, Powerup and runtime are all aligned. Once you begin downloads, you will want to freeze changes until all controllers are downloaded. The Database verify cannot identify issues between the database and the runtime. It's not that you can't download, but there is added risk you change. Use diagnostics to verify modules status and error bits.

    Follow instructions from the GSC d get alignment. Hope this is useful.

    Andre Dicaire

  • Leon Gonzales

    In reply to Andre Dicaire:

    Andre, can you elaborate on your last note a little more?
    Once we restore the database and begin total downloads of the controllers, we should freeze changes until all controllers are downloaded?
    Example, after we restore the database and have downloaded maybe half of our controllers. Are you saying we should not make any changes to the downloaded controllers until ALL of our controllers are downloaded?
  • Andre Dicaire

    In reply to Leon Gonzales:

    I'm just saying you should keep the problem contained and not introduce other variables.

    I assumed from the use of the term "Restore" that you are restoring from a Database Backup. That is important. if you recovered using a new database and an FHX Export, the new database content will be created in alphabetical order rather than chronological order. This will affect some global items, especially alarm related data.

    For online recovery of a database, you need good Database Backups, and by good, I mean recent so that there is a minimum loss of changes. And you need a good MOC process that allows you to redo the missing work to bring the database current. If those are available, you can recover the database to align with the runtime. Downloads are still recommended to completely sync runtime to the recovered database.

    If you use an FHX export as your backup, you will have to do a total download to all nodes.

    You should consider getting some SureService assistance if you need to do downloads over time and what help confirming Powerup changes and such. That would not be a service engagement covered under Guardian Support.

    Andre Dicaire

  • krishna sen
    If I am using .fhx export file for backup and after taking this export file I am not changing anything in offline database and online in controller then is it necessary total download of controller ? Can I only total download proplus and is it possible to sync proplus  with the controller?   

  • Andre Dicaire

    In reply to krishna sen:

    Good Question. My understanding is that you should perform total downloads to controllers and workstations if you perform a total database recovery using FHX file. If you restore a database backup (it must be recent, with respect to changes) the restored database will be consistent with the runtime. If you Import an FHX into empty database, the newly created database will likely have differences due to the order in which items are created and referenced.

    I recommend always doing Database Backup and FHX export. The FHX allows one to extract information and can be useful for many things. The Backup is your recommended way to restore the system. If the backup is not current you end up having to recreate the missing work and in doing that you again may not create things in the same chronological order, so syncing the recovered database and the runtime system likely requires total downloads.

    Total downloads to the runtime system is the only way to sync the runtime environment with the database with 100% certainty. With a Restored Database and a careful recreation of missing changes, one can get close for usre, and may be certainty. You either need to do some meticulous checking and verification or you can do total downloads. With a restored database that is recent, you can recover and know the changes to validate and move forward with minimal disturbance.

    Do your Database backups. Or be content with total downloads. depending on the process, that is not a bad option.

    Andre Dicaire

  • Andre Dicaire

    In reply to Andre Dicaire:

    So I had a peak at the Database Recovery KBA. It deals only with getting the database recovered, either from FHX or from Backup. It does point out that the FHX can be scheduled while the Database Backup is a manual task. Which means most customers are likely backing up with the FHX.

    So I then had a look at my v14 FHX export in order review what possible issues I could find, which would result in some nuanced differences in the new database versus the a database restored from a backup, and now they might differ from the Run time System. All the items I checked confirme the FHX matched the Runtime download files for all the global data. So I'm gonna walk back my statement about the database restore being better than the FHX. It is quicker, but if an FHX is what you have, that's fine. You have to be careful to ensure there were no import errors, but once imported, items should be correctly implemented.

    The Online nature of the recovery still has to address the "missing" configuration that was done since the last backup.

    When adding global items like Named Sets and Alarm types, these are indexed as they are created. You don't have control over this indexing. As you redo the configuration changes, adding items in a different order can result in different indices from the runtime. This will impact how data is presented or recorded in the workstations. By doing total downloads, everything is realigned. So that is the easiest path to full recovery. If you cannot do these downloads, you must ensure the configuration changes are progressed in the correct order.

    The Powerup directory holds all the scripts downloaded in run time. If you know what to reconfigure, make sure that any global items are compared to the runtime global scripts and that will tell you which Plant area to create first, or which Alarm Type to add first. You can export these items to confirm the index matches what is in the runtime.

    Any parameters uploaded need to be manually applied to the database to align with the runtime. This will set change flags, but there is a way to clear them. The GSC can assist with that.

    Sorry for the confusion.

    Andre Dicaire

This is the official online community site of the Emerson Global Users Exchange, a forum for the free exchange of non-proprietary information among the global user community of all Emerson Automation Solution's products and services. Our goal is to improve the efficiency and use of automation systems and solutions employed at members’ facilities by sharing our knowledge, experiences, and application information.

User Groups | World Areas | Community Guidelines | Legal Information | Privacy Policy | Contact Community Manager

Website translation provided by Website Translator

© 2015-2022 Emerson Global Users Exchange. All rights reserved.