Emerson Exchange 365

• Not Answered

Corrupted database restore on a running process

Has anyone attempted to restore a backup database on a live (running) process?

Running a virtual system on DeltaV 13.3.1.

We have a corrupt database and are in need of restoring the database from a backup.   We can download some of the controllers, but others will require coordination with operations and/or wait for unit outages.  We want to know if anyone has attempted this process and what the outcome was.  

Leon Gonzales

• 19 Jun 2020 7:58 PM
• Cancel

17 Replies

• David Scott

  

  • 23 Jun 2020 6:22 AM

  In reply to Andre Dicaire:

  Leon,
  
  While the idea of disagreeing even slightly with Andre terrifies me (having followed and learned from a great number of his posts, and recognising his knowledge and experience on the subject), I wanted to chip in with our experience from doing online upgrades.
  
  Regarding changes to Named Sets, this normally only requires downloading Setup Data to all workstations & controllers, but does not require a Total Download (I say this from experience - almost all the work we do is on systems which run 24 hour so we don't have the 'luxury' of waiting for downtime!).
  
  In terms of importing an FHX file into a clean database on a live system, the one thing that is key is that when you get a prompt regarding replacing existing items (I think the first prompt is for 'TheSite'), you MUST click 'Yes to All'. This replaces all existing definitions in the database with those from the backup - if you don't do this then something as minor as the built-in AND function block having a slightly different timestamp will require you to download your entire system. If all items are replaced, then the imported database should be identical to what you have running. If you have made minor changes since your last backup (e.g. to a small number of modules) then an Update Status should show these with blue triangles as it normally would.
  
  It is worth downloading Setup Data to all workstations following a full import, but it should not be necessary to do Total Downloads. Again, I am speaking from personal experience of doing an online upgrade of a live DeltaV system without needing to do a single Total Download.
  
  Andre - apologies for disagreeing with you, and please feel free to point out what I have missed!

  • Cancel
• Andre Dicaire

  

  • 23 Jun 2020 10:48 PM

  In reply to David Scott:

  David, No apologies please. I'm glad to see your inputs. I got a little sidelined on the FHX vs Database restore thing, based on some comments. A database recovery and bringing the database current to what is running in the controllers can have a wide range of challenges, all related to how recent the backup is and the extent and type of changes made that must be reapplied. I can think of ways to use the available information such as the powerup folder, but I haven't located any KBA's that go beyond the initial recovery of the database. There may be some, but I did not find them in a quick search yesterday.
  
  In order to do any partial download after recovery, we have to ask if the download will "take away" something the controllers are using. Lets say we added a Named set, used by a module in a controller. If we add that named set, and it matches it's previous structure, including named state text, we should be good. But if we added two Named Sets, what happens if we add them in a different order than we did originally? Each named set is indexed as it is added. If we create them in reverse order, their indices are swapped. Similarly for the Alarm Types, Plant Areas and such.
  
  IF we do a total download, it doesn't matter as everything will move to the new order (Star Wars reference). But in order to download this to the runtime, and only to a subset of nodes, we have to make sure the changes are applied consistently. Not all changes are impacted by the order in which they are mode. Modules can be added in any order as they are named items and are not indexed in the module table.
  
  However, I took a look at the runtime Module table in v14 and see that it now has a "DOID" for each module, which is a global unique Id. I'm not sure how this DOID is used. If I had added a module and had to recreate it, it likely has a new DOID. Once I download the Setup data to workstations, the new MODT.SCR will have the new OID. What does this mean to the running module? It may not matter, but I don't know, so I'll be trying to figure that out. On a bigger note, it doesn't matter because you need to recreate the module. The question is would I need to download the module explicitly? I guess, I'd be downloading the module anyway since I need to align it with runtime.
  
  Getting back to downloads, I would think that once you have recovered and recreated the configuration, and you are comfortable that the changes are all valid and Setup data is accurately recreated, partial downloads would work, and all will work. If your standby controllers are all configured and available, and you don't see any unresolved errors in diagnostics, and all displays are healthy and correct, you might try a switchover. If the Standby comes up configured and available, that is a good sign. But if it doesn't, you'll need to do a total download to resolve whatever discrepancy exists.
  
  It all depends on the extent of the recreated configuration and how it is implemented to align with the lost database. The more you have to recreate, the more chance something is not perfect and the consequence might me minor or significant. Each situation brings its specific challenges.
  
  I guess I'd sum it up as saying:
  If you can do total downloads, do them and move on. The database and runtime are synced and you know it.
  If the process is up and can't be interrupted, but you can wait and do total downloads, then wait. Use that time to reconfigure what is missing.
  If you cannot do total downloads, but need to implement changes and move forward, proceed cautiously and confirm everything aligns with Powerup folder download scripts, especially the global data. You will need to download the modules and items to recreate. Partial downloads or modules and Setup data and manually updating uploaded parameters can get you there, but as soon as you can, perform the total downloads.
  
  Just my opinion.

  Andre Dicaire

  • Cancel