Emerson Exchange 365

Microsemi (AKA Symmetricom) S250 and other devices will suffer a rollback issue on 17-SEP-2022 to 18-SEP-2022 transition

I've seen a few of these still in use.  I need more information than in the customer notification but expect on 17-SEP-2022 UTC 23:59:59 these devices will roll back 1024 weeks (10 bits) / 19.6 years to roughly 2-FEB-2003.  They've been obsolete / de-supported for a number of years.  But as the units have continued to work, it may not be apparent that they will have an impending failure that can disrupt historians, active directory, etc. 

On a DeltaV system, this generally results in non-recoverable, corrupt historical and event data.  Possibly security / active directory problems up to and including the inability to logon to a system and completely losing visibility.  

The manufacturer doesn't give a lot of details, but what I'd be on the lookout for an unexpected time change perhaps as this: 17-SEP-2022 23:59:59 -> 02-FEB-2003 00:00:00, (All UTC) if these devices are still on the network.

https://www.milexia.fr/sites/milexia.fr/files/gps_week_number_rollover_event_7_avril_2019_doc_3.pdf

Randy Pratt

• 11 Aug 2022 7:10 PM
• Cancel

2 Replies

• Andre Dicaire

  

  • 11 Aug 2022 10:58 PM

  thanks Randy. This is a call for action.
  
  What would happen if the S250 was powered down and disconnected until say Sep 20, 2022. Then after powering it up while still disconnected from the DeltaV network, can a user manually set the time to Sept 20 hh:mm:sec and will it then run for another 10 ears? Or is it simply unable to represent time beyond Sep 17? Just asking, for a friend.
  
  Bottom line, users need to remove these devices before Sep 17, 2022.
  
  The DeltaV NTP can be reconfigured to use an App Station/ProPlus to act as the Master Time node until a replacement NTP server can be obtained and commissioned. Also, the App Station may be able to get an IT sourced Time reference to help work through the issue. Or users can manually monitor the time and minimize drift. But with all nodes syncing to the App Station, event records will stay relatively accurate for Sequence of Event investigations and trend correlation.
  
  Note that the App Station NTP master maintains a looser resolution than the local NTP server so systems with critical time requirements across controllers should restore a local NTP server as quickly as possible.
  
  I'd also suggest making sure your history servers have their datasets exported and saved so that if you have an issue, you don't lose any historical data. It's unlikely Continuous Historian or Event Chronicle have current data sets registered and online from 10 years ago. But any current data will be lost if the historian time jumps back 10 years as there will be no available data set to write to, I suspect.
  
  The Advanced Historian based on OSI PI handles time differently and if an Archive file exists, it will try to insert the data in the older archive, corrupting the archive with time stamped new values that overlap the valid historical data. Backups would restore the old data, but the new data will be lost. Also, since the data is back in time according to the last "valid" value before time change, every value is now written to the archive as "back in time " data, disabling any compression. that will resolve when the server's time is restored and moving into the future. (Setting the server time forward in time and then back to current time will cause the same issue. Don't do it)
  
  The event chronicle uses SQL database datasets and events are stored in the active dataset. Events are stored with an Ordinal number that tracks the order events are received. But the time span of the dataset will be confused at best. The PHV tools would likely have issues and in the end, the data set would be out of time and useless.
  
  Forcing the historians to new active datasets or archives and saving all such datasets and archives prior to the 17th will allow recovery if the server's need to be rebuilt. My thought is to disable the NTP server and run of an App Station until replacement NTP server can be brought on line, and deal with any time drift at that point, which will be a tiny slice in time, not 10 years.
  
  The Active Directory has time related checks. I think its called Tomb stone that checks to make sure the DC's are working properly and updating. That will not be a good day. If the AD is lost/corrupted, restoring can be mean rebuilding a new AD and all computers will need to join the new Domain, even if you name it the same.
  
  Seems to me the only viable option is to disconnect these NTP Servers and order replacements and manage the time through App station/ProPlus until it/they arrive.

  Andre Dicaire

  • Cancel
• Randy Pratt

  

  • 12 Aug 2022 12:47 PM

  In reply to Andre Dicaire:

  Going by general behavior of GPS time, the devices affected by this issue will never be usable again. The original GPS setup used a 10 bit week counter, that's 1024 weeks or roughly 19.6 years The current standard is 13 bits. The last I was aware of, only one of the current GPS constellation is13 bit capable. Manufacturers will often design their software to run an offset from the GPS satellite week counter. So if I introduce a new GPS time server at week 512 per the GPS satellites, I'll take the info coming from the GPS satellites, subtract 512 and get a 19.6 years to end of life for my device. As such, I expect the devices affected by this issue can only supply accurate time between 02-FEB-2003 and 18-JUN-2022 (or write you own NTP client and add/subtract the appropriate number of weeks to what your GPS server tells you).
  
  Without more details of the firmware that's my assessment/educated guess. Just about every GPS device out there has some sort of kludged code to deal with the 1024 week roll over that does not coincide with the actual GPS satellite rollover. GPS time format to NTP time format is a whole other nightmare of brute force conversion even when things work properly.
  
  Other things to lose sleep over. My understanding is that all GPS / NTP time server devices are set up to handle leap seconds as the earth's rotation slows down. But apparently the earth's rotation has unexpectedly sped up a bit lately. So there is speculation about subtracting a second sometime in the not to distant future. So it'll be something like 23:59:58 -> 00:00:00 one of these times. Problem is some time server devices count leap seconds to "guess" which 1024 week epoch the system is in and this subtracted (anti-leap?) second may cause a bit of havoc on various time server software implementations as well. Keep an eye out for that, hopefully it won't happen.
  
  I really hope this is not a surprise to anyone. We've been putting information out about the EOL of the S250 for several years now. Yet I know of systems that still had them just a month or two ago.
  
  Of note: KBAs NK-2100-0387 & NK-1900-1231. if you have the right hotfixes on v13 & v14 and are configured properly, the DeltaV supplied NTP daemon should deal with unexpected large swings, but in doing so shuts down the NTP functionality and the system will then be unsynchronized / free running time-wise. Left unattended, you might not notice a problem on 18-SEP, but about November when your system is 10 minutes off, you'll know then. Without the hotfixes, the system time will follow the week number roll over back 1024 weeks.
  
  As Andre notes, get these things off the network if you have one!

  • Cancel