Emerson Exchange 365

An entire control system crash is nearly unheard of. But it did happen. The incident you are describing was due to a software issue. Basically a 1-256 issue in the database vs. a 0-255 issue in the communications layers of DeltaV (or similar, but this simplifies what happened so we can understand...). When a full name set of 255 was configured, index 256 was allocated in the communications layer, which didn't have a place for that last value. As soon as auto-update / yellowpages distributed that named set to all nodes, every node (computer and controller) experienced an allocation failure in one of the communication tasks and that task failed. From the outside view, everything crashed.

I was one of the support engineers that took that call and helped determine the root cause. The system was in staging and not yet on production. To the best of my knowledge, this only happened the one time. I wouldn't call it extended plant downtime as the customer was able to tell us he'd just changed a named set and as we looked at the configuration, we noted a set was 'full' and it took very little time for us to suspect that was the issue and test / duplicate the symptom and offer a workaround. This issue was published in KBA AUS1-115-010423161631 Addendum to V5.3SP1 / V5.3.1 Release Notes as issue 33294. (23-MAY-2001).

Many customers mitigate the unintended behavior risk by having a separate development system to test configuration changes before introducing changes to the production system. I get that it may not be possible to have a development system. One has to weigh the risks vs. the benefits to justify that cost.

I'm not a configuration expert by any means, so I am not in a position to offer what is or isn't likely to happen on a name set change. But the incident you heard of did happen and resulted in design and test changes so that something similar shouldn't make it to released software ever again.