DeltaV virtualization and upgrade

 

Service offered: Project Management & Validation

Project intro: DeltaV is Distributed Control System (DCS) provided by Emerson. In this case the system was rated as GMP as it was used by two pilot plants for producing Active Pharmaceutical Ingredients (API).  

Business case: The business benefits of the upgrade project were to improve business continuity, maintenance support and the overall compliance for the system. The hardware, software and overall validation package were very old dating from a previous upgrade done approximately 9 year before. An upgrade was imminent and the added benefits of having it virtualized were obvious since it allowed for high availability but also easy integration with the customer’s backbone network for services such as time synchronization, SMTP, backup service & GxP fileshare access, Citrix secured remote access and added firewall protection.  

Technical setup – the legacy and the new: The legacy setup consisted of a large number of physical servers, networking components and thin clients distributed throughout the two pilot plants. There was limited redundancy in the legacy architecture, besides the networks which were split into primary and secondary.  

In the new setup the legacy system was upgraded where all the legacy physical servers were converted to virtual machines which ran in a virtualized environment on the host servers placed at two different locations. This allowed for higher availability, disaster recovery and HW maintenance with little or no shutdown time required.    

The new setup provided redundancy for the batch executive and the advanced continuous historian servers, ensuring that batches can continue to run and that data can continue to be collected even in the event of a host failure or application crash. 

The new setup was also interfaced with a shared historian server and an RtReports server for generating batch reports. 

Project Management: Alpha Workflows managed the project according with the client’s project execution model. Conceptual, Basic and Detailed Design phases were carried out over which the specifications and project scoping were matured resulting in a better understanding of the project risks, budget needs and timeline. Upon having the contract signed with the supplier, Emerson, the Execution phase started followed by Verification and Handover. Critical were the initial assessment phases for having a common understanding of the scope and the alignment with the pilot plants on the shutdown period required during Site Acceptance Testing and Qualification.   

Validation Management: The validation effort was split into two tracks both of them described in the Validation Plan: the infrastructure and the PCS itself. All gates and prerequisites were described in the Validation Plan and followed throughout the project. A documentation split was also made with Emerson, where their side of the documentation (design, FAT and SAT) was anchored in the Supplier Quality Plan based on their QMS.

Tips and Lessons Learned:

  1. Ensure you have a good DeltaV Admin or System Manager from the client on your side. Since DeltaV is a complex system you need to have the end-user interests represented via a capable and competend System Manager who understand the end user’s needs.
  2. Conducting a Proof of Concept in the Basic Design where the current systems were virtualized helped with understanding the technical obstables ahead and in clarifying the initial requirements with Emerson
  3. The closing window must be agreed to in advance and prioritized by the Management since it will impact the operations and maintenance activities of the departments relying on the PCS. These department normally conduct maintenance activities in the factory closing window and for that they need the PCS which will no longer be available. Negotiating the closing window and committing to it is crucial!
  4. With the complete replacement of the network and thin clients around the factory, the time required for replacing the network cables must not be underestimated. What we carefully considered would take 2-3 full days, ended up taking more than a week, eating precious time from the closing window. If possible, do as much as you can prior to the closing window.
  5. Leverage supplier documentation as much as possible. Emerson has a very good QMS based on which the design documentation, FAT and SAT can be written. Add-on User Acceptance Tests based on the client’s QMS can be done after SAT.