Vault Service Disruption affecting all PODs in Region Veeva Align, Veeva Vault, Veeva CDMS, MyVeeva (ePRO, eConsent), Veeva SiteVault, Vault CRM

Veeva Vault PODs  Veeva Align, Veeva Vault, Veeva CDMS, MyVeeva (ePRO, eConsent), Veeva SiteVault, Vault CRM are currently experiencing a service disruption. Veeva engineering teams are working to return the service to normal as quickly as possible.

We are continuing to investigate the issue.  We are in the process of reinitiating the Vault service across the regions.  Another update will be provided within 30 minutes.  Thank you for your patience.

We are currently performing tests across the regions.  Vaults will remain in maintenance during this initial testing period.  As testing completes, Vault services will be resumed.  

We are very sorry about this Vault side outage and are working as hard and fast as we can.. AP , EU and US West service is currently available. We are continuing to see issues, especially with US East.I will post again within 15 minutes. - Avril England (General Manager, Vault)

We are continuing to see authentication service spikes from US East and are working to simultaneously isolate the issue and stabilize the service.   AP, EU and US West remain online for now.  East remains unstable.Next post will be at 8:30am Pacific.  - Avril England (General Manager, Vault)

As of 8:20 a.m. Pacific, production service has been restored across all regions and appears to have stabilized. Sandbox service is offline for US East while we continue to troubleshoot the issue.There is no evidence to suggest that this is related to a DDoS attack.  However, we have been seeing an increase in the number of calls, which in turn is creating a backlog that is not dissipating normally.  This backlog prevents authentication requests from being processed on a timely basis from the user activity on the Vaults, resulting in a Vault outage.Please know that we are treating with the utmost urgency across Veeva and recognize the significant impact to your businesses.  Next update at 8:45 a.m. Pacific.Avril England, General Manager Vault

US East PODS are still seeing some instability and users are experiencing slow / unresponsive Vaults. US East sandboxes remain offline.  All other regions are currently stable. We are focusing our efforts on US East production at this time.Next update at 9:00 a.m. Pacific.Avril England, General Manager Vault

AP, EU and US West Vaults remain online and available.US East Vaults are now online but some Vaults had continued to experience the lingering effects of the outage.  These lingering issues should now be addressed. The team is still carefully monitoring all production operations for further issues. Next update at 9:15 a.m. Pacific.Avril England, General Manager Vault

All Vault regions are online with production Vaults. US East Sandboxes remain offline.The team continues to monitor the load and it appears to have stabilized.Next Update at 9:30 a.m. Pacfic. Avril England, General Manager Vault

All production vaults remain stable and available.We are now going to begin bringing US East sandboxes slowly back online.This slow reintroduction will be done with the goal of identifying which, if any, Vaults are creating unusual load on the authentication services. I will provide the next update at 10:00 a.m. Pacfic.  -Avril England, General Manager Vault

Service remains stable for all production vaults in all regions.The US East sandboxes are being slowly back online. As of 10am Pacific, we are approximately 15% complete with restoring service to US East sandboxes.Sandboxes in all other regions are operating normally.The team is currently working on (1) restoring remaining sandbox service, (2) analyzing data for root cause and (3) evaluating changes to protect the service while root cause investigation continues.I'll provide the next update at 10:30 a.m. Pacific. - Avril England, General Manager Vault

Service remains stable for all production vaults in all regions.The US East sandboxes are approximately 50% back online now.  We are continuing the gradual restoration of service to the remaining 50%. Sandboxes in all other regions are operating normally.Given the criticality of the situation, establishing root cause and sharing it with you is of the utmost importance. I will post information regarding root cause, mitigation strategies and preventive measure as soon as they are known.  An official RCA / Incident Report covering all three incidents will be available 10 working days after the root cause has been established.  I'll provide the next update at 11:00 a.m. Pacific. - Avril England, General Manager Vault

All Vaults are back online across all regions.Mitigation strategies for peak times, while this issue persists, are under review and I will share them as soon as we finalize the plans for tomorrow.Root cause analysis also continues with urgency.I will post my next update at 12:00 Pacific. - Avril England, General Manager Vault

The Vault service is fully operational across all regions for production and sandbox vaults. Our engineering teams remain focused on investigating root cause and exploring mitigations for system stability while analysis continues.We are in the process of scheduling a customer zoom call with Veeva's senior leadership for tomorrow.  I'll post the information with the next update at 1:00 p.m. Pacific. - Avril England, General Manager Vault

The Vault service is fully operational across all regions for production and sandbox vaults. For tomorrow, Thursday, April 25,  Vault Sandboxes will be offline during the peak period from 2 a.m. - 9 a.m. Pacific.  This is not a fix for the load issue, but rather a mitigation effort, with the goal of reducing load on Vault's authentication service. We are also hosting a zoom meeting where Peter Gassner, Veeva CEO and I  will provide an overview of the outage details and corrective actions.  We will also be available for questions as part of that call.  Please register here: https://veeva.zoom.us/webinar/register/WN_XzbLJcigTMidsumr4263eAI will not post further updates today, unless the status of the Vault service changes. Again, thank you for your patience as we work through this issue. - Avril England, General Manager Vault