Vault Performance Degradation on POD VV1-1169

[in progress]
We are proceeding with the deployment of a fix tonight ~9:00 pm PT to resolve today's performance degradation. This update optimizes the problematic queries to help restore full stability.The Safety Maintenance Release page has been updated with additional details.  Email updates to those that are registered for Service Availability notifications will receive an email within the hour.  We will resolve the incident ticket once the deployment has completed.We sincerely apologize for the disruption this has caused to your operations. We recognize that system performance is critical to your daily workflows, and we understand the frustration that this instability has created for your teams. Thank you for your continued patience as we implement this resolution.

The Vault Engineering team has confirmed that a resolution has been identified and is currently being implemented and tested.  Once testing has been completed, we will deploy a maintenance release tonight (Pacific Time).  The exact time is still to be determined.  Another update will be provided within the next 2 hours.  Thanks for your patience.

The Vault Engineering team has pinpointed the cause of the ongoing performance degradation.Code changes introduced in the 25R3 General Release in certain circumstances led to database queries to fetch all records in two objects that were found to be involved.  Given the size of the two objects in question, this strained the search infrastructure and led to the observed performance degradation.An imminent maintenance release will constrain the searches and safeguard Vault resources. A subsequent update will confirm the timing of the maintenance release.  A forthcoming Root Cause Analysis will provide details of this incident.  We apologize for the impact and thank you again for your patience on this matter. 

The Veeva Engineering team is currently investigating the potential root cause of the performance issue.  We are expecting to provide a fix within 24 hours.  We will provide more details and a timeline update within the next 2 hours.We apologize for the impact and thank you for your patience on this matter. 

We have recently provisioned additional server capacity to mitigate the performance issues. Users may experience intermittent slowness while this scaling operation is in progress.  The root cause is currently under investigation.Thank you for your patience as we continue our investigations.

Veeva Vault POD VV1-1169 is currently experiencing intermittent degraded performance. Veeva engineering teams are working to return the service to normal as quickly as possible.