Outage in Pantheon

Solr 8 Schema Reversion and Performance Issues

Resolved Minor
November 12, 2024 - Started about 1 month ago - Lasted 6 days
Official incident page

Need to monitor Pantheon outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Pantheon, and never miss an outage again.
Start Free Trial

Outage Details

Between November 5 and November 8, 2024, a critical security update was applied to Solr 8 across our platform. During this process, a subset of customers experienced issues with their Solr schema reverting to a previous version. This schema reversion may lead to challenges such as limited indexing of new content and disrupted search functionalities. We believe we've discovered the source of the issue affecting schema stability, and an update to the Drupal Search API Pantheon module will be available soon. In the interim, if you encounter any Solr indexing or search-related issues, please reach out to our support team for prompt assistance.
Components affected
Pantheon Customer Sites
Latest Updates ( sorted recent to last )
RESOLVED about 1 month ago - at 11/18/2024 08:00PM

We are moving this incident to resolved as the immediate impact has been addressed. Below is the final update summarizing the current status, resolution steps, and ongoing work:
Current Status
The release of the Drupal module search_api_pantheon v8.2.1 has resolved the critical issue impacting Solr schema reverting. Customers who have upgraded to this version and re-posted their Solr schema should no longer experience issues following server reboots.
Next Steps for Customers

Upgrade Module: Ensure your site is using the latest version of the search_api_pantheon module (v8.2.1). Release notes here.
Post Schema: Use the module admin interface or Drush to re-post your schema to the Solr server.

Dashboard path: /admin/config/search/search-api/server/pantheon_solr8/pantheon-admin/schema
Drush command: drush search-api-pantheon:postSchema
If you have any questions or need assistance, our Support team is available to help.
Ongoing Engineering Efforts
While immediate issues have been addressed, Pantheon engineers are working to ensure that schemas persist during major platform updates. No further Solr 8 updates are planned until this enhancement is complete.
We appreciate your patience and collaboration throughout this process. If any related concerns arise, please don’t hesitate to reach out to our support team.

MONITORING about 1 month ago - at 11/15/2024 10:47PM

We are pleased to announce the release of version 8.2.1 on Drupal.org, which addresses the critical issue affecting the Pantheon Search API Admin page. Apply this update to resolve the issue and restore optimal performance.

For reference:
https://www.drupal.org/project/search_api_pantheon/releases/8.2.1

During our testing of the module updates, we identified an additional edge case that could cause content reversion. As part of our ongoing efforts to resolve this, we are implementing an update to the Pantheon internal services to address this behavior.

If you have any questions or need assistance with this update, our support team is here to help. We appreciate your patience and understanding as we work to ensure the best possible experience.

MONITORING about 1 month ago - at 11/15/2024 09:37PM

We are actively monitoring the situation and working closely with our Engineering team to resolve a critical issue impacting the Pantheon Search API Admin page.

Issue Overview:
A fatal error has been identified in certain scenarios after applying configuration to the Solr server.

Error Message:
Uncaught PHP Exception TypeError: "Drupal\search_api_pantheon_admin\Form\PantheonSolrAdminForm::getViewSolrFile(): Argument #2 ($contents) must be of type string, array given, called in /code/web/modules/contrib/search_api_pantheon/modules/search_api_pantheon_admin/src/Form/PantheonSolrAdminForm.php on line 82" at /code/web/modules/contrib/search_api_pantheon/modules/search_api_pantheon_admin/src/Form/PantheonSolrAdminForm.php line 108

We are in the final stages of developing a patch to address this error and will release it as soon as it's ready.

Please rest assured that resolving this issue is our top priority. If you encounter any further challenges, our support team is here to help. Thank you for your patience and understanding as we finalize the solution.

MONITORING about 1 month ago - at 11/15/2024 12:46AM

We are continuing to monitor for any customer reports of success or issues with the new module version.

For reference:
https://www.drupal.org/project/search_api_pantheon/releases/8.2.0
https://docs.pantheon.io/release-notes/2024/11/solr-module-8-2-0

Our support team is here to help with any questions or guidance needed. Please don't hesitate to reach out for further assistance.

MONITORING about 1 month ago - at 11/14/2024 09:45PM

We have published search_api_pantheon v8.2.0 on Drupal.org, which includes a critical patch to resolve Solr performance issues and prevent reversion to previous versions. Customers experiencing these issues will need to apply this update and subsequently post schema to the Solr server from their website through the module admin UI or with Drush.

For reference:
https://www.drupal.org/project/search_api_pantheon/releases/8.2.0
https://docs.pantheon.io/release-notes/2024/11/solr-module-8-2-0

Our support team is here to help with any questions or guidance needed. Please don't hesitate to reach out for further assistance.
Thank you for your patience as we worked to deliver this essential improvement.

IDENTIFIED about 1 month ago - at 11/14/2024 02:32AM

A pre-release branch of the search_api_pantheon module has been pushed to the 8.1.x-dev branch on drupal.org. With this update, a core reload is called following a schema post from the dashboard (i.e. "/admin/config/search/search-api/server/pantheon_solr8/pantheon-admin/schema") or with Drush ("drush search-api-pantheon:postSchema"). This branch does not yet trigger a reload when "drush search-api-solr:reload" is invoked.

To avoid Solr schema reverts, customers can install the new branch of the module on their site with:

composer require 'drupal/search_api_pantheon:8.1.x-dev@dev'

And then post their schema update in the admin dashboard or with Drush. With the core reload attached to posting the schema, sites should no longer see schema revert.

We are continuing to work towards a stable release of this fix that will include reloads through search-api-solr.

IDENTIFIED about 1 month ago - at 11/13/2024 10:16PM

We're making steady progress on resolving this issue and will provide another update in about 6 hours, or sooner if we have additional information.

Thank you for your patience as we work through this.

IDENTIFIED about 1 month ago - at 11/13/2024 04:13PM

We are continuing to work on a fix for this issue and making progress. We'll post another update in 6 hours, or sooner if significant progress is made.

IDENTIFIED about 1 month ago - at 11/13/2024 10:26AM

We are continuing our efforts to fix this issue and will provide updates within the next 6 hours.

IDENTIFIED about 1 month ago - at 11/13/2024 03:51AM

Our team is actively working toward a solution for this issue. We'll post another update in 6 hours, or sooner if significant progress is made.

IDENTIFIED about 1 month ago - at 11/13/2024 01:20AM

We're making progress on a fix for this issue and will provide another update in 2 hours, or sooner if more details become available.

IDENTIFIED about 1 month ago - at 11/12/2024 11:02PM

We are continuing to work on a fix for this issue. The next update will be in 2 hours or as soon as we have more details.

IDENTIFIED about 1 month ago - at 11/12/2024 09:57PM

Between November 5 and November 8, 2024, a critical security update was applied to Solr 8 across our platform. During this process, a subset of customers experienced issues with their Solr schema reverting to a previous version. This schema reversion may lead to challenges such as limited indexing of new content and disrupted search functionalities.

We believe we've discovered the source of the issue affecting schema stability, and an update to the Drupal Search API Pantheon module will be available soon.

In the interim, if you encounter any Solr indexing or search-related issues, please reach out to our support team for prompt assistance.

Need to know when vendors go down? You’re in the right place

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3278 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime