Metadata indexing failure and instability on the REST API (Starting Friday 12th 20:00 UTC and continuing) affecting several other tools

We have updated the service that synchronises new and updated metadata records with the REST API; all three pools of the REST API are performing well as a result (as are the services that run on it). Thus, we are resolving this incident. -IF

We've identified and fixed one issue which was affecting citation counts. We're still working on two additional tasks: upgrading Elasticsearch in the REST API and updating the service that synchronises new and updated metadata records with the REST API. -GH

The public and polite pools are still experiencing some instability, which is affecting downstream tools such as Participation Reports, Search, and Crossmark. We're working on a fix incl. infrastructure upgrades that should help long-term.
-GH

We are continuing to monitor for any further issues.

The public and polite pools are still showing as unstable with long response times and timeouts occurring. We are still looking into this as a priority and will post with more information as we know it.

A fix has been implemented and we are monitoring the results. Do to the heavy workload of indexing a large volume of records yesterday, we have observed longer response times in the Public and Polite pools of the REST API. We believe those response times should improve throughout the day. -IF

Our technical team is re-pushing all of the records that have failed to be indexed since Friday, 12 September. Those records should be indexed in the REST API in the next 24 hours. -IF

We are continuing to investigate this issue.

We have found that some records failed to be indexed by the REST API on Friday 12th September, and continuing. Our technical team is investigating.