Our incident management team have now reached the final stages of this incident and are processing the current news mention backlog. We shall be doing this over the course of the next couple of days so that normal operation remains unaffected. The next update on this incident will be by 1700 UTC/UK on Friday 12th November.
We are nearing resolution for the issues currently impacting news mention processing and expect to be able to resume updates as early as tomorrow morning. We expect the current news mention backlog to take some further time to process once the stream is up and running. We will post an update tomorrow with our progress at that time.
We can confirm that the data lost from the original incident has now been recovered and fully synchronised across our databases. Unfortunately during this recovery process, it had an unexpected impact on our news processing, and as result we are still behind in processing news mentions from November 2nd onwards. The team are working to catch up and we will have another update on Monday 8th November. We thank you for your continued patience while we ensure the stability and integrity of our data.
Our incident management team have completed the recovery of the missing data and are now synchronising this data across our databases. Doing this while our data is continuously changing means doing so slowly however we have reduced our recovery time objective. We had thought that it could take up to another week to fully recover and test all aspects of the data, we've revised this estimate to the end of this current week. Our teams will continue to work until the incident is resolved and if anything changes in the meantime, we shall update this page. Thank you for being patient while we ensure the stability and integrity of our data.
Our incident team have continued to work on validating the accuracy of our data following the incident. We identified: Up to a 2 hour period from 25th Oct that has now been resolved Up to a 7 hour period from 20th Oct The outstanding data from the 20th, requires our teams to rebuild the data in an alternative environment for testing prior to repeating in Production. This is a slow process of recreating and manually checking, which is likely to take us at least 2 weeks and we'll continue to report our progress here. We would like to apologise again for any impact that this incident has had on our users.
The system has successfully worked through the backlog of data, and all data has been successfully synced apart from a small 2 hour period that will require some manual curation to re-sync. We are setting out a plan to conduct this work and will continue to keep this incident updated with progress.
The system is continuing to work through the processing backlog and at current rates we hope to be back up to date within the next 24 hours. This means that vast the majority of data will be correct between the Details Pages and the Explorer once our daily snapshot process has completed on Wednesday 27th October. We will update again tomorrow morning (Tuesday 26th)
Our major incident team has identified a backlog of processing which is not expected to clear before early next week. Non-essential processing has been paused to allow the Altmetric Explorer processing to take priority. Incremental updates will be posted here as the become available before a full update before 1700UTC/1800BST on Monday 25th October. We're very sorry for any users who are impacted. The Details Pages and Details Page API remain available throughout.
Our major incident team have reconvened this morning to review the overnight processing progress and are assessing the likely recovery time for Altmetric Explorer data. The impact on our Explorer services remains the same and we expect to provide a further update before 12:00UTC/1400BST 22nd Oct
We have identified the root cause of the issue which relates to a backend service responsible for preparing the nightly snapshot used to populate the Altmetric Explorer database for our Explorer application and API. Our teams are working to identify the quickest way to restore full access to the latest data while safeguarding the availability of the service. We would like to apologise for the inconvenience to our users and will provide a further update before 0900UTC/1000BST 22nd Oct.
We are currently investigating the root cause of an issue which is causing: New mentions are not visible in the Altmetric Explorer since 23:59 on 19th Oct Research outputs that have not previously been mentioned, have not appeared in the Altmetric Explorer since 23:59 on 19th Oct The Altmetric score on Publisher badges may not match the score within the Altmetric Explorer Altmetric searches may not be as performant as customers would normally experience The explorer database is usually updated on a nightly basis, this means that one update is currently missing for customers.
"I spend 2 hours trying to solve an issue and then realize it's due to an [EXTERNAL SERVICE] outage"
Every engineer at some point in time
It has never been easier to understand the outages in your external cloud services.
All of your service statuses in one place
Check the status page aggregated of all your services in one place. No more going to each of the status pages and managing them individually.
Notifications of incidents in real time
We monitor 24 hours a day, 7 days a week and will notify you if there is an incident. No more wasting time trying to figure out why something isn't working.
Notifications in your favorite channel
Get instant notifications in your email, Slack, or Discord when we detect a service outage.
Keep track of scheduled maintenance
Never again be caught off guard by unexpected maintenance from your services. A feed of the next scheduled maintenances is available.
Set the notification level for each service
Configure which notifications you want to receive from each service. You can choose to receive notifications for all incidents, only critical incidents, or just display them on the dashboard.
Integrate with your current workflows
Using Zapier or Webhooks, you can easily integrate notifications into your processes.
Receive a Weekly Digest
Every Monday, you'll receive a weekly summary of what happened the previous week as well as the maintenance schedule for the following week.
Multiple Dashboards & Profiles
Create one dashboard for each of your teams. Monitor only the services that each teams uses. Dedicated dashboard with custom notification settings.
Only receive notifications for specific components
Filter notifications by service components. You can opt to receive notifications only when a specific component is affected.
You already monitor your internal systems. What about the external services? Monitor the services your business depends on. Don't waste time looking elsewhere when external outages are the cause of issues.
Detect external outages before your clients tell you. Anticipate possible issues and make the necessary arrangements. Having proactive communication builds trust over clients and prevents more work.
5 minutes to set everything up
Start with a trial account that will allow you to try and monitor up to 40 services for 14 days.
There are 1742 services to choose from, and we're adding more every week.
You can get notifications by email, Slack, and Discord. You can also use Zapier or Webhooks to build your workflows.
You'll start getting alerts when we detect outages in your external dependencies! No more wasting time looking in the wrong place!
Increase the productivity and efficiency of your team. Enable monitoring for your services, and start receiving real-time alerts when your external dependencies have outages.Start today for FREE