AWS outage continuing to have some impact on Learnosity services in VA (us-east-1)

As of 21:30 UTC, all systems keep operating normally, so we are resolving the status page about the AWS outage affecting Learnosity Services in the VA (us-east-1) region.

Learnosity Support and Systems Engineering teams will continue to monitor the systems closely while AWS continues to work on the issues on their side.

Please reach out if you have any questions or concerns.

As of 20:40 UTC, the error rate remains low, the infrastructure is scaled to meet demand, and the scoring queues are empty and new submissions are being processed as they come in.

All services are operational and we are processing the remaining backlog of messages from the incident.

AWS still lists 93 affected services and a severity of "Impacted".

Learnosity Support and Systems Engineering teams are continuing to monitor the issue, but at the moment all systems are fully operational.

As of 19:30 UTC, the error rate has fallen to 0.1% and continues to fall, we continue to steadily process the backlog of scoring messages.

As of their latest update AWS has downgraded the severity of outage to "Impacted" but still list 109 services as affected.

Learnosity Support and Systems Engineering teams are continuing to actively monitor the issue, and will follow on with an update and resolution as soon as possible.

As of 20:00 UTC, the error rate has continued to drop, systems remain scaled and the backlogs of scoring messages continues to drop.

AWS is still reporting 109 impacted services and a severity of "Degraded"

Learnosity Support and Systems Engineering teams are continuing to monitor the issue and are manually adjusting the infrastructure where needed to ensure stability.

As of 19:10 UTC, we are beginning to see errors rates diminish and some Learnosity services are beginning to scale again.

AWS is still reporting a severity of "Degraded" with problems to 104 of their services in us-east-1.

Learnosity Support and Systems Engineering teams are continuing to monitor and manually intervene when possible to mitigate the effects of the outage.

As of 15:25 UTC, we are seeing increasing error rates.

Data API is the most affected service at the moment and we are seeing an increase in scoring delays, but we are continuing to score incoming messages to the best rate our current capacity allows.

AWS is still reporting a severity of "Degraded", but now with an increased count of 81 affected services.

Learnosity Support and Systems Engineering teams are continuing to monitor the issue, and will continue our mitigation efforts wherever possible.

Learnosity services in VA are continuing to be partially impacted by the major AWS outage.

AWS first reported the incident at 7:11 UTC As of 14:50 UTC they are continuing to work on its full resolution, but have again set their impact level to "Degraded".

The AWS Health Status dashboard where updates are being posted by the AWS team can be found here: https://health.aws.amazon.com/health/status

Overall, the error rate has fallen and has stayed below what we saw earlier in the day and we have been successful in slowly scaling up the infrastructure, but we're still seeing some issues stemming from the the ongoing AWS outage as we attempt to bring online some new instances, and we have begun to see an uptick in errors.

These have included errors associated with Data API, Feedback Aide, and the delivery of adaptive assessments, delays in scoring and video encoding, as well as other smaller spikes across different systems.

While the overall error rate remains low, Learnosity Support and Systems Engineering teams are actively monitoring the outage and manually intervening where possible to mitigate the issues we're facing.

In spite of our best efforts it is likely that the APIs may not be fully operational in VA until AWS's service is fully restored to normal operation, but we are taking every step within our power to try to minimize the disruption to our services.