Possible delays affecting the availability of session data in US-East-1 (VA) region

As of 3:10 UTC, a further 30 minutes of monitoring has confirmed that all delays affecting the availability of session data in the US-East-1 (VA) region have been eliminated and all functions are operating normally.

Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalised any next steps or preventative measures required.

As of 2:40 UTC, availability of session data is fully restored, the message backlog has been eliminated, and load balancing is performing normally. Data API, Reports API, and Firehose streams all operating as expected.

No impact was seen in authoring or assessment data flows, and persisting of learner data was also unaffected.

Learnosity Support and Systems Engineering teams are continuing to actively monitor the issue, and will follow on with an update and resolution as soon as possible.

At approximately 2:10 UTC, Learnosity engineers identified an apparent scaling issue and have manually scaled available server instances, to immediately impact message backlog. Queues are steadily reducing and we are actively monitoring.

Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

2:05 UTC. We are continuing to investigate possible delays affecting the availability of session data in the US-East-1 (VA) region. Firehose messages and Data API queries started showing slower than normal response times at approximately 1:20 UTC.

Learner submissions, persisting data, and scoring remain unaffected, as do all authoring and assessment APIs which continue to operate normally.

Learnosity Support and Systems Engineering teams will follow on with an update and resolution as soon as possible.

As of 1:40 UTC, we are investigating possible delays affecting the availability of session data in the US-East-1 (VA) region. Scoring is not affected, nor are any authoring or assessment APIs..

Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.