US data center infrastructure issue

AWS's us-east-1 region, where our US data center is hosted, experienced an issue affecting some services Cronofy's platform relies upon.

The impact to service was low, at its peak resulting in a small degradation in performance within our US data center and a handful of server errors being returned by the service.

This incident has been resolved.

AWS are reporting many services as fully recovered in us-east-1 and we have observed stable service in our US data center since around 21:45 UTC.

We will continue to monitor but expect to resolve this incident soon.

The issue with AWS Security Token Service in us-east-1 has become worse for roughly the past 15 minutes.

This is mainly impacting scheduling tasks from triggering as they are not able to authenticate properly within the US data center to perform their tasks. This will lead to degraded service in tasks such as the polling of Apple calendars.

The impact of this has been minimized for core services as we prevented them from scaling in earlier in the incident.

The impact on service in our US data center still appears to be minimal.

AWS are resolving the underlying issue in us-east-1 and we will continue to monitor until their incident is closed.

We have seen elevated errors in our US data center, these appear to be related to an open issue in AWS us-east-1: https://health.aws.amazon.com/health/home#/account/dashboard/open-issues?eventID=arn:aws:health:us-east-1::event/MULTIPLE_SERVICES/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE_798F8_DDDD18AFADD&eventTab=details

The impact appears low at present, but given it is at the infrastructure level it could quickly become significant and so we have opened this incident.