AWS Major Service Outage

We are aware of a major AWS outage affecting the us-east-1 region. It is currently not affecting Halo service, but may progress throughout the day as it comes closer to working hours. We are working to ensure that it does not affect any US customers.

Email processing and scheduled actions are currently affected. We are working to bring these back.

Email processing and scheduled actions have been restored.

Automatic scaling services could be hampered after the following AWS Health update:

[03:35 AM PDT] The underlying DNS issue has been fully mitigated, and most AWS Service operations are succeeding normally now. Some requests may be throttled while we work toward full resolution. Additionally, some services are continuing to work through a backlog of events such as Cloudtrail and Lambda. While most operations are recovered, requests to launch new EC2 instances (or services that launch EC2 instances such as ECS) in the US-EAST-1 Region are still experiencing increased error rates. We continue to work toward full resolution. If you are still experiencing an issue resolving the DynamoDB service endpoints in US-EAST-1, we recommend flushing your DNS caches. We will provide an update by 4:15 AM, or sooner if we have additional information to share.

Amazon have resolved the issue and their core systems are back online now. No Halo application downtime has been reported. Issue affected email processing from 6:30am - 930 am UTC but is now operational again.

AWS Service Update

After an earlier fix, the issue has returned to degraded in the AWS health dashboard, and is causing delays with automations and task scheduling.