Long load balancer registration times

AWS has marked this issue RESOLVED as of 19:19 UTC, and we have not observed any issues in the last hour.

The issue has been resolved and all services are operating normally.

The latest update from AWS indicates that operations created around 17:10 through 17:20 UTC were impacted, which matches our internal metrics. AWS has promised another update by 18:00 UTC, and we will continue to monitor the situation until we are satisfied that it is resolved.

We're again seeing degradation and failure to register new load balancer targets in about about 10% of running operations.

Loadbalancer registration appears to be working as expected at this time. We will continue to monitor operations until AWS resolves their service degradation notice.

AWS has acknowledged the impact we are seeing and opened an incident:

> We are investigating increased load balancer back-end instance registration times in the us-east-1 Region. September 26, 2024 at 16:21:43 UTC

Since 16:05 UTC, Aptible is observing some recovery, about half of endpoint target registrations are succeeding at this time.

This service impact only applies to resources hosted in the `us-east-1` region.

Customers may notice operations reaching timeout, but at this point all operations are rolling back successfully to the previous state.

We are investigating abnormally long registration times for new targets with AWS Load Balancers. This may be causing extended operation times for releases (Deploy, Scale, Restart) for services that have Endpoints.