Sync system affected by AWS operational issue

This incident has been resolved.

We're seeing the sync starting to work as normal. The store data are being resynced. We'll keep monitoring.

"[08:01 PM PDT] We continue to work towards resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. We continue to see improvement to network mapping latencies but they are not at normal levels yet. Other AWS services are also starting to see recovery as network mapping latencies improve. We will continue to keep you updated until network mapping latencies have returned to normal levels."

Update from AWS: https://health.aws.amazon.com/health/status

"Sep 18 6:56 PM PDT We continue to work toward resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. While network mapping propagation times have remained stable, we have not yet seen the improvement in propagation latencies that we had hoped for. We continue to work on multiple paths to mitigation. We have completed the first phase to address the resource contention within this subsystem, and are actively working on the second and third phases in parallel. Our current plan is to implement these mitigations within a single AZ (usw2-az1), and verify recovery occurs before proceeding to perform these same mitigations in usw2-az2. We will continue to provide updates every 60 minutes, or as we have additional information to share."

So while waiting for usw2-az1 and usw2-az2 to recover, we are working on deployment of our services on another AZs: usw2-az3.

We are continuing to work on a fix for this issue.

The sync system is being affected by on-going AWS cloud issue. Other components are working normally.

The cloud incident status being reported here
https://health.aws.amazon.com/health/status.
https://downdetector.com/status/aws-amazon-web-services/