A power outage in an AWS data center caused all DMS tasks in Etleap's hosted EU environment to restart, resulting in corrupted CDC files and parsing errors for some pipelines using CDC sources. The incident also caused a small amount of data loss between the last good file and the first post-restart file. Etleap identified all affected pipelines, directly notified impacted customers with next steps, and is working with AWS DMS team to prevent future data integrity issues during power outages.
Trusted by 1,000+ teams
Stop finding out about outages from your users. Monitor 6,320+ cloud services and get alerted the second something breaks.
Earlier today between approximately 10:57 UTC and 11:20 UTC, a power outage in an AWS data center caused all DMS tasks in our hosted EU environment to restart. The restart resulted in corrupted CDC files being written to S3 (zero-filled regions and partial records), as well as a small amount of data loss between the last good file and the first post-restart file. We have opened a dialog with the DMS team at AWS and are pushing for a post-mortem on their side.
We have identified all potentially affected pipelines and notified the impacted customers directly with next steps. If you have not received a notification from us, your pipelines were not affected. We will be working with the DMS team to explore ways to guarantee that future power outages won't cause data integrity issues.
We have determined that the parsing errors are anomalous, i.e. that pipelines that are configured not to stop on parsing errors are unaffected by this issue. We are currently determining whether any pipelines are stopped as a result of these anomalous parsing errors.
Some pipelines using CDC sources are partially impaired. You may see parsing errors as a result of the DMS files produced during the earlier AWS outage that affected the eu-west-3 Paris region.
With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.
Start free trialNo credit card required · Cancel anytime · 6320 services available
Integrations with