Outage in Treasure Data

[US Region] High Error rate at Custom Script and some DataConnector

Resolved Major
July 30, 2024 - Started about 2 months ago - Lasted about 8 hours
Official incident page

Need to monitor Treasure Data outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Treasure Data, and never miss an outage again.
Start Free Trial

Outage Details

We are currently experiencing a high error rate in Custom Script service on Treasure Workflow (US Region) due to an ongoing incident with our infrastructure provider (AWS). This issue is increased error rates with the following error message like: > Task failed with unexpected error: null (Service: AWSLogs; Status Code: 503; Error Code: null; Request ID: xxxxxx; Proxy: null) At this time, we do not have an estimated time for full resolution. We will provide further updates as soon as more information becomes available.
Latest Updates ( sorted recent to last )
RESOLVED about 2 months ago - at 07/31/2024 07:53AM

This incident has been resolved, all affected components (Custom Script and some DataConnector) are now back to normal.

MONITORING about 2 months ago - at 07/31/2024 05:25AM

According to our infrastructure provider (AWS), this issue has already been resolved.
We also see that the failure rate has been reduced, so we will update this incident to Monitoring status and the affected components to Operational status.

IDENTIFIED about 2 months ago - at 07/31/2024 03:16AM

Due to the degradation of Amazon Ads system https://status.ads.amazon.com, our connectors for Amazon Ads platform are currently not working properly. So if you are using any of the below connectors, your jobs may not be running correctly.

- Amazon Marketing Cloud export
- Amazon Marketing Cloud import
- Amazon Ads export
- Amazon DSP export

We will provide further updates as soon as more information becomes available.

IDENTIFIED about 2 months ago - at 07/31/2024 02:24AM

This issue is still ongoing, we are still seeing custom script tasks fail.
Custom script user may also encounter some errors about AWS Cloud Watch logs.

According to our infrastructure provider (AWS), they are working on recovery and there are some improvements being seen internally, but they expect it to take 1-2 hours for full recovery.

We will provide further updates as soon as more information becomes available.

IDENTIFIED about 2 months ago - at 07/30/2024 11:52PM

We are currently experiencing a high error rate in Custom Script service on Treasure Workflow (US Region) due to an ongoing incident with our infrastructure provider (AWS). This issue is increased error rates with the following error message like:

> Task failed with unexpected error: null (Service: AWSLogs; Status Code: 503; Error Code: null; Request ID: xxxxxx; Proxy: null)

At this time, we do not have an estimated time for full resolution. We will provide further updates as soon as more information becomes available.

Vendor Downtime? Keep Your Team Informed with an Internal Status Page

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3242 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime