[US/Tokyo Region] Performance issues

Incident Impact Summary (Plazma / Table Resources)

We have confirmed that the system in the Tokyo region is operating normally with no remaining issues.

What happened
- All operations accessing Plazma table resources experienced increased latency.

When
- US region: 17:00 PST – 18:40 PST
- Tokyo region: 09:00 JST – 12:50 JST

Temporary impact during recovery
- Updates to table row counts and preview data were temporarily paused.

Recovery status
- US region: Table metadata updates resumed at 20:00 PST.
- Tokyo region: Table metadata updates resumed at 13:40 JST.
Both regions are now fully recovered.

Data integrity
- No data loss or data corruption occurred.

We will prepare and publish a post-mortem document to explain the circumstances under which this incident occurred one day after the recent system maintenance.

With this confirmation, the incident is now closed.

The system in the US region is operating normally with no observed issues.

In the Tokyo region, table metadata updates have also been resumed. At this point, the Tokyo region is expected to be fully recovered, and we will continue to monitor system performance.

Once stability is fully confirmed, we plan to close this status page.

In the US region, table metadata updates have been resumed. At this point, the system in the US region is expected to be fully recovered, and we are continuing to monitor it closely.

For the Tokyo region, metadata updates will be resumed after we complete confirmation of system performance. We will provide another update when this step begins.

Performance in the Tokyo region improved at approximately 12:50 JST. The Tokyo region had been experiencing gradually increasing performance degradation since around 09:00 JST.
We are currently proceeding with the steps to resume updates of table metadata in Data Workbench.

Due to the temporary pause of metadata updates, application metrics that rely on this information, such as Parent Segment size, have also been temporarily not updated.

We will continue monitoring the system and share further updates as needed.

We are continuing to work on a fix for this issue.

We identified the same issue in the Tokyo region and applied database parameter tuning. We are currently assessing the impact of this change on system performance.
As a result of this temporary mitigation applied in both the US and Tokyo regions, the following impact is observed:
In Data Workbench, updates to table metadata for row counts and preview data are temporarily paused.
We will continue monitoring and provide further updates as we make progress.

Starting at approximately 17:00 PST, we observed increased latency affecting all operations accessing table resources in Plazma.
At around 19:00 PST, we applied temporary tuning to the database managing table resources and confirmed recovery based on internal performance metrics.

We are currently continuing to monitor the system to ensure overall performance remains stable.

Visible impacts are:

- Streaming, Mobile, and JavaScript/Browser imports may delay

- Jobs execution may delay

- Table creation may error

- Console execution may delay

- Plazma Public API may return errors

- Treasure Workflow may delay and fail

- Workflow REST API may be unavailable

- Workflow operations in Console and CLI may become unavailable

- Presto JDBC/ODBC queries and CDP segmentation queries may fail

- Console Table preview update may delay

- The REST API to submit and cancel jobs may be unavailable

- Data Connector Integrations may become unavailable

- ADH (Ads Data Hub) and DCR (Data Clean Room) service may be unavailable

- ADL (Active Data Layer) service may be unavailable

We will send an additional update in 30 minutes