High Device Streaming Latency experienced on the Platform

Hello CVaaS Users,

This incident is closed as the original high impact on June 26th, 2026 where cv-prod-us-central1-b service region experienced a period of high ingestion latency that may have caused devices incorrectly being marked as Inactive on CloudVision, or experiencing sustained periods of elevated streaming latency also as reported by CloudVision. This original high impact has been effectively mitigated by June 27th, 2026.

Since then our team has been working on fixing brief, intermittent streaming latencies (under 1 minute) that affect a small number of devices.

At present, cv-prod-us-central1-b service region is fully operational. We expect users to be able to continue using Provisioning and Telemetry features as normal.

No other service regions are impacted by this incident.

We will follow up with a postmortem of this incident, especially regarding the impact on June 26th, 2026.

We are continuing to investigate resolutions to the intermittent elevated P99 tail latency seen for data ingestion on the platform.
The platform is fully operational.

Thank you for your understanding and patience.

We are continuing to investigate resolutions to the intermittent elevated P99 tail latency seen for data ingestion on the platform.
The platform is fully operational.

Thank you for your understanding and patience.

We continue to make progress towards identifying mitigation and causes for the brief elevated ingestion latencies seen on the platform.

We continue to make progress towards identifying mitigation and causes for the brief elevated ingestion latencies seen on the platform.

Thank you for your understanding and patience.

Our teams are still working around the clock to mitigate the brief spikes of high streaming latency and determine various contributing causes.

Thank you for your patience.

We are investigating brief periods of elevated streaming latency. The platform is still operational.

We are continuing to monitor this service region. There will be no further updates to help reduce notification spam unless there is a significant change in the service region's state.

Device streaming latency and data ingestion are still steady. We are still monitoring the platform. The next update will be provided if the situation changes.

We have seen steady stabilization since June 27th, 17:00 UTC, and brief ingestion latencies ( < 5m ) being resolved with mitigations since June 28th, 02:00 UTC. Device streaming latency and data ingestion are holding steady at this time.

We are continuing to monitor the service region cv-prod-us-central1-b.

We are continuing to improve the stability of the platform. At this time we are seeing steady stabilization since June 27th, 17:00 UTC.

Our team is observing the various mitigations in place currently prior to transitioning to monitoring phase.

We are continuing to improve the stability of the platform.

We have made significant progress since the start of this incident. We have seen steady stabilization since June 27th, 17:00 UTC, with platform ingest rates and streaming latency demonstrating consistent periods of stability.

At this time, our team is focusing on mitigating brief periods ( < 5m ) of ingestion spikes. We are working on resolving this aspect before fully transitioning to the monitoring phase.

Thank you for your continued patience.

We are continuing to stabilize the platform. We will post updates every 3 hours to reduce spam.

We are continuing to stabilize the platform. We will post updates every 3 hours to reduce spam.

We are continuing to stabilize the platform. We will post updates every 3 hours to reduce spam.

We are continuing to stabilize the platform. We will post updates every 3 hours to reduce spam.

We are continuing to stabilize the platform. We will post updates every 3 hours to reduce spam.

We are continuing to stabilize the platform. We will post updates every 3 hours to reduce spam.

Our team is continuing to respond to this incident. At this time, some devices may see intermittent streaming latency increases, and potentially erroneous "Device Stopped Streaming" events, but not at broad scale. We expect this to impact about < 1 % of our devices currently for cv-prod-us-central1-b.

We are continuing to stabilize the platform.

We are continuing to stabilize the platform.

We are continuing to stabilize the platform.

We are continuing to stabilize the platform.

We are continuing to stabilize the platform.

We are continuing to stablize the platform.

Currently, lag has resolved and our publish latencies are stable - however we are seeing elevated error increase, which we are investigating.

We have fully recovered from the ingestion lag. We are continuing to monitor.

Our Publish latency for P90 and P95 have stabilized. We are continuing with recovery.

We are actively investigating as well as working on recovery.

We are experiencing high device streaming latency on this affected cluster.

Some impact may be user visible, such as:
- A delay in processing streamed device data. Some events such as "Device stopped streaming" or "Streaming Latency Breached Threshold" may fire as a result of this. In this case, the device updates are behind schedule and will be processed with some delay during the impact window.
- There may be some stale data, or sporadic timeouts viewing parts of the UI.
- Some provisioning features may be delayed or timeout due to the high latency.

We are currently investigating the issue and working on mitigation. Thank you for your patience.