Tinybird Outage History

Tinybird experienced connectivity issues in the AWS us-east region for 5.8 hours, causing increased latency and occasional delays for a subset of user requests. The problem was traced to network-level packet loss between the load balancer infrastructure and the public internet. The team investigated infrastructure and network layers while working on mitigation options to reduce impact during the incident.

Tinybird experienced degraded data ingestion infrastructure in the gcp-us-east4 region, causing significant delays in data processing for 2.9 hours. The engineering team worked to stabilize the affected component, with live ingestion becoming stable after approximately 1.5 hours, followed by processing of the backlog of delayed inserts. The incident was fully resolved once all delayed data had been successfully ingested and the system caught up to real-time processing.

Tinybird experienced a job queue degradation issue lasting 4.9 hours that caused some jobs to be delayed or fail to process as expected. The engineering team applied a fix to resolve the processing issue and restored normal queue operations. The system was then monitored to ensure stability and that all pending jobs were processing correctly.

Tinybird initiated a credential rotation across their environment variables in response to a security incident at Vercel, causing temporary disruptions to services that depend on those credentials. The incident was classified as minor and lasted 1.2 hours. The service disruptions were resolved after the credential rotation process was completed.

Tinybird's load balancer in GCP US East became unresponsive, causing intermittent errors for users accessing the service. The incident lasted 4.8 hours before being resolved, with all workflows returning to normal operation.

The Tinybird UI at cloud.tinybird.co was inaccessible for 1 hour due to a problem with an internal monitoring application. The issue caused UI load errors and prevented users from accessing the platform. The problem was identified and fixed, with access restored after approximately 47 minutes of downtime.

Tinybird's GCP us-east4 region experienced degraded performance for 1.3 hours, with users encountering high latencies and potential rate limiting. The incident was classified as minor and has been resolved.

Tinybird experienced degradation with their Kafka ingestion service for 2.2 hours. The issue affected data ingestion capabilities through Kafka. The root cause was identified and resolved, with the Kafka service returning to normal operations.

Tinybird experienced a service degradation across the entire gcp-us-east4 region except for Kafka services, with full degradation occurring from 11:20-11:25 UTC. The incident lasted 41 minutes total before service was restored and returned to normal operation.

Tinybird experienced increased errors in their GCP-us-east4 region for 55 minutes. The service team identified the root cause and applied a fix to resolve the degradation. The incident was classified as minor and the situation was being monitored following the resolution.