Tinybird Outage History

Tinybird experienced service degradation across multiple regions due to a merge request that caused unexpected pod movements in Kubernetes, resulting in a spike of 503 errors. The incident lasted 30 minutes before the errors subsided. The team identified the root cause and continued monitoring to ensure stability.

Tinybird experienced increased latencies in the aws-us-west-2 region that lasted approximately 91 hours. The service team identified the root cause and applied a fix, restoring latencies to normal levels.

Tinybird experienced connectivity issues in the AWS us-east region for 5.8 hours, causing increased latency and occasional delays for a subset of user requests. The problem was traced to network-level packet loss between the load balancer infrastructure and the public internet. The team investigated infrastructure and network layers while working on mitigation options to reduce impact during the incident.

Tinybird experienced degraded data ingestion infrastructure in the gcp-us-east4 region, causing significant delays in data processing for 2.9 hours. The engineering team worked to stabilize the affected component, with live ingestion becoming stable after approximately 1.5 hours, followed by processing of the backlog of delayed inserts. The incident was fully resolved once all delayed data had been successfully ingested and the system caught up to real-time processing.

Tinybird experienced a job queue degradation issue lasting 4.9 hours that caused some jobs to be delayed or fail to process as expected. The engineering team applied a fix to resolve the processing issue and restored normal queue operations. The system was then monitored to ensure stability and that all pending jobs were processing correctly.

Tinybird initiated a credential rotation across their environment variables in response to a security incident at Vercel, causing temporary disruptions to services that depend on those credentials. The incident was classified as minor and lasted 1.2 hours. The service disruptions were resolved after the credential rotation process was completed.

Tinybird's load balancer in GCP US East became unresponsive, causing intermittent errors for users accessing the service. The incident lasted 4.8 hours before being resolved, with all workflows returning to normal operation.

The Tinybird UI at cloud.tinybird.co was inaccessible for 1 hour due to a problem with an internal monitoring application. The issue caused UI load errors and prevented users from accessing the platform. The problem was identified and fixed, with access restored after approximately 47 minutes of downtime.

Tinybird's GCP us-east4 region experienced degraded performance for 1.3 hours, with users encountering high latencies and potential rate limiting. The incident was classified as minor and has been resolved.

Tinybird experienced degradation with their Kafka ingestion service for 2.2 hours. The issue affected data ingestion capabilities through Kafka. The root cause was identified and resolved, with the Kafka service returning to normal operations.