GDSC Datamart Ingestion Issue

The ingestion service has returned to stable operation, and the platform is processing data normally. Ingestion has caught up to current data, and all core services are functioning as expected.
Our teams will continue to monitor the system, but no further operational impact is anticipated related to this incident.
Thank you for your patience.

Ingestion has caught up to current data, and the platform is operating in a stable state. Backlog queues have been processed successfully, and new incoming data is being handled normally as it arrives.
Downstream processing, including risk generation, has resumed and data is flowing as expected. We will continue to monitor closely to ensure sustained stability as regular ingestion cycles continue.

Ingestion continues to make steady progress. The new ingest queue is decreasing, and retry volume is now very low and stable. Queued data is actively being processed, and the platform remains stable under the current operating conditions.
We will continue to monitor closely as the remaining backlog drains and ingestion continues toward full normalization.

Our Engineering and SRE teams worked through the weekend and made some adjustments to the ingestion service process today. The platform is actively processing queued data now and ingestion throughput is improving. We expect the ingestion of data to catch up by tomorrow morning.
Our teams remain closely engaged and are monitoring progress to ensure continued recovery. Further updates will be provided about the progress to completely restoring all services.

We continue to experience material degradation in data ingestion and downstream event generation. While some progress has been made, ingestion throughput remains inconsistent and backlog processing is ongoing. As a result, risk events and outliers may be delayed or incomplete.

A mitigation has been successfully implemented and we are monitoring the results.
The data warehouse engine rollback has completed, and data ingestion throughput has recovered as expected. The processing backlog is decreasing rapidly, and downstream components dependent on ingestion are beginning to normalize.
We will continue to closely monitor ingestion performance and data freshness to ensure sustained stability before marking the incident as fully resolved.

Status: Mitigating
We are continuing to address degraded data ingestion performance, which is causing delays in downstream components that rely on ingested data. The platform remains available; however, data freshness is impacted.
The root cause has not yet been confirmed. In coordination with Cloud provider Support, current findings suggest database lock contention in Data warehouse DB as a contributing factor.
We are actively tuning ingestion concurrency, scaling database capacity, and working with Cloud provider to restore normal processing rates.
Further updates will be provided as mitigation progresses.

A fix has been applied and issue is resolved. We are monitoring the service.

We are continuing to investigate an issue affecting the datamart ingestion service. Our engineering team is actively working to identify the root cause and restore normal operations. During this time, some data may be delayed or incomplete. We will provide further updates as more information becomes available.

The root cause has been identified, and we are currently working on implementing a potential fix for the issue.

The datamart ingestion pipeline is experiencing partial failures, causing some datasets not to be ingested successfully. We are actively investigating the issue to determine the root cause and restore normal operation.