Outage in The Graph

Cosmos Chain Halted

Resolved Minor

June 05, 2024 - Started almost 2 years ago - Lasted about 5 hours
Official incident page

Incident Report

[FIRING:1] Cosmos-Hub-4: Block ingestor lagging behind Production (partial_outage indexers P2-High true Infrastructure) https://thegraph.grafana.net/alerting/list **Firing** Value: B0=0 Labels: - alertname = Cosmos-Hub-4: Block ingestor lagging behind - cmp = partial_outage - component = indexers - grafana_folder = Production - priority = P2-High - statuspage = true - team = Infrastructure Annotations: - description = Triggered when the block ingestor is lagging - statuspage = Cosmos-Hub-4: Block ingestor lagging behind Source: https://thegraph.grafana.net/alerting/grafana/ddjeivh5d5ypsc/view?orgId=1 Silence: https://thegraph.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DCosmos-Hub-4%3A+Block+ingestor+lagging+behind&matcher=cmp%3Dpartial_outage&matcher=component%3Dindexers&matcher=grafana_folder%3DProduction&matcher=priority%3DP2-High&matcher=statuspage%3Dtrue&matcher=team%3DInfrastructure&orgId=1

Components affected

The Graph Network: Gateway The Graph Network: Explorer The Graph Hosted Service: Queries The Graph Hosted Service: Subgraph Health

Latest Updates ( sorted recent to last )

RESOLVED almost 2 years ago - at 06/06/2024 02:38AM

This incident has been resolved.

IDENTIFIED almost 2 years ago - at 06/05/2024 09:27PM

The issue has been identified and a fix is being implemented.

INVESTIGATING almost 2 years ago - at 06/05/2024 09:27PM

We are continuing to investigate this issue.

INVESTIGATING almost 2 years ago - at 06/05/2024 09:19PM

[Comment from Opsgenie]Ruslan Rotaru acknowledged alert: "[Grafana]: Cosmos-Hub-4: Block ingestor lagging behind"

INVESTIGATING almost 2 years ago - at 06/05/2024 09:19PM

[FIRING:1] Cosmos-Hub-4: Block ingestor lagging behind Production (partial_outage indexers P2-High true Infrastructure)
https://thegraph.grafana.net/alerting/list

**Firing**

Value: B0=0
Labels:
- alertname = Cosmos-Hub-4: Block ingestor lagging behind
- cmp = partial_outage
- component = indexers
- grafana_folder = Production
- priority = P2-High
- statuspage = true
- team = Infrastructure
Annotations:
- description = Triggered when the block ingestor is lagging
- statuspage = Cosmos-Hub-4: Block ingestor lagging behind
Source: https://thegraph.grafana.net/alerting/grafana/ddjeivh5d5ypsc/view?orgId=1
Silence: https://thegraph.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DCosmos-Hub-4%3A+Block+ingestor+lagging+behind&matcher=cmp%3Dpartial_outage&matcher=component%3Dindexers&matcher=grafana_folder%3DProduction&matcher=priority%3DP2-High&matcher=statuspage%3Dtrue&matcher=team%3DInfrastructure&orgId=1

Latest The Graph outages

Degraded Indexing Performance (Gnosis) - 4 months ago

Subgraph Indexing Issue - 4 months ago

Upgrade Indexer - DB Unavailable and Subgraph not allocated - 4 months ago

DatasourceError - 4 months ago

Polygon (Matic): Block ingestor lagging behind - 4 months ago

Never miss outages in third-party dependencies

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 6320 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook