No update files being produced for rrc07.ripe.net

The remaining files have been created. Consumption of data caught up.

We have made a configuration change (i.e. added a IPv4 DNS record) that seems to have mitigated the issue. The MRT update files have been produced for the period between 08:00 and 19:00 UTC. We are monitoring the recovery of other processes.

Our processing pipeline hits a bug where the message size of the message we are constructing becomes larger than the calculated message size. This happens while processing the messages that were generated when ExaBGP was shutting down (which is a rare situation). This causes the processing pipeline to abort and get stuck.

The technical detail is that the code ends up using an IPv6 local address in a MRT message for a IPv4 peering session. The underlying root cause is either DNS resolution failing (and fallback to the link local address) or a change in collector configuration.

After maintenance earlier today, no MRT "update" files are being produced for rrc07.ripe.net. In addition there is a delay in the data for ~10 peers for the RIPEstat looking glass. Other processes consuming from rrc07 have recovered.

We are investigating the issue in our processing pipeline.