CloudBoxx Events (RabbitMQ) issues (Rare cases of increased delays)

All maintenance work was completed on schedule with plenty of buffer. The median values were below the alarm threshold the whole time, and there are no outliers now either.

The first maintenance of the fiber-optic link has been successfully completed, and the latency values are back to a very low level. We will continue to monitor the situation, but so far there have been no median values for delays above the limits. We will leave the status page switched on until the end of the maintenance window tomorrow morning at 6 UTC.

The summary of the current situation:
Maintenance is currently being carried out on the fiber optic line between our data centers. This maintenance was announced and is leading to slightly longer transmission times in the low double-digit millisecond range. However, our transmission times for the systems are still within the normal range, with rare cases of increased delays. The maintenance is continuing as planned, and we are keeping an eye on the situation.

To clarify the current situation: There is currently no increase in the median value of the delay of events, etc. However, there are some outliers in terms of transmission time. We are in contact with the data center teams to improve it.

The restrictions that we are currently measuring in processing are due to maintenance work that is currently being carried out on the interconnection between the data centers. According to initial information, this should not have any impact. Commands to the vehicles are not affected.

We are currently experiencing issues with our CloudBoxx Events Message Queue. We noticed a delay for some messages. Our emergency standby staff is investigating. We’ll post an update with details and scope of the incident as soon as possible.