Investigating a performance issue in the Integration Platform

This incident has been resolved.

After many improvements on the infrastructure and network side, the situation is stable now.
Connection can be done instantly.
However, there might be a very small customer base that are experiencing problems due to some other reasons. This has nothing to do with the incident, those issues were happening before as well. For those customers, please try to restart the VCG service this time. If the problem persists, contact us on Slack on #vth-ipp-public
We are continuing to monitor the connections.

Considering the upcoming maintenance on the firewall, we are aiming to have all connections established by tomorrow morning. However, as stated in the document posted before, since the root cause was not yet determined, it is very hard to say when everything will be 100% functional.

We are continuing to monitor for any further issues.

We have created a document with details and latest updates, please have a look:

https://docs.google.com/document/d/11NqGz_IRLdK8SZjoPc7CsVoiCCVH_CXTG9L0a2-RVKc/edit?tab=t.0

We are continuing to monitor and implement small improvement while trying to now affect established connections.

The average connection time is now around 5 minutes, and once connected it is stable. Customers are advice not to restart the VCG service and let it connect automatically. We are working continuously on doing improvements until everything is fully operational.

We are continuing to monitor and do improvements along the way.

Still monitoring the solution. What we notice is slowness on connection between 5 and 60 minutes due to load. But once established, the connection is stable. So please avoid restarting the Visma Cloud Gateway service, it will connect automatically.

Small network incident last night on a border router caused all connections to reset. Due to high load, it takes a bit for all tunnels to reconnect. We are still monitoring the situation.

A fix has been implemented and we are monitoring the results. More resources have been added to handle the additional load.

Networking team is looking into a rollback of the Firmware upgrade they had on Thursday.

A high impact incident has been opened on the network squad side, as we were unable to find anything related to our service. We are now waiting for an investigation on their side, and will post an update as soon as we have new info.

We are currently investigating some issues related to our SSH cluster in the Visma datacenters. Time outs and slowness in response might be experienced by some of the customers.