Slow application response times

Dear customers,

I'm writing to give you the final update on the recent incident, which we are now marking as resolved.

From around 7pm UTC on the Wednesday 20th December a change was rolled out to all customers and since then we've observed stability in response times and performance even when there was a high volume of traffic running through the platform. This has given us confidence that we've addressed the root cause and solved the underlying issue.

During this process, we identified other areas of improvement and have already rolled out several changes which will provide longer term benefits to our customers. We'll continue to double down on further areas of improvement off the back of this.

We sincerely apologise for any inconvenience caused to you and your teams, especially as we approach the holiday season.

Best wishes,

Danny Hannah
CTO

Yesterdays deployment has seen a significant improvement to performance and stability. We're continuing to monitor activity closely and make additional improvements. We will leave this incident open whilst we observe the coming days and provide updates accordingly.

We have deployed a change which looks promising. We will continue to monitor and provide any updates in due course. We apologise for the continued inconvenience this has cause.

We're continuing to see some issues with certain customers response times and are working to restore stability as soon as possible.

We've managed to stabilise the system and will continue to monitor as we continue work towards a root cause. We apologies for the continued disruption.

Unfortunately we've started to see a recurrence of slow response times, and are working to mitigate as soon as possible.

Over the weekend, the Convertr team has migrated all customers to a new cluster and performed a series of stress tests which have all been successful.

We will continue to monitor as customer workloads increase in the coming days and update the incident accordingly.

We are continuing to work with AWS on the root cause which appears to be related to the underlying AWS cluster, however we will provide more details in due course.

We thank you for your continued support, and apologise for the disruption which has been caused.

Best regards
Danny Hannah
CTO

We are continuing to investigate this issue as a priority. We have migrated some customers to a new cluster and will continue to monitor the situation. The team are continuing to work on this issue over the weekend to restore stability as soon as possible.

The team are continuing to investigate and explore options to restore service to normal.

We are continuing to work with AWS and explore options to restore service as soon as possible. We are sorry for the inconvenience caused.

We are continuing to work with AWS and explore options to restore service as soon as possible. We are sorry for the inconvenience caused.

We are continuing to work with AWS and explore options to restore service as soon as possible. We are sorry for the inconvenience caused.

Dear customers,

As we near the end of the week we wanted to provide you with an update on the intermittent platform issues you may have experienced starting from Tuesday 12th this week.

Around 2.06pm (UK) we had a partial outage. By 3.24pm we were fully operational again. Throughout the majority of that time, the system was available albeit with degraded performance. The cause of this was a recommended AWS configuration change.

On Wednesday 13th around 7.44am we reverted our status back to Degraded Performance as we'd identified slower processing times overnight. We still believed this to be an AWS issue and continued working with their Support team on the investigation. By 3.14pm on 13th we updated our status to be Operational again having implemented changes to work around the issues identified.

Overnight it became clear that these workarounds weren't having the desired effect and by 12:18pm on Thurs 14th we returned to a status of Degraded Performance. By 3.06pm we'd implemented more workarounds that appeared to have addressed the issue and we became Operational once again. Our work with AWS to monitor the situation and resolve the root cause was still ongoing throughout.
This morning (Fri 15th) we had to return to a Degraded Performance status. By this afternoon things were largely stabilised (although we know some customers are still experiencing intermittent issues). At the time of writing, we're still monitoring the situation and working with AWS on a fix for the root cause.

Despite the platform being available throughout almost the entirety of this issue, we understand how slower response times impact your work and we want to apologise for any inconvenience this has/is causing. Whilst there are issues our advice is to wait whilst processing takes place (rather than retrying).

We'd like to assure you that we're doing everything possible to restore stability and performance - and have a team working round the clock together with our Support from AWS.

The best way to stay updated with the latest information is to subscribe to our status page at https://status.convertr.io/ where you can see a full history of this incident together with updates as they're posted.

We thank you for your patience and will continue to work on this until full service is restored.

Kind regards
Danny Hannah
CTO

Unfortunately we continue to see issues this morning after a sustained stable period. We are continuing to investigate as a priority.

We have implemented a fix and have seen positive signs of recovery. We are continuing to monitor the situation and work with AWS (our hosting provider) on the root cause.

Unfortunately, mitigations put in place yesterday, which were monitored through the night, have improved stability but response times continue to be slow. The team are continuing to work on this issue as a priority. We are very sorry for the inconvenience and are working hard to restore normal service as soon as possible.

A fix has been implemented and we're monitoring the results. We will continue to provide updates.

This issue is ongoing, we're continuing to explore options to restore normal service as quickly as possible.

We are continuing to work on a resolution to this issue. We're very sorry for the inconvenience caused, and continue to work on this as a priority.

We are continuing to work on a resolution to this issue. We're very sorry for the inconvenience caused, and continue to work on this as a priority.

The team have been working with AWS and identified a potential issue, we are working with them to implement a resolution as soon as possible.

The team are investigating slow response times on the Convertr application, we will keep this incident updated as we understand more.