Platform API: Process endpoint degraded performance

We haven't see any spikes in our job handling infrastructure anymore, so our fixes have resolved the problem.

Today at 15:00 CET we deployed the second fix, from now on calls to the process endpoint are rate limited. We're still seeing some unexpected errors being returned for calls to the process endpoint. However they are mostly for clients who do more than 100 requests an hour. We'll continue to monitor the situation till tomorrow and then write a final update.

See our public API documentation for more details on the rate limiting: https://api-docs.productsup.io/#post

We are continuing to monitor for any further issues.

This morning at 11:55 CET we deployed a new version which contained the first fix, out of a series of two fixes. Since then we've been monitoring the API requests. Our fix reduced the impact of unexpected peaks already and the job-hanling infrastructure remains more stable.
We continue to work on a second peak that will implement rate limiting for all clients on the process endpoint. It will allow no more than a single call per site per 5 minutes. We expect that no client workflow is affected by this.

Today we've stared working on a fix and that will be implemented tomorrow morning! Following that deployment we're continuing to harden our endpoint from being overloaded.

We're experiencing an unexpected amount of requests to the process endpoint and during peaks it can overload the job-handling infrastructure. Hence the error rate is higher than usual, more 500 and 429 error are to be expected when making calls to the process endpoint.

The product upload and commit endpoints are not affected!