Platform API: Process endpoint degraded performance
Incident Report for Productsup
Resolved
We haven't see any spikes in our job handling infrastructure anymore, so our fixes have resolved the problem.
Posted Jan 22, 2021 - 17:43 CET
Update
Today at 15:00 CET we deployed the second fix, from now on calls to the process endpoint are rate limited. We're still seeing some unexpected errors being returned for calls to the process endpoint. However they are mostly for clients who do more than 100 requests an hour. We'll continue to monitor the situation till tomorrow and then write a final update.

See our public API documentation for more details on the rate limiting: https://api-docs.productsup.io/#post
Posted Jan 21, 2021 - 18:16 CET
Update
We are continuing to monitor for any further issues.
Posted Jan 20, 2021 - 18:59 CET
Monitoring
This morning at 11:55 CET we deployed a new version which contained the first fix, out of a series of two fixes. Since then we've been monitoring the API requests. Our fix reduced the impact of unexpected peaks already and the job-hanling infrastructure remains more stable.
We continue to work on a second peak that will implement rate limiting for all clients on the process endpoint. It will allow no more than a single call per site per 5 minutes. We expect that no client workflow is affected by this.
Posted Jan 20, 2021 - 15:28 CET
Update
Today we've stared working on a fix and that will be implemented tomorrow morning! Following that deployment we're continuing to harden our endpoint from being overloaded.
Posted Jan 19, 2021 - 17:17 CET
Identified
We're experiencing an unexpected amount of requests to the process endpoint and during peaks it can overload the job-handling infrastructure. Hence the error rate is higher than usual, more 500 and 429 error are to be expected when making calls to the process endpoint.

The product upload and commit endpoints are not affected!
Posted Jan 19, 2021 - 17:16 CET
This incident affected: Platform API.