Platform availability issues
Incident Report for Productsup
Postmortem

We experienced some problems over the last weeks which led to partial or major outages of the platform and/or components. We posted Postmortems to give you as much information as possible.

Unfortunately we had another outage on Tuesday which was caused by the Image Designer. The Image Designer allows real-time manipulation of images with product data to be delivered directly to (advertising) partners. What happened yesterday was a result of a partner requesting more images at the same time then we ever received before on the Image Designer. This caused the maximum capacity of queries we can send to our internal database to be reached, sadly this meant that our Platform was also unavailable.

We have taken a few approaches to resolve this issue in the future to prevent this from happening again. One being setting stricter rate limits on how many images can be requested by a single partner. The second is sending the queries from the Image Designer to another database. In next few days we’re working on adding an extra layer between the Image Designer and the database to allow for short-time caching to relieve even more stress from the systems. Also for a more global presence we’ve started the roll-out of extra servers in a different datacenter in Finland.

Posted Jul 12, 2018 - 16:25 CEST

Resolved
The measures to reduce traffic on our systems were successful. all systems operational. We are closing this issue.
Posted Jul 11, 2018 - 12:31 CEST
Monitoring
We've identified the issue down to a specific IP range that sent a huge amount of requests to our image service, causing critical backend services to fail in consequence. We've limited that range to a number of requests per second to keep our service available and will consider other improvements in our software to prevent such attacks in the future.
Posted Jul 10, 2018 - 22:55 CEST
Investigating
we are experiencing issues again.
Posted Jul 10, 2018 - 22:21 CEST
Monitoring
we have implemented a fix and all systems are working again. we are going to monitor the situation for the next hours.
Posted Jul 10, 2018 - 21:45 CEST
Identified
The issue has been identified and a fix is being implemented.
Posted Jul 10, 2018 - 21:29 CEST
Investigating
we experience issues with the platform. We are working hard to find the issue of this problem
Posted Jul 10, 2018 - 20:59 CEST
This incident affected: Productsup Platform and Image Service.