Transport HTTP server issues
Incident Report for Productsup
Postmortem

Issue Summary

We experienced some storage outages over the last weeks. In all of these incidents our primary goal was to bring up the service as soon as possible. In order to do that we had to start up new storage systems and separate the workload (Productsup Server vs. Productsup FTP vs. Export2Datasource) on these systems. We were able to get back into operational mode but we still have some more things to do.

Corrective and Preventative Measures

We already started to separate the workload. We even go one step further and will provide a new FTP location in the near future to also logically separate Productsup Server and FTP which is both actually transport.productsup.io.

We also push and read Productsup Server data directly from Amazon S3 to make it more reliable. We will monitor this approach and test some others over the next days and weeks to find out if they are suitable approaches.

Productsup is committed to continually and quickly improving our technology and operational processes to prevent future outages. Unfortunately, we were not able to prevent yesterday's loss. For this, we sincerely apologize for the inconvenience this has caused you, your team, and your organization. We thank you for your continued support.

Posted Aug 02, 2018 - 14:41 CEST

Resolved
We monitored the servers and they are working as expected. Anyway, we started looking into alternatives for our FTP hosting. We will provide more information soon.
Posted Aug 02, 2018 - 10:56 CEST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Aug 01, 2018 - 19:09 CEST
Update
we were able to restart processing. We currently only process sites which dont use our FTP service. The FTP service should also be available soon.
Posted Aug 01, 2018 - 18:37 CEST
Identified
We stopped our transport service including FTP. Processing Jobs will be queued and executed later. We apologize for any scheduling woes in the meantime.
Posted Aug 01, 2018 - 16:15 CEST
Investigating
We're currently encountering issues with the HTTP Layer of our transport server. FTP is not impacted, but as a result we have to pause Processing while we fix the issue.
Posted Aug 01, 2018 - 14:11 CEST
This incident affected: Data Processing, Destination "Productsup Server", and FTP Service.