We are experiencing problems with the transport server

Incident Report for Productsup

Postmortem

Dear Client,

Issue Summary

We experienced an outage of one of our storage nodes in our datacenter on Sunday, 11th of January. We were able to bring the node back into the cluster and monitoring showed that everything was working as expected. Unfortunately the node was still broken and caused connectivity problems. We removed the node from the cluster but this resulted in performance problems as the load was too much for the remaining cluster. The replacement node was available on monday evening and we started rebuilding the cluster which caused additional load. We resumed normal operations Tuesday 13th.

Corrective and Preventative Measures

We are currently creating a new storage cluster with additional resources and a different storage solution to provide a better reliability and disaster management.

Productsup is committed to continually and quickly improving our technology and operational processes to prevent outages. We appreciate your patience and again apologize for the impact to you, your users, and your organization. We thank you for your business and continued support.

Posted Feb 14, 2018 - 09:54 CET

Resolved

We have restarted all affected sites and have increased monitoring on all servers. All systems are fully operational from 11:30 AM today.

We apologise for any inconvenience

Posted Feb 13, 2018 - 15:11 CET

Update

we experienced huge load on the servers which led to random read timeouts on the transport server. the affected sites will be restarted and we expect that all sites will run without problems now.

Posted Feb 13, 2018 - 11:30 CET

Monitoring

The server is back up with degraded performance. It is possible due to DNS records not being resolved correctly yet that connecting isn’t possible yet from certain locations.
We’re monitoring the situation until all servers are fully operational again.

Posted Feb 12, 2018 - 18:09 CET

Identified

We have identified the issue and isolated the problem. The transport server is active again and all files are available. However we are not back to regular performance and still working on adding more resources to the transport service.

Posted Feb 12, 2018 - 16:28 CET

Investigating

we experience problems with the transport server. We are already looking into this and will provide more information once we found the reason for that issue.

Sorry for any inconvenience.

Posted Feb 12, 2018 - 15:35 CET