Feed Processing

Incident Report for Productsup

Postmortem

The incident was caused by one of the FTP client servers crashing. While we were able to restart it, the client held some metadata locks on our shared filesystem which entered a degraded mode, causing the FTP service to be unavailable globally. When we stopped the faulty client, the metadata locks were released and the filesystem became available again.

We will upgrade our clients shortly (no downtime) so the issue will not likely reproduce in case of further server failure.

Posted Nov 12, 2019 - 12:13 CET

Resolved

The queue has now been cleared up. Thank you for your patience while we got that resolved.

Posted Nov 12, 2019 - 09:12 CET

Monitoring

We were able to identify and solve the issue. Now we're monitoring the affected systems. The incident created a little queue but feeds are now being processed again. Some delays are expected as we process backlogged data.

Posted Nov 12, 2019 - 08:03 CET

Investigating

We're seeing increased error levels from our feed processing and we're investigating the issue.

Posted Nov 12, 2019 - 07:37 CET

This incident affected: Data Processing, Destination "Productsup Server", and FTP Service.