The problem is fixed. We experienced issues while processing the product feeds this morning. The problem was in the underlying network which caused packet losses and lead to system instabilities on our new storage exporter. We found the issue in the network driver and will update it tomorrow.
We didn't get notified as our monitoring system was not set up correctly on the new storage cluster for this error. We fixed that. We also increased log level as well as monitoring metrics to see any deviation as soon as possible over the next days.
Sorry for the inconvenience. If you still experience strange behavior or any unexpected errors don't hesitate to contact us.
Posted Mar 19, 2018 - 10:35 CET
Monitoring
We are back at normal state processing the feeds. we are still investigating to find the root cause.
Posted Mar 19, 2018 - 09:07 CET
Identified
We have found the issue and fixed it already. System is returning into normal state. We are still investigating to find the root cause for the problem and will update soon.
Posted Mar 19, 2018 - 08:41 CET
Investigating
We are experiencing issues in processing. we already started investigation and will send an update soon.