Image Service Incident (Image Designer)

Incident Report for Productsup

Postmortem

Incident Overview

On 13.12.2024, an upgrade from Chromium 124 to Chromium 130 was deployed on our servers, impacting the Image Service's ability to generate scalable vector images (SVGs). This incident was triggered by our automated security upgrade process, which is in place to maintain the security integrity of our software.

Timeline

Upgrade Date: 13.12.2024 07:00 - Chromium was upgraded from version 124 to version 130.
Incident Detection: 13.12.2024 09:00 - Users reported issues with SVG generation.
Investigation: 13.12.2024 09:30 - The team began investigating the root cause.
Rollback: 13.12.2024 12:00 - Chromium was rolled back to version 124, restoring SVG generation functionality.
Current Status: Fix for Chromium 130 is in progress and will be thoroughly tested before redeployment.

Root Cause

The upgrade to Chromium 130 introduced incompatibilities that affected SVG generation. This was an unanticipated consequence of the automated security upgrades, which are designed to keep our systems secure by ensuring all software is up to date.

Resolution

To mitigate the immediate issue, we rolled back Chromium to version 124, which restored the SVG generation functionality. The team is currently working on a fix for version 130 to ensure that it can be deployed without causing similar issues in the future.

Action Items

Testing Fix for Chromium 130: Ensure that the upcoming fix is thoroughly tested before any deployment.
Review Upgrade Process: Assess our automated upgrade process to identify potential improvements that minimize risks associated with future upgrades.
Monitoring Enhancements: Implement enhanced monitoring to quickly identify and address similar issues in the future.

Conclusion

We recognize the impact that this incident had on our users and are committed to ensuring that our systems remain secure while minimizing disruptions. We appreciate your understanding and patience as we work towards a permanent solution for the Chromium 130 upgrade.

Posted Dec 13, 2024 - 12:49 CET

Resolved

Description:
We have successfully rolled back Chromium to an earlier version, and SVG generation is now functioning as expected.

Impact:
Users can now generate SVG images without any issues.

Thank you for your patience during this incident. If you experience any further issues, please reach out to our support team.

Posted Dec 13, 2024 - 12:41 CET

Investigating

Description:
We are currently experiencing issues with Image Designer (SVG) generation in our Image Service. This is due to a necessary security upgrade, which has temporarily affected our ability to generate images.

Impact:
Users are unable to generate SVG images at this time.

Next Steps:
We are aware of the issue and are actively working on a workaround. Updates will be provided as soon as more information is available.

Thank you for your patience as we resolve this matter.

Posted Dec 13, 2024 - 11:46 CET

This incident affected: Image Service.