Service Incident Notification - Ceph Storage Cluster (Processing)

Date: September 9, 2025
Incident Reference: CEPH-2025-009
Status: RESOLVED

What Happened

On September 9, 2025, our storage cluster experienced a service disruption while implementing a planned infrastructure upgrade to enable multi-datacenter operations. During the activation of "stretch mode" (a feature that allows data to be synchronized across multiple datacenters for improved disaster recovery), the cluster's internal authentication system encountered an unexpected failure.

Customer Impact

Duration: 12:30 → 15:00 (2hr30m)
Affected Services: Data Processing
Data Safety: All customer data remained secure and intact throughout the incident. No data was lost or corrupted.

During the incident period, customers may have experienced:

• Temporary inability to access stored workspaces or export2datasource
• Intermittent connectivity issues with applications using the storage service

Root Cause

The issue occurred when our storage system automatically reconfigured all data pools during the multi-datacenter setup process. This reconfiguration inadvertently affected critical system components responsible for user authentication, preventing normal access to the cluster even though the underlying data remained safe and accessible.

Resolution

Our engineering team worked to restore service by upgrading the storage cluster software to a newer version that provided additional recovery options not available in the previous version. This upgrade allowed us to safely reverse the multi-datacenter configuration and restore normal cluster operations.

What We're Doing to Prevent This

• Enhanced Testing: We are improving our testing environments to better replicate production conditions and catch similar issues before they affect live services
• Software Version Management: We are updating our deployment standards to use newer software versions that provide better recovery capabilities for major configuration changes
• Monitoring Improvements: We are implementing additional real-time monitoring for authentication systems during major infrastructure changes
• Rollback Procedures: We are developing more comprehensive rollback procedures for complex infrastructure modifications

Our Commitment

We sincerely apologize for any inconvenience this incident may have caused. Data security and service reliability are our highest priorities. We are committed to learning from this experience and implementing the necessary improvements to prevent similar incidents in the future.

If you have any questions or concerns about this incident, please contact our support team.