BlueDolphin Service Interruption - Remediation and Status Update

Incident summary

Between 09:00 10-11-2022 CET and 12:00 CET on 11-11-2022, BlueDolphin customers in Europe may have experienced intermittent connectivity issues and timeouts/errors.

Preliminary Root Cause: We determined that a combination of corrupted data arriving at the platform, firewall “retaliation”, and orchestrator nodes which were updated by Azure ( Microsoft Cloud) caused inconsistent behavior.

Next steps

Our team will continue to investigate the chain of events to establish the full root cause and prevent future occurrences.

We have added an additional layer of monitoring and alerting to detect such or similar anomalies in behavior and to automate failover and auto-recovery mechanisms.

We have enhanced our support - R&D - DevOPs processes and elevated priority on all related tasks and the allocated resources on addressing these reliability topics ASAP.