Issue:

We experienced a service disruption on September 10th, 2024, starting at approximately 12:07 AM UTC, due to an expired security certificate. This caused customers to receive invalid certificate errors when trying to access our platform, and in some cases, completely blocking access to the site. The issue was identified and resolved within two hours (by 1:58 AM UTC).

Timeline:

• 12:07 AM UTC: The issue with the certificate expiration is identified internally. We immediately set to work renewing the certificate, working directly with our vendor to hasten the process.
• 1:48 AM UTC: The new certificate is received.
• 1:56 AM UTC: The certificate is successfully deployed to testing environments and we confirmed everything was working.
• 1:58 AM UTC: The certificate is deployed to all production environments, and customers were confirmed to be able to access the site.

Cause:

An internal configuration change inadvertently prevented us from receiving alerts about the expiring certificate.

Remediation:

• We’ve addressed the configuration issue to our original alerts, so notifications go out as expected.
• We have established additional notification channels, using a variety of notification methods (SMS, email, etc.) to guarantee timely alerts on certificate expiration in the future.