All clear! We've confirmed that the failures stemming from our ESP's failed SSL certificate have cleared our system.
Apologies for the disruption this caused for your sending today, if you have any questions please feel free to contact us at firstname.lastname@example.org. Cheers!
Posted Jan 23, 2019 - 21:24 UTC
We've observed that the broken certificate chain at our ESP appears to be repaired. Internal test emails have successfully delivered and the overall failure rate of email delivery in Customer.io has dropped significantly. We're continuing to monitor the incident from our own system and see that there are still occasional failures due to the repaired certificate chain not yet fully propagating.
We'll update and resolve this incident once we've confirmed a full recovery and emails that previously failed to send are accepted by our ESP.
Posted Jan 23, 2019 - 20:37 UTC
Our primary ESP deployed an incomplete and invalid SSL certificate to their SMTP infrastructure. This is preventing outbound email messages from sending through Customer.io for accounts that use SendGrid to deliver emails. This includes all accounts that are not using our Custom SMTP option and any Custom SMTP account that is using SendGrid.
We are in communication with our ESP to receive updates and ensure a timely recovery. Emails that have failed to send will be retried once our ESP has resolved the SSL certificate error in their infrastructure.
Posted Jan 23, 2019 - 20:25 UTC
Started: 19:00 UTC Severity: High - sends are being attempted, but receiving errors Who is affected: all Customer.io accounts that are not using custom SMTP
Our engineering team is currently investigating the issue, which is resulting in the error message "x509: certificate signed by unknown authority"