Emails failing to be sent
Incident Report for Customer.io
Resolved
All clear! We've confirmed that the failures stemming from our ESP's failed SSL certificate have cleared our system.

Apologies for the disruption this caused for your sending today, if you have any questions please feel free to contact us at win@customer.io. Cheers!
Posted 7 months ago. Jan 23, 2019 - 21:24 UTC
Monitoring
We've observed that the broken certificate chain at our ESP appears to be repaired. Internal test emails have successfully delivered and the overall failure rate of email delivery in Customer.io has dropped significantly. We're continuing to monitor the incident from our own system and see that there are still occasional failures due to the repaired certificate chain not yet fully propagating.

We'll update and resolve this incident once we've confirmed a full recovery and emails that previously failed to send are accepted by our ESP.
Posted 7 months ago. Jan 23, 2019 - 20:37 UTC
Identified
Our primary ESP deployed an incomplete and invalid SSL certificate to their SMTP infrastructure. This is preventing outbound email messages from sending through Customer.io for accounts that use SendGrid to deliver emails. This includes all accounts that are not using our Custom SMTP option and any Custom SMTP account that is using SendGrid.

We are in communication with our ESP to receive updates and ensure a timely recovery. Emails that have failed to send will be retried once our ESP has resolved the SSL certificate error in their infrastructure.
Posted 7 months ago. Jan 23, 2019 - 20:25 UTC
Investigating
Started: 19:00 UTC
Severity: High - sends are being attempted, but receiving errors
Who is affected: all Customer.io accounts that are not using custom SMTP

Our engineering team is currently investigating the issue, which is resulting in the error message "x509: certificate signed by unknown authority"

We will provide another update by 20:40 UTC
Posted 7 months ago. Jan 23, 2019 - 20:12 UTC
This incident affected: Email Sending.