Performance Issues On US Servers
Incident Report for Customer.io
Postmortem

On Tuesday November 16th 2021 between 17:35 and 18:05 UTC our data collection service (track.customer.io, track-eu.customer.io) and API endpoints (api.customer.io, api-eu.customer.io, customerioforms.com, forms.customer.io, forms-eu.customer.io) suffered an outage. The outage was the result of a networking failure on our cloud provider platform: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh

During the outage all requests to the aforementioned endpoints were failing with an HTTP 404 error code.

Customer.io would like to apologize for the impact of this outage. We are committed to learn from this event and use it to drive improvement across our services.

Posted Nov 18, 2021 - 16:47 UTC

Resolved
GCP is still reporting performance issues but this outage no longer impact Customer.io customers.

https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh
Posted Nov 16, 2021 - 19:57 UTC
Update
We are monitoring the Google outage here - https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh
Posted Nov 16, 2021 - 18:17 UTC
Monitoring
We've identified an issue with GCP which was impacting our ability to process data.

The incident appears to have been resolved by Google but we are monitoring to confirm.
Posted Nov 16, 2021 - 18:12 UTC
Investigating
We are currently investigating this issue.
Posted Nov 16, 2021 - 18:02 UTC
This incident affected: Data Processing.