The incident has been resolved and the backlog has cleared out. Segment clients would have experienced delays in processing but no data was lost. That's all folks!
Posted Jan 11, 2020 - 06:30 UTC
Update
All our services are stable at this point and we have not seen any more abnormal spikes while the API traffic is moving down. Our backlog queue is still being processed and clearing out. An update will be provided by 06:30 UTC.
Posted Jan 11, 2020 - 05:57 UTC
Monitoring
We received a massive amount of API requests which our cluster size was not able to handle and the cluster did not autoscale automatically. Manually increasing the size has been able to handle the incoming requests which seem to have dropped off. We are monitoring the traffic and keeping an eye on the backlog remaining. An update will be provided by 05:45 UTC.
Posted Jan 11, 2020 - 05:14 UTC
Update
We have increased our cluster capacity and have started accepting traffic. An update will be provided by 05:00 UTC.
Posted Jan 11, 2020 - 04:40 UTC
Investigating
We're experiencing some issues with one of our data-collection clusters and are investigating. Customers integrated with Customer.io via Segment are affected. An update will be provided by 04:30 UTC.