Sorry for the radio silence folks! We've been hard at work scaling up our external API cluster and are back to processing data from Segment.com. We've set some new records on request processing and are monitoring as we catch up on the backlog from today's delays.
Once we've caught up and verified business is back to usual we'll resolve this incident. Thank you for the patience today.
Posted Apr 01, 2019 - 23:28 UTC
Update
We're continuing to work with 3rd parties to resolve the issue. We'll touch base again by 2019-04-01 20:30 UTC
Posted Apr 01, 2019 - 20:09 UTC
Update
We're continuing to work with 3rd parties to resolve the issue. We'll touch base again by 2019-04-01 20:00 UTC
Posted Apr 01, 2019 - 19:34 UTC
Update
We're continuing to work with 3rd parties to resolve the issue. We'll touch base again by 2019-04-01 19:30 UTC
Posted Apr 01, 2019 - 18:56 UTC
Update
Apologies for the delay on this update! We're still working with 3rd parties to resolve the issue and working on increasing our capacity to handle the increased traffic. We'll touch base again by 2019-04-01 18:55 UTC
Posted Apr 01, 2019 - 18:35 UTC
Update
We are still working on scaling up our Segment.com data collection cluster. We're also engaging with Segment's team to mitigate this issue within their system.
Once we finish scaling up the collection cluster on our side we will observe the results to determine if the added capacity will allow us to return to normal processing capability. Our next update will be at 2019-04-01 18:05 UTC
Posted Apr 01, 2019 - 17:51 UTC
Update
We are now scaling up our Segment.com data collection cluster to mitigate the DDoS level of traffic coming from their systems. We will update again with results from this increase in our processing capacity by 2019-04-01 17:45 UTC
Posted Apr 01, 2019 - 17:18 UTC
Identified
The delays in processing inbound API requests from Segment.com integrations is due to a recent spike in traffic from one of our customers. We've reached out to both the customer and Segment to request they mitigate the flood of traffic. We will update by 2019-04-01 17:10 UTC
Posted Apr 01, 2019 - 16:56 UTC
Investigating
We are investigating delays in processing inbound API events from customers using Segment.com to integrate with Customer.io. We will update by 2019-04-01 16:55 UTC with more details.