All Systems Operational
Uptime over the past 90 days. View historical uptime.
Javascript Tracker ? Operational
90 days ago
100.0 % uptime
Today
Data Collection ? Operational
90 days ago
100.0 % uptime
Today
Data Processing ? Operational
90 days ago
99.91 % uptime
Today
Email Sending ? Operational
90 days ago
99.91 % uptime
Today
Management Interface ? Operational
90 days ago
99.98 % uptime
Today
Knowledge Base ? Operational
90 days ago
100.0 % uptime
Today
Third-Party Services Operational
Email Composer Image Uploads ? Operational
DNS ? Operational
CDN Gateway ? Operational
SendGrid SMTP Operational
Google Cloud Platform Operational
Google Compute Engine Operational
Google Cloud Networking Operational
Google Cloud Storage Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
had a major outage
had a partial outage

No incidents or maintenance related to this downtime.

Past Incidents
Nov 18, 2019

No incidents reported today.

Nov 17, 2019

No incidents reported.

Nov 16, 2019

No incidents reported.

Nov 15, 2019

No incidents reported.

Nov 14, 2019
Elevated errors in our data processing
Resolved - Still all clear! Thanks for the patience today as we recovered from a failure in our hosting infrastructure. Have a great day!
Nov 14, 20:01 UTC
Monitoring - Our data processing is fully operational again.

We identified a number of internal services that restarted due to failures at our cloud infrastructure provider starting at approx. 19:00 UTC. Some of these services, chief among them being our track.customer.io API workers did not restart gracefully.

At 19:15 UTC our SRE team restarted these API workers which unblocked our delayed data processing and we are monitoring now to ensure our system remains stable.

If we don't identify additional issues by 20:00 UTC we will resolve this incident.
Nov 14, 19:28 UTC
Investigating - We're investigating a sudden spike in errors for our data processing infrastructure. This causes a delay in inbound API processing and outbound message delivery.

We'll provide an update by 19:30 UTC as we learn more.
Nov 14, 19:15 UTC
Nov 13, 2019
Delays in Segment processing
Resolved - This incident has been resolved.
Nov 13, 13:10 UTC
Monitoring - The issue was identified and a temporary fix is in place. Processing performance is restored. We will keep monitoring and working on a permanent fix.
Nov 13, 12:46 UTC
Investigating - We are investigating an issue that is causing delays in the processing of segments.
Nov 13, 11:20 UTC
Nov 12, 2019

No incidents reported.

Nov 11, 2019

No incidents reported.

Nov 10, 2019

No incidents reported.

Nov 9, 2019

No incidents reported.

Nov 8, 2019

No incidents reported.

Nov 7, 2019
Partial data processing outage
Resolved - Everything remains operating normally. Thank you for the patience today while our team investigated and isolated a very tricky edge case.

Your friends at Customer.io
Nov 7, 21:38 UTC
Monitoring - Great news! The edge case we identified and patched does appear to be the cause of today's incident. Our team is monitoring to ensure everything continues operating normally.

Unless we identify additional issues we'll resolve this incident at 21:30 UTC
Nov 7, 20:39 UTC
Update - We've identified an edge case in our backend that may cause these processing slowdowns and are deploying a fix for it. We will monitor to determine if this resolves the issue.

We'll update again by 21:00 UTC
Nov 7, 20:32 UTC
Update - We are still investigating. Currently we're working to isolate the processing slowdown to a specific customer environment. Once we've identified the source we'll be able to remediate the issue.

We'll update again by 20:30 UTC
Nov 7, 20:05 UTC
Update - We're continuing to investigate. Nothing new exciting to share at this update.

We'll update again by 20:00 UTC
Nov 7, 19:31 UTC
Update - We are continuing to investigate. The rollback was successful and we've eliminated a recent change as a possible cause. We will provide an update by 19:30 UTC
Nov 7, 19:02 UTC
Update - We're still investigating the problem database and are rolling back to an earlier deploy to isolate the root cause. We will provide another update by 19:00 UTC
Nov 7, 18:21 UTC
Update - Investigation is still ongoing. We've deployed changes to restore processing and will monitor.

We will provide an update by 18:00 UTC
Nov 7, 17:35 UTC
Update - We're still investigating, attempting to restore processing for a subset of the affected workspaces.

We will provide an update by 17:30 UTC
Nov 7, 17:08 UTC
Update - Investigation is still ongoing.

We will provide another update by 17:00 UTC
Nov 7, 16:31 UTC
Update - Investigation is still ongoing, we have not isolated the issue yet.

We will provide another update by 16:30 UTC
Nov 7, 16:04 UTC
Investigating - We're having issues with one shard in our database resulting in no data processing and no messages being sent for a portion of our customers. Additionally, the management interface for the affected workspaces is not functional. Data collection is not affected.

We will provide an update by 16:00 UTC
Nov 7, 15:29 UTC
Nov 6, 2019

No incidents reported.

Nov 5, 2019

No incidents reported.

Nov 4, 2019

No incidents reported.

← Incident History Powered by Statuspage