On Friday May 30th, 2025, between 16:15 UTC and 16:31 UTC customers may have experienced errors when accessing the Customer IO application. The issue did not affect data processing or any other functionality of the system.
An update with a flaw was deployed to the Customer IO application which results in the user interface being unavailable for some users.
While investigating the issue the team rolled back to the previous version of the application at 16:31 UTC. This resolved the issue, and all users could use the application. Once the code error was identified the team corrected the code and re-deployed the application at 17:40 UTC. There were no further errors after 16:31 UTC.
The team has identified improvements to validation of code changes in efforts to eliminate code errors being deployed to production systems. In addition, improvements to system monitoring were identified to be notified more proactively when the system is returning errors. These improvements are prioritized and tracked in our internal development process queue.