Timeline:
Start Time: 01/12/2024 12:41 PM PT
End Time: 01/12/2024 12:58 PM PT

Customer Impact:
A recent production deployment led to certain traffic being routed to servers that necessitated deletion prior to the deployment, resulting in an inaccurate measure of production traffic on Hearsay’s system.  This led to a misunderstanding of how many servers were needed for production traffic, and users were led to the inability to access Hearsay’s platform and/or an error message stating “Something Went Wrong.”

Resolution:
More application servers were provisioned to handle active requests, which resolved the issue that caused the increased latency.

Future Improvements:
Hearsay has now established best practices for handling this issue by creating new processes to address these instances faster and manually scale up prior.