Timeline:
Start Time: 02/20/25 06:10 AM PT
End Time: 02/20/25 08:15 AM PT
Customer Impact:
Due to a production deployment on February 19th, 2025, Hearsay’s application became temporarily unavailable on February 20th, 2025. Users accessing Hearsay during this incident would see a 500 Internal Server Error message.
The impact on Relate resulted in inbound and outbound text messages being temporarily queued. Once service was restored, all queued inbound and outbound messages were successfully received and delivered with a delay. New inbound and outbound voice calls during this incident were unavailable.
The impact on Social resulted in a delay in scheduled content, with some content now flagged as “Needs Attention” for a manual review.
The impact on our Activity Sync was disrupted, preventing custom workflows for a limited number of customers.
The impact on Actions resulted in unsuccessful API calls through our Public API. All requests during the following period, 6:12 AM PST to 7:52 PST, were not successfully processed. The already queued messages have been sent out. The API returned 500 response codes, which could be gathered on the customer’s side.
All archiving activities have since resumed and may be delayed.
During this incident, there was no data loss.
Resolution:
Hearsay’s engineers rolled back the infrastructure upgrade that caused the problem and redeployed production services to use the earlier stable version.
Future Improvements:
We are exploring alternative methods to achieve the same infrastructural upgrade that was the original goal of the change that caused the outage. In addition, although a significant amount of planning and care was taken to minimize customer impact, any future changes in this area will involve additional scrutiny and testing.