Contacts not saving

Incident Report for Hearsay Systems

Postmortem

Timeline:

Start Time: 07/12/23 12:55 PM PT

End Time: 07/12/23 2:49 PM PT

Customer Impact:

An increase in load caused a backup of requests and a delay in processing across Hearsay’s contacts management system, which resulted in automatic texts not being orchestrated for new leads as well as a larger time between saving a contact and being able to search for it.

Resolution:

Additional processing was rerouted to the increased load to assist in processing the delayed queue. During this time, no data loss occurred.

Future Improvements:

Our engineering team adjusted and added more safety checks and alerts to our system to ensure we are proactively notified. Additionally, we are looking to fine-tune our autoscaling capabilities to automatically and dynamically handle a spike in load.

Posted Jul 19, 2023 - 13:12 PDT

Resolved

Hearsay Engineers have resolved the ongoing issue with contacts not saving. We would like to express our sincere apologies for any inconvenience caused by the recent disruption. We will be posting a full post-mortem on the incident with additional details on the issue and how it can be avoided in the future. We greatly appreciate your patience and understanding as we worked to resolve this issue.

Posted Jul 12, 2023 - 14:57 PDT

Investigating

Hearsay is actively investigating an issue where contacts are not being saved. Hearsay's engineers are actively looking into this matter and are treating this as our top priority. We truly apologize for any disruption and will provide an update shortly.

Posted Jul 12, 2023 - 14:22 PDT

This incident affected: Relate.