Relate Outage

Incident Report for Hearsay Systems

Postmortem

Timeline:

Start Time: 09/16/22 10:02 AM PT

End Time: 09/16/22 10:40 AM PT

Customer Impact:

A new feature related to Hearsay's contact service was introduced into production that caused 500 errors via one of Hearsay's internal APIs. This resulted in Hearsay Relate failing to send and receive messages during the time span outlined above.

Resolution:

The code causing the problem was reverted to its former state and after the code was reverted, both our contact service and Relate functioned normally.

Future Improvements:

Hearsay will improve pre-deployment (staging) testing in order to catch issues like this before they make it to production. Internal tooling improvements will also be made to reduce the duration of any outages like this in the future.

Posted Sep 22, 2022 - 13:09 PDT

Resolved

Hearsay's engineers have resolved the ongoing issue with Relate. Users should no longer be experiencing issues sending messages or making calls via Relate. Thank you so much for your patience, and we truly apologize for any disruption.

Posted Sep 16, 2022 - 10:52 PDT

Investigating

We're currently experiencing an issue with Hearsay Relate in which users encounter problems sending messages and making calls. Hearsay's engineers are actively looking into this matter and are treating this as our top priority. We truly apologize for any disruption and will provide an update shortly.

Posted Sep 16, 2022 - 10:36 PDT

This incident affected: Relate.