Timeline:
Start Time: 01/22/2021 11:30 AM PT
End Time: 01/22/2021 12:45 PM PT
Customer Impact:
Hearsay Relate was inaccessible to some users due to an error in the web client. The cause of the error was a bug in a specific configuration setting that only some organizations had enabled. Organizations that did not have this configuration as well as those using the Hearsay Relate mobile app were unaffected. Inbound messages were being received and were not lost due to this incident.
Resolution:
The issue was fixed by temporarily disabling the setting that prevented uses from accessing Hearsay Relate at 12:16 PM PT. Hearsay Engineers were then able to patch the error that had been added at 12:45 PM PT and turn the setting back on for users putting us back into a stable state.
Future Improvements:
In response to this incident, we are improving our testing process in order to prevent configuration-specific errors like this in the future.