Intermittent Error Page

Incident Report for Hearsay Systems

Postmortem

Timeline:

Start Time: 03/07/2024 5:55 AM PT

End Time: 03/07/2024 7:01 AM PT

Customer Impact:

Hearsay deployed an update where an index was added to a database. This resulted in an increase in CPU consumption, which led to Hearsay’s users seeing an error page on our platform. There was no data loss during this time.

Resolution:

Hearsay’s engineers identified the regression and reverted back the change to our database.

Future Improvements:

Hearsay will add more internal alerting with more aggressive thresholds to ensure we act even faster if this were to reoccur. Additionally, Hearsay’s engineers will avoid making similar changes during, or close to, business hours.

Posted Mar 12, 2024 - 13:29 PDT

Resolved

Hearsay's engineers were able to resolve this matter, and users should no longer experience the error.
We truly apologize for this disruption and we will publish a post-mortem after we complete a full investigation. Thank you so much for your patience and understanding.

Posted Mar 07, 2024 - 07:28 PST

Update

Our engineering team is still investigating this matter to ensure a permanent solution. We will provide another update soon to share our progress on this incident. Again, we want to apologize for this disruption to your day. Thank you for your patience and understanding.

Posted Mar 07, 2024 - 06:45 PST

Investigating

Hearsay is currently experiencing an incident where users may intermittently see an error page. Our engineers are investigating this matter now and are treating this as our top priority. We apologize for this interruption and will provide an update shortly.

Posted Mar 07, 2024 - 06:13 PST

This incident affected: Platform.