Intermittent Error And Slow Load Times

Incident Report for Hearsay Systems

Postmortem

Timeline:

Start Time: 07/07/23 6:23 AM PT

End Time: 07/07/23 7:11 AM PT

Customer Impact:

A recent code deployment to Hearsay’s settings service caused an increase on load times, which resulted in users experiencing slowness on the Hearsay site or the webpage throwing an error message.

Resolution:

Hearsay’s engineers increased the capacity of the application to provide extra bandwidth to the settings service, which temporarily resolved the issue. After the incident was resolved, and with thorough testing, Hearsay’s engineers enabled caching for this service to prevent a future spike in calls and load times.

Future Improvements:

In addition to enabling caching, Hearsay will look into fine-tuning our automatic scaling to avoid similar instances in the future. Extra alerts will be added for the settings service, which will allow our engineers to react much faster in resolving these incidents if they should come up.

Posted Jul 18, 2023 - 08:32 PDT

Resolved

Hearsay's engineers discovered high load times and increased our bandwidth, which resulted in resolving this incident. We will conduct a formal root cause analysis and once we have collected all the necessary information, we will add a post-mortem to this incident. Thank you for your patience and understanding as we worked to resolve this matter.

Posted Jul 07, 2023 - 07:11 PDT

Update

Hearsay's engineers are still investigating this incident where the Hearsay site is slow to load and may throw an error page. We are working hard to determine the root cause and the proper solution. Another update will be provided to this incident shortly. Thank you so much for your patience.

Posted Jul 07, 2023 - 07:00 PDT

Investigating

Hearsay is currently investigating a matter where some users may experience slower than usual load times on our website as well as potentially seeing an error page. Hearsay's engineers have already begun looking into this matter and we will provide an update shortly. We apologize for any inconvenience and thank you for your patience.

Posted Jul 07, 2023 - 06:32 PDT

This incident affected: Platform.