Intermittent Error And Slow Load Times

Incident Report for Hearsay Systems

Postmortem

Timeline:

Start Time: 07/07/23 8:30 AM PT

End Time: 07/07/23 8:48 AM PT

Customer Impact:

A recent code deployment to Hearsay’s settings service caused an increase in load times, which resulted in users experiencing slowness on the Hearsay site or the webpage throwing an error message.

Resolution:

Hearsay’s engineers increased the capacity of the application to provide extra bandwidth to the settings service, which temporarily resolved the issue. After the incident was resolved, and with thorough testing, Hearsay’s engineers enabled caching for this service to prevent a future spike in calls and load times.

Future Improvements:

In addition to enabling caching, Hearsay will look into fine-tuning our automatic scaling to avoid similar instances in the future. Extra alerts will be added for the settings service, which will allow our engineers to react much faster in resolving these incidents if they should come up.

Posted Jul 18, 2023 - 08:38 PDT

Resolved

Hearsay's engineers resolved this matter regarding longer-than-expected load times and intermittent error messages. We will conduct a root cause analysis and once we have collected all the necessary information, we will add a post-mortem to this incident. Thank you for your patience and understanding as we worked to resolve this matter.

Posted Jul 07, 2023 - 09:19 PDT

Investigating

Hearsay is currently investigating a matter that we saw earlier today where some users may experience slower than usual load times on our website as well as potentially seeing an error page. We have already started reviewing this matter and we will provide an update shortly. We apologize for any inconvenience and thank you for your patience.

Posted Jul 07, 2023 - 08:45 PDT

This incident affected: Platform.