Intermittent Hearsay Error
Incident Report for Hearsay Systems
Postmortem

Timeline:

Start Time: 09/13/22 11:15 AM PT

End Time: 09/13/22 12:40 PM PT

Customer Impact:

Hearsay’s web application experienced an outage from approximately 11:15 AM PT to 12:40 PM PT. During this time, our public-facing web interface was unavailable. Users who attempted to access the web application during this time were served error messages. Note that no data or messages were lost.

Resolution:

Hearsay’s Engineers were able to trace this outage to resourcing issues with our cloud provider during a deployment. At the time of the incident, our production system had requested additional cloud resources that were not provided by the cloud provider. As a result, our system health began to deteriorate and my.hearsaysocial.com became unstable. Production was stabilized once the resources were requested.

Future Improvements:

Hearsay will add enhanced health checks to our deployment process. These health checks will ensure the needed cloud resourcing is available prior to deployment to ensure we do not reencounter this issue.

Posted Sep 21, 2022 - 11:25 PDT

Resolved
This incident has been resolved.
Posted Sep 13, 2022 - 13:17 PDT
Monitoring
Hearsay Engineers are seeing recovery from the intermittent access issues. We will continue to monitor the situation and provide a full analysis after a thorough investigation. Thank you so much for your patience, and we truly apologize for any disruption.
Posted Sep 13, 2022 - 13:11 PDT
Identified
Hearsay Engineers have identified the issue as stemming from an issue with one of our third-party providers, AWS. We are currently working towards a resolution. We will have another update soon, and again, we apologize for this disruption.
Posted Sep 13, 2022 - 12:22 PDT
Update
Hearsay Engineering is continuing its investigation of the previously mentioned access issue. We are actively looking into this matter and are treating this as our top priority. We will have an update shortly and apologize for any disruption to your business.
Posted Sep 13, 2022 - 11:55 PDT
Investigating
We're currently experiencing an intermittent access issue with some users experiencing error messages when trying to access Hearsay. We are actively looking into this matter and are treating this as our top priority. We will have an update shortly and apologize for any disruption to your business.
Posted Sep 13, 2022 - 11:31 PDT
This incident affected: Platform.