Timeline:
Start Time: 09/13/22 11:15 AM PT
End Time: 09/13/22 12:40 PM PT
Customer Impact:
Hearsay’s web application experienced an outage from approximately 11:15 AM PT to 12:40 PM PT. During this time, our public-facing web interface was unavailable. Users who attempted to access the web application during this time were served error messages. Note that no data or messages were lost.
Resolution:
Hearsay’s Engineers were able to trace this outage to resourcing issues with our cloud provider during a deployment. At the time of the incident, our production system had requested additional cloud resources that were not provided by the cloud provider. As a result, our system health began to deteriorate and my.hearsaysocial.com became unstable. Production was stabilized once the resources were requested.
Future Improvements:
Hearsay will add enhanced health checks to our deployment process. These health checks will ensure the needed cloud resourcing is available prior to deployment to ensure we do not reencounter this issue.