Internal Server Error Page

Incident Report for Hearsay Systems

Postmortem

Timeline:

Start Time: 07/25/2024 6:00 AM PT

End Time: 07/25/2024 6:10 AM PT

Customer Impact:

During a routine deployment to the production environment, auto-scaling was incorrectly decreased, resulting in limited server bandwidth. Users saw an Internal Server Error page when logging in or navigating the Hearsay platform.

Resolution:

Hearsay’s engineers quickly reverted to the previous state of the platform, which increased auto-scaling to normal thresholds.

Future Improvements:

Hearsay will improve our approach to production deployments by fixing the root cause of the failing automation and introducing new monitoring and alerting. We will also update all relevant internal technical documentation as necessary.

Posted Aug 06, 2024 - 06:41 PDT

Resolved

Hearsay's engineers have monitored this situation and confirmed that it has been fully resolved. We will conduct a formal post-mortem and publish our findings regarding this incident. Thank you for your patience and understanding.

Posted Jul 25, 2024 - 06:43 PDT

Monitoring

Hearsay's engineers quickly reverted their change, and the error page no longer appears. We will continue monitoring the situation and provide an update shortly confirming the resolution.

Posted Jul 25, 2024 - 06:18 PDT

Investigating

Hearsay is experiencing an incident in which users may see a 500 Internal Server Error page. Our engineers have already been alerted and are prioritizing this to ensure a quick resolution. We truly apologize for this disruption and will publish an update shortly.

Posted Jul 25, 2024 - 06:13 PDT

This incident affected: Platform.