Intermittent Internal Server Error

Incident Report for Hearsay Systems

Postmortem

Timeline:

Start Time: 10/19/2023 11:00 AM PT

End Time: 10/19/2023 11:21 AM PT

Customer Impact:

Hearsay is set to auto-scale when an increase in requests occurs. However, a conflict transpired when one instance of our auto-scalers failed to respond to incoming traffic. This led to users intermittently experiencing a 500 Internal Server Error.

Resolution:

Hearsay’s engineers manually scaled up the service right away to handle the excess load, which solved the incident immediately.

Future Improvements:

Auto-scaling has been updated to observe a different threshold value. Additionally, we are already exploring adding a more resilient auto-scaling policy to ensure this does not reoccur.

Posted Nov 30, 2023 - 09:36 PST

Resolved

Hearsay's engineers were able to resolve this matter, and users should no longer experience the error from before. A formal post-mortem will be provided to this incident after a thorough investigation from our engineers. We greatly appreciate your patience and understanding during this time.

Posted Oct 19, 2023 - 11:24 PDT

Investigating

Hearsay users may intermittently experience a 500 Internal Server Error when using Hearsay. Our engineers are treating this as our top priority and will work to resolve this as quickly as possible. We truly apologize for this experience and will provide an update shortly.

Posted Oct 19, 2023 - 11:06 PDT

This incident affected: Platform.