Internal Server Error for Supervision

Incident Report for Hearsay Systems

Postmortem

Timeline:

Start Time: 11/20/24 10:06 AM PT

End Time: 11/20/24 20:42 PM PT

Customer Impact:

Hearsay was unable to display the results of a compliance search for organization administrators. When admins attempted to view their content feeds or compliance reviews, they saw an error page. This was due to an issue that arose during an infrastructure upgrade.

Resolution:

Hearsay’s engineers rebuilt the infrastructure to get it back to a known working state.

Future Improvements:

Hearsay already has plans to upgrade our search domains to a new service, which would allow for much quicker reindexing. Additionally, Hearsay’s processes around these tasks have been updated to ensure this type of event does not happen during business hours.

Posted Nov 26, 2024 - 09:59 PST

Resolved

Hearsay's engineers were able to resolve this matter, and administrators should no longer see error messages. After we complete a full and thorough investigation, a post-mortem will be published for this incident. Thank you for your patience as we resolved this incident.

Posted Nov 20, 2024 - 13:00 PST

Identified

Hearsay engineers have identified the root cause and are working on a fix. We will share another update shortly. Again, thank you for your patience while we resolve this for you.

Posted Nov 20, 2024 - 11:41 PST

Investigating

Hearsay is currently experiencing an incident where the Review tab for administrators results in a 500 Internal Server Error or a 504 Gateway Timeout error. Our engineers are working on a quick resolution to fix this immediately. An update will be provided shortly, and we apologize for the disruption.

Posted Nov 20, 2024 - 10:52 PST

This incident affected: Compliance.