Spinning Wheel When Creating Content

Incident Report for Hearsay Systems

Postmortem

Timeline

Start time: July 10th, 4:36PM PST
End time: July 11th, 10:52AM PST

Customer Impact

On July 10th, we updated our software to use a new domain for loading user interfaces. This caused issues for customers with strict domain allowlists, affecting parts of our application like the content library. Those without strict allowlists were unaffected. There was no data loss or data corruption.

Resolution

The change was reverted on July 11th, restoring service for affected customers.

Future improvements

We plan to implement end to end tests that check against our allow list to ensure no changes like this affect those users in the future. We also have planned infrastructure improvements to mitigate the number of UIs going down at the same time.

Posted Jul 16, 2024 - 14:14 PDT

Resolved

We are now marking this incident as resolved. After we complete a full investigation with our Engineering Team, we will publish a post-mortem. Again, thank you for your patience.

Posted Jul 11, 2024 - 12:29 PDT

Monitoring

Hearsay's engineers were able to resolve the incident regarding the Content Library. Posts should now be visible in the Content Library, and reports can be downloaded successfully. We will continue to monitor the situation, and we truly apologize for this disruption. We will publish a post-mortem after we complete a full investigation. Thank you so much for your patience and understanding.

Posted Jul 11, 2024 - 11:38 PDT

Update

Our engineering team continues to investigate the Content Library incident thoroughly to find a permanent solution. Some users may also experience issues with the download button missing from reports or errors downloading a report.
Rest assured, this remains a priority for our team, and we will provide you with an update shortly. We apologize for any inconvenience this has caused and appreciate your patience and understanding.

Posted Jul 11, 2024 - 10:29 PDT

Update

Hearsay's engineers are still investigating this incident regarding the content library and are committed to resolving it as quickly as possible. As our investigation continues, another update will be published later. We greatly appreciate your patience as we navigate this matter.

Posted Jul 11, 2024 - 08:57 PDT

Update

We are continuing our investigation of this matter as this remains our most urgent priority. Hearsay's engineers are diligently working to resolve this as quickly as possible. We apologize for any disruption this may have caused and will provide another update shortly.

Posted Jul 11, 2024 - 08:16 PDT

Investigating

Hearsay is currently experiencing an incident where users may view a spinning wheel when creating content within the Hearsay platform. This is currently preventing users from viewing the Content Library. We are treating this as our top priority and will provide an update on this incident shortly.

Posted Jul 11, 2024 - 07:40 PDT

This incident affected: Social.