Intermittent Linkedin Publishing Error

Incident Report for Hearsay Systems

Postmortem

Timeline:

Start Time: 04/20/23 1:54 PM PT

End Time: 04/20/23 4:30 PM PT

Customer Impact:

Hearsay experienced an incident where users would intermittently encounter extended loading times or time-outs when publishing content to LinkedIn. This was due to an increase in larger-than-expected files being published to LinkedIn at the same time while Hearsay queued up the posts.

Resolution:

Due to the nature of the incident, which was caused by throttling in our backend services and the slow upload speed to LinkedIn, Hearsay was eventually able to catch up with processing the queues and restored service back to normal.

Future Improvements:

Hearsay’s engineers updated our processing power in order to handle incoming messages much faster. Hearsay will also implement a system to handle on-demand and scheduled posts with a different priority to avoid any future negative experience given similar circumstances.

Posted May 16, 2023 - 10:02 PDT

Resolved

Hearsay Engineers have resolved the ongoing issue with publishing to Linkedin. Users may still encounter some delays when making posts to Linkedin, but the posts in question will ultimately publish. Our systems are currently working through a backlog of posts and Hearsay Engineering will continue monitoring through completion. We would like to express our sincere apologies for any inconvenience caused by the recent disruption. We will be posting a full post-mortem on the incident with additional details on the issue and how it can be avoided in the future. We greatly appreciate your patience and understanding as we worked to resolve this issue.

Posted Apr 20, 2023 - 16:32 PDT

Update

Hearsay Engineers have identified the issue and continue to work toward a resolution. We will have another update soon, and again, we apologize for this disruption.

Posted Apr 20, 2023 - 15:40 PDT

Identified

Hearsay Engineers have identified the issue and are currently working toward a resolution. We will have another update soon, and again, we apologize for this disruption.

Posted Apr 20, 2023 - 15:07 PDT

Update

Hearsay Engineers continue to actively investigate this matter and are treating this as our top priority. We truly apologize for any disruption and will provide an update shortly.

Posted Apr 20, 2023 - 14:35 PDT

Investigating

Hearsay is currently encountering an intermittent issue when attempting to publish to Linkedin. Users may encounter extended loading times or potential time-outs. Hearsay's engineers are actively looking into this matter and are treating this as our top priority. We truly apologize for any disruption and will provide an update shortly.

Posted Apr 20, 2023 - 14:07 PDT

This incident affected: Social.