Meta Functionality Temporarily Inactive

Incident Report for Hearsay Systems

Postmortem

Timeline:

Start Time: 06/11/24 11:01 AM PT

End Time: 06/13/24 2:18 PM  PT

Customer Impact:

Meta continuously tests all third-party applications that utilize their Facebook integration or authentication to ensure companies are complying with Meta’s policies.  Meta frequently tests Hearsay’s Facebook integration and authentication, and Hearsay always provides the necessary information to our partners at Meta to pass their third-party application testing process.  During the most recent test of Meta’s Facebook authentication, Meta performed an ad-hoc review of Hearsay’s platform with invalid credentials that were not provided by Hearsay, and without contacting Hearsay.  This resulted in Meta deactivating Hearsay’s Facebook integration on June 11, 2024 without notice when those credentials did not work.  Hearsay immediately appealed the decision to deactivate our integration.  Meta continued to respond over the next two days trying to retest our authentication, leading them to their own Meta error page, resulting in numerous back-and-forths of Hearsay clearly outlining and explaining to Meta that Meta needs to reactivate our account for them to test accurately.

For additional context, Meta has reduced its support staff, which we feel has significantly impacted the size and availability of their API and App Review teams.  This translates to longer than desired response times for support issues, impacting vendors like Hearsay and end users like Hearsay’s customers.

Resolution:

Hearsay was able to work with Meta’s testing team to get them to reactivate Hearsay’s account on June 13, 2024.

Future Improvements:

Hearsay will deprecate using Facebook as an authentication option and eventually remove it altogether in favor of more common authentication options (e.g., Microsoft or Google) for those without SAML providers.  Hearsay is also removing our mobile applications from the scope of Meta reviews to minimize the footprint of our services subject to periodical reviews, and to reduce the likelihood of shutdown without warning.

Posted Jun 14, 2024 - 13:06 PDT

Resolved

We can confirm that Meta was able to fix this matter, and all Facebook and Instagram functionalities are now fully restored. Users will be able to publish content to Facebook and Instagram, update their Meta profiles, and connect their Meta accounts to Hearsay. Any users who log in to Hearsay via Facebook authentication will now be able to log back in.

Hearsay will crawl back the data during the outage period to capture activities for supervision and archiving. However, posts that were scheduled during this period will need to be manually rescheduled. Your CSMs will provide a list of affected posts for your organization.

Thank you for your patience and understanding as we navigated this incident with our partners at Meta.
Posted Jun 13, 2024 - 14:29 PDT

Update

Hearsay is still actively working with Meta to fully resolve this matter. We sincerely apologize for any disruption this may have caused to your business operations. Rest assured, we are committed to keeping you informed with regular updates as the situation progresses. Thank you for your patience and understanding.
Posted Jun 13, 2024 - 07:29 PDT

Update

Hearsay continues to collaborate closely with our partners at Meta to resolve this issue.

Hearsay will be able to crawl back the data during the outage period to capture activities for supervision and archiving. However, posts that were scheduled during this period will need to be manually rescheduled. Your CSMs will provide a list of affected posts for your organization.

We greatly appreciate your patience and will share another update as soon as we have more information.
Posted Jun 12, 2024 - 17:06 PDT

Update

Hearsay continues to collaborate closely with our partners at Meta to resolve this issue. We greatly appreciate your patience and will share another update as soon as we have more information.
Posted Jun 12, 2024 - 14:44 PDT

Update

Hearsay is still actively collaborating with our trusted partners at Meta to diligently address and resolve this pressing matter. We are committed to providing updates on the progress of this incident until Meta has successfully and completely resolved it.
Posted Jun 12, 2024 - 11:44 PDT

Update

Hearsay is continuing to work with Meta to have them resolve this incident. We sincerely apologize for any disruption and will provide more information later today.
Posted Jun 12, 2024 - 06:59 PDT

Update

Hearsay is actively collaborating with our partners at Meta to address the current situation. We deeply regret any inconvenience this may have caused. We will promptly provide an update as soon as we have more information.
Posted Jun 11, 2024 - 16:06 PDT

Update

Hearsay has already contacted our counterparts at Meta to resolve this matter. Once we hear back and have more information to share, we will provide an update on this incident.
Posted Jun 11, 2024 - 11:57 PDT

Investigating

At this moment, Hearsay is experiencing an incident where functionality around Facebook and Instagram within the Hearsay platform is temporarily down. This includes connecting accounts and publishing content to Facebook and Instagram. For any customers logging into Hearsay with Facebook authentication, you will be unable to log in to Hearsay until Meta resolves this matter.

We truly apologize for this major disruption. Hearsay is prioritizing this matter immediately and will provide an update shortly.
Posted Jun 11, 2024 - 11:21 PDT
This incident affected: Partner Integrations (Facebook, Instagram).