Timeline:

Start Time: 01/25/2024 01:44 PM PT

End Time: 01/25/2024 02:48 PM PT

Customer Impact:

Hearsay’s engineers made an update to how users can search for phone numbers, and this specific index was not reindexed prior to the production deployment, resulting in unavailability across our platform.

Resolution:

Engineers reverted back to a previous version of the feature flag that contained this index and have since reindexed any missing indices.

Future Improvements:

Hearsay will improve our approach to production deployments that contain feature flags, specifically investigating how we can implement a slower, phased rollout. Hearsay’s engineers will also invest in more production-like load testing environments to catch future edge cases.