Database failover

Through our investigation we have identified a subset of customers that were potentially affected by this issue. If you are one of the customers that we have identified, our Customer Success team will be reaching out to you via phone or email.

What we know at this time is that we experienced multiple hardware failures that required us to roll back a subset of our databases to an earlier restore point. Any data that was added, edited, or deleted between 10:00pm EDT, and 11:00pm EDT on 10/22/2024 for these identified customers has been lost and our Customer Success team will be reaching out to you to answer any further questions.

We are still in the process of investigating alternative remediation steps and will be conducting an internal postmortem to better understand and prioritize future infrastructure changes and processes to remove the possibility of this ever happening again. We know that any amount of data loss can be a problem, and we apologize for any inconvenience this issue has caused.

On 10/22/2024 at approximately 10:30 PM EDT FMX experienced a server failure that required manual intervention to restore it. On 10/23/2024 at 4:10 AM EDT we had successfully restored the affected databases and are continuing to investigate the extent of data that was not restored between 10 and 11 pm EDT