Filtering Service Outage

This incident has been resolved.

Normal service of the SMX Filtering Gateway has now been restored. We have identified 43 customer configuration changes made between the most recent backup and the outage that we will manually restore over the course of the day.

Access to the SMX email admin portal has been restored.

We have completed the database restore on 1 of 6 nodes, and mail flow through SMX Filtering is now working again, albeit with degraded performance. We are now working to configure replication from the database primary host to the remaining 5 read-only replicas, which will restore the service to full working capacity. There will be further work required to manually recover system changes made between the most recent backup (6AM this morning) and the time at which the database became corrupted.

Database restore and re-index is in progress, after which we will perform basic health checking before bringing it back into serivce. We will provide a further update in 30 minutes or when service is restored, which ever comes first.

We believe the problem relates to corruption of an internal configuration database. We are working to restore the database from a recent backup now. We have manually disabled the SMX Email Admin Portal to prevent writes to the database while it is being restored.

We are aware of problems affecting the SMX NZ/AU Filtering gateway. SMX technical staff are investigating and will provide a further update in 30 minutes or as new information becomes available.