API and SMTP relay degradation

We have been made aware of this issue reoccurring again and are currently investigating.

The incident has already been mitigated by killing the affected pod. We will share an update once we know more.

On September 1, between 04:01 and 04:06 UTC, one of our API instances returned errors for a small number of requests. Some customers sending email through our API or SMTP service might have seen a submission or authentication error. Other instances remained healthy, and most requests continued normally.

No email already accepted by Lettermint was lost. Failed requests stopped before we accepted or created the message, so clients could retry them after service recovery. A small number of email-open events might not have been recorded. No stored customer data was changed or exposed.

The cause was a defect in the PHP JIT. We already had a safeguard for this issue, but it did not cover this additional code path.

We expanded the safeguard and removed the affected date calculation from the application startup process. This lets us keep the performance optimisation active for the rest of the application. We will also install the permanent PHP fix when it becomes available in our production runtime which at the time of writing is not the case.

We continue to monitor the service. We are sorry for the disruption.

We are investigating failing requests and retries affecting SMTP relay.