Exclaimer Status

Resolved US: O365 Message Processing Issues

Summary: We are investigating an issue email processing in our US datacenter. Issue: A subset of customers may experience mail delays due to issues with O365 not processing SMTP requests in a timely manner Symptoms: The potential of messages being delayed as they wait to be retried within O365. Though delayed, the emails will be processed and delivered with a signature. Next Update In: 1h

Resolved US: GSuite Message Delays

Summary: We are investigating an issue with mail routing for GSuite. Issue: A subset of customers may experience mail delays due to GSuite not accepting all connections attempted by Exclaimer Cloud Symptoms: The potential of messages being delayed due to retries within the GSuite system. Though delayed, the emails will be processed and delivered with a signature. Next Update In: 1h

Resolved Microsoft Issues causing incomplete user data

Summary: Microsoft Issues causing incomplete user data Issue: Exclaimer are aware of an issue within the Microsoft Azure Active Directory, this issue can be found in the Microsoft Office admin portal, ID: EX226193 - "Users may be unable to access Exchange Online via multiple connection methods". This issue relates to problems within the Microsoft directory infrastructure - the infrastructure that Exclaimer Cloud queries when syncing user data for use in signatures. When Exclaimer Cloud performs a data sync a small proportion of customers' data is not provided by Microsoft and is not stored by Exclaimer. Symptoms: This is experienced by a subset of customers through missing users/groups in the signature management UI, and missing signatures on some users emails. If you believe you are impacted then we recommend performing a manual synchronisation via the settings pane in the Exclaimer UI as Microsoft are reporting that the "majority of services have been restored" Next Update In: 2h

Resolved US: G Suite issues affecting Exclaimer Cloud

Issue: Google's servers are sporadically rejecting connections from Exclaimer Cloud to G Suite. Symptoms: We are currently seeing delays in processing messages due earlier processing issues that occurring in GSuite, this is due to the backlog of messages created by this issue - https://status.cloud.google.com/incident/zall/20005 and rate limiting applied by Google. Next Update In: 1h

Resolved US: Delays to G Suite messages.

Summary of Impact: Between 15:00 and 22:00 UTC customers may have experienced issues with delayed delivery of emails due to G Suite rejecting connections from Exclaimer Cloud. Root Cause: Transient rejections were issued by G Suite's SMTP relay service when Exclaimer Cloud attempted to initiate the connection. Exclaimer Cloud will not accept the email from G Suite until we have confirmed we are able to complete the return connection - this ensures that email is always 'owned' by G Suite and never Exclaimer. As we are unable to create the return connection we are in-turn unable to accept the message from G Suite. These emails then sit in G Suite's own queue awaiting retry - this queue has long waits between retries, exacerbating the issue (~5min/15min/40min). Mitigation: G Suite stopped rejecting sessions and the issue was resolved. Next Steps: We are working closely with Google and are deploying a number of changes to the way that Exclaimer handles retries and distributes connections for G Suite emails.

Resolved AU: Microsoft 365 Issues impacting Exclaimer Cloud

Summary of Impact: Between 01:40 and 05:10 UTC customers may have experienced issues with queued messages in Exchange Online (EXO) due to Microsoft Issues. Root Cause: Issues with Microsoft 365 meant that some messages were not being received by 365 after having been sent to Exclaimer. Microsoft have published the root cause to this issue under ID EX196121. Mitigation: Microsoft determined that a change made to their 365 infrastructure was the cause of the problems and reverted the change. Next Steps: Microsoft are reviewing their deployment procedures to help prevent similar problems in the future. More information can be obtained directly from Microsoft by raising a case with them against the ID quoted above.

Resolved US: Exchange Online causing mail delays for Exclaimer Customers

Summary of Impact: Between 13:50 and 16:30 UTC customers may have experienced issues with queued messages in Exchange Online (EXO) due to Microsoft Issues. Root Cause: Issues with Microsoft Office 365 meant that messages were not being delivered to Exclaimer in a timely manner. A small percentage of messages were not reaching our infrastructure and therefore they entered EXOs retry loop. Mitigation: Microsoft resolved their service degradation. Once EXO could send us the message correctly, customers no longer experienced any issues. Next Steps: We are working with Microsoft to understand why their service suffered this issue. We have made changes to our monitoring platform to help us alert customers faster if we see similar issues again.

Resolved UK: Mail processing delays caused by Office 365 issues

Summary of Impact: Between 09:00 and 12:30 UTC customers may have experienced issues with delayed delivery and/or non-imprinted email due to Office 365 Issues Root Cause: Transient issues with Exchange Online's SMTP servers meant that O365 was unable to send a subset of email to Exclaimer Cloud between 09:00 and 10:50. Approx. 20% of messages were queued within O365 during this period. Exclaimer processed the backlog of queued messages between 10:50 and 12:30 Mitigation: Microsoft resolved their issue and service was restored. Exclaimer Cloud scaled to meet demand during its busiest periods. Next Steps: We are working with Microsoft to understand the issue in more detail and if Exclaimer Cloud can be more resilient to these sorts of issues in the future.

Resolved Global: Mail processing delays caused by Microsoft networking issues

Summary of Impact: Between 19:36 and 22:28 UTC customers may have experienced issues with delayed delivery and/or non-imprinted email due to widespread Microsoft network issues. Root Cause: Microsoft experienced issues with their DNS service as summarised - https://azure.microsoft.com/en-gb/status/history/. This issue caused internal routing issues between Microsoft 365 and Microsoft Azure services and therefore Exclaimer Cloud was not able to route messages back to 365 correctly. Mitigation: Microsoft resolved their global issue and service was restored. Next Steps: We are working with Microsoft to understand the issue in more detail and if Exclaimer Cloud can be more resilient to these sorts of issues in the future.

Resolved Global: Issues with the Exclaimer Cloud Portal

Summary of Impact: Between 07:30 and 11:15 UTC the Exclaimer Cloud Portal was down, affecting users ability to access templates. All mail flow/imprinting services were unaffected. Root Cause: Our back end data service was experiencing an issue with the continuous replication from primary to secondary sites. Engineers took the portal offline as a precaution to ensure that there was no data loss or corruption. Mitigation: Engineers resolved the issue and brought the portal back online. Next Steps: A project to change the way that our replication works has been ongoing for some time, this work requires a significant redesign of the Portal infrastructure and is nearing the final stages of completion. This should make the Portal more resilient and reliable.