Exclaimer Status

Resolved US: Message Processing Delays

Summary: We have identified an issue with increased message duration through the Exclaimer Cloud Platform. The cause of this issue is a confirmed Microsoft 365 incident: EX252124. Issue: A subset of customers may experience mail delays due to issues with O365 not processing SMTP requests in a timely manner. Symptoms: The potential of messages being delayed as they wait to be retried within O365. Though delayed, the emails will be processed and delivered with a signature. Next Update In: 1h

Resolved US: Issues with Imprinting Service

Summary : A subset of customers may experience issues with mail delays/messages not being imprinted when mail is routed through our US2 datacenter Issue : We are currently experiencing an issue in our US2 datacenter, however we are unable to fail messages away from that datacenter due to current issues with the Microsoft Azure Portal meaning that SRE engineers are unable to enact remediation work. Symptoms : Messages delayed or not imprinted Next Update In : 1h

Resolved US: O365 Message Processing Issues

Summary: We are investigating an issue email processing in our US datacenter. Issue: A subset of customers may experience mail delays due to issues with O365 not processing SMTP requests in a timely manner Symptoms: The potential of messages being delayed as they wait to be retried within O365. Though delayed, the emails will be processed and delivered with a signature. Next Update In: 1h

Resolved US: GSuite Message Delays

Summary: We are investigating an issue with mail routing for GSuite. Issue: A subset of customers may experience mail delays due to GSuite not accepting all connections attempted by Exclaimer Cloud Symptoms: The potential of messages being delayed due to retries within the GSuite system. Though delayed, the emails will be processed and delivered with a signature. Next Update In: 1h

Resolved Microsoft Issues causing incomplete user data

Summary: Microsoft Issues causing incomplete user data Issue: Exclaimer are aware of an issue within the Microsoft Azure Active Directory, this issue can be found in the Microsoft Office admin portal, ID: EX226193 - "Users may be unable to access Exchange Online via multiple connection methods". This issue relates to problems within the Microsoft directory infrastructure - the infrastructure that Exclaimer Cloud queries when syncing user data for use in signatures. When Exclaimer Cloud performs a data sync a small proportion of customers' data is not provided by Microsoft and is not stored by Exclaimer. Symptoms: This is experienced by a subset of customers through missing users/groups in the signature management UI, and missing signatures on some users emails. If you believe you are impacted then we recommend performing a manual synchronisation via the settings pane in the Exclaimer UI as Microsoft are reporting that the "majority of services have been restored" Next Update In: 2h

Resolved US: G Suite issues affecting Exclaimer Cloud

Issue: Google's servers are sporadically rejecting connections from Exclaimer Cloud to G Suite. Symptoms: We are currently seeing delays in processing messages due earlier processing issues that occurring in GSuite, this is due to the backlog of messages created by this issue - https://status.cloud.google.com/incident/zall/20005 and rate limiting applied by Google. Next Update In: 1h

Resolved US: Delays to G Suite messages.

Summary of Impact: Between 15:00 and 22:00 UTC customers may have experienced issues with delayed delivery of emails due to G Suite rejecting connections from Exclaimer Cloud. Root Cause: Transient rejections were issued by G Suite's SMTP relay service when Exclaimer Cloud attempted to initiate the connection. Exclaimer Cloud will not accept the email from G Suite until we have confirmed we are able to complete the return connection - this ensures that email is always 'owned' by G Suite and never Exclaimer. As we are unable to create the return connection we are in-turn unable to accept the message from G Suite. These emails then sit in G Suite's own queue awaiting retry - this queue has long waits between retries, exacerbating the issue (~5min/15min/40min). Mitigation: G Suite stopped rejecting sessions and the issue was resolved. Next Steps: We are working closely with Google and are deploying a number of changes to the way that Exclaimer handles retries and distributes connections for G Suite emails.

Resolved AU: Microsoft 365 Issues impacting Exclaimer Cloud

Summary of Impact: Between 01:40 and 05:10 UTC customers may have experienced issues with queued messages in Exchange Online (EXO) due to Microsoft Issues. Root Cause: Issues with Microsoft 365 meant that some messages were not being received by 365 after having been sent to Exclaimer. Microsoft have published the root cause to this issue under ID EX196121. Mitigation: Microsoft determined that a change made to their 365 infrastructure was the cause of the problems and reverted the change. Next Steps: Microsoft are reviewing their deployment procedures to help prevent similar problems in the future. More information can be obtained directly from Microsoft by raising a case with them against the ID quoted above.

Resolved US: Exchange Online causing mail delays for Exclaimer Customers

Summary of Impact: Between 13:50 and 16:30 UTC customers may have experienced issues with queued messages in Exchange Online (EXO) due to Microsoft Issues. Root Cause: Issues with Microsoft Office 365 meant that messages were not being delivered to Exclaimer in a timely manner. A small percentage of messages were not reaching our infrastructure and therefore they entered EXOs retry loop. Mitigation: Microsoft resolved their service degradation. Once EXO could send us the message correctly, customers no longer experienced any issues. Next Steps: We are working with Microsoft to understand why their service suffered this issue. We have made changes to our monitoring platform to help us alert customers faster if we see similar issues again.

Resolved UK: Mail processing delays caused by Office 365 issues

Summary of Impact: Between 09:00 and 12:30 UTC customers may have experienced issues with delayed delivery and/or non-imprinted email due to Office 365 Issues Root Cause: Transient issues with Exchange Online's SMTP servers meant that O365 was unable to send a subset of email to Exclaimer Cloud between 09:00 and 10:50. Approx. 20% of messages were queued within O365 during this period. Exclaimer processed the backlog of queued messages between 10:50 and 12:30 Mitigation: Microsoft resolved their issue and service was restored. Exclaimer Cloud scaled to meet demand during its busiest periods. Next Steps: We are working with Microsoft to understand the issue in more detail and if Exclaimer Cloud can be more resilient to these sorts of issues in the future.