Resolved Microsoft Issues causing incomplete user data
Summary: Microsoft Issues causing incomplete user data Issue: Exclaimer are aware of an issue within the Microsoft Azure Active Directory, this issue can be found in the Microsoft Office admin portal, ID: EX226193 - "Users may be unable to access Exchange Online via multiple connection methods". This issue relates to problems within the Microsoft directory infrastructure - the infrastructure that Exclaimer Cloud queries when syncing user data for use in signatures. When Exclaimer Cloud performs a data sync a small proportion of customers' data is not provided by Microsoft and is not stored by Exclaimer. Symptoms: This is experienced by a subset of customers through missing users/groups in the signature management UI, and missing signatures on some users emails. If you believe you are impacted then we recommend performing a manual synchronisation via the settings pane in the Exclaimer UI as Microsoft are reporting that the "majority of services have been restored" Next Update In: 2h
Resolved US: G Suite issues affecting Exclaimer Cloud
Issue: Google's servers are sporadically rejecting connections from Exclaimer Cloud to G Suite. Symptoms: We are currently seeing delays in processing messages due earlier processing issues that occurring in GSuite, this is due to the backlog of messages created by this issue - https://status.cloud.google.com/incident/zall/20005 and rate limiting applied by Google. Next Update In: 1h
Resolved US: Delays to G Suite messages.
Summary of Impact: Between 15:00 and 22:00 UTC customers may have experienced issues with delayed delivery of emails due to G Suite rejecting connections from Exclaimer Cloud. Root Cause: Transient rejections were issued by G Suite's SMTP relay service when Exclaimer Cloud attempted to initiate the connection. Exclaimer Cloud will not accept the email from G Suite until we have confirmed we are able to complete the return connection - this ensures that email is always 'owned' by G Suite and never Exclaimer. As we are unable to create the return connection we are in-turn unable to accept the message from G Suite. These emails then sit in G Suite's own queue awaiting retry - this queue has long waits between retries, exacerbating the issue (~5min/15min/40min). Mitigation: G Suite stopped rejecting sessions and the issue was resolved. Next Steps: We are working closely with Google and are deploying a number of changes to the way that Exclaimer handles retries and distributes connections for G Suite emails.
Resolved AU: Microsoft 365 Issues impacting Exclaimer Cloud
Summary of Impact: Between 01:40 and 05:10 UTC customers may have experienced issues with queued messages in Exchange Online (EXO) due to Microsoft Issues. Root Cause: Issues with Microsoft 365 meant that some messages were not being received by 365 after having been sent to Exclaimer. Microsoft have published the root cause to this issue under ID EX196121. Mitigation: Microsoft determined that a change made to their 365 infrastructure was the cause of the problems and reverted the change. Next Steps: Microsoft are reviewing their deployment procedures to help prevent similar problems in the future. More information can be obtained directly from Microsoft by raising a case with them against the ID quoted above.
Resolved US: Exchange Online causing mail delays for Exclaimer Customers
Summary of Impact: Between 13:50 and 16:30 UTC customers may have experienced issues with queued messages in Exchange Online (EXO) due to Microsoft Issues. Root Cause: Issues with Microsoft Office 365 meant that messages were not being delivered to Exclaimer in a timely manner. A small percentage of messages were not reaching our infrastructure and therefore they entered EXOs retry loop. Mitigation: Microsoft resolved their service degradation. Once EXO could send us the message correctly, customers no longer experienced any issues. Next Steps: We are working with Microsoft to understand why their service suffered this issue. We have made changes to our monitoring platform to help us alert customers faster if we see similar issues again.
Resolved UK: Mail processing delays caused by Office 365 issues
Summary of Impact: Between 09:00 and 12:30 UTC customers may have experienced issues with delayed delivery and/or non-imprinted email due to Office 365 Issues Root Cause: Transient issues with Exchange Online's SMTP servers meant that O365 was unable to send a subset of email to Exclaimer Cloud between 09:00 and 10:50. Approx. 20% of messages were queued within O365 during this period. Exclaimer processed the backlog of queued messages between 10:50 and 12:30 Mitigation: Microsoft resolved their issue and service was restored. Exclaimer Cloud scaled to meet demand during its busiest periods. Next Steps: We are working with Microsoft to understand the issue in more detail and if Exclaimer Cloud can be more resilient to these sorts of issues in the future.
Resolved Global: Mail processing delays caused by Microsoft networking issues
Summary of Impact: Between 19:36 and 22:28 UTC customers may have experienced issues with delayed delivery and/or non-imprinted email due to widespread Microsoft network issues. Root Cause: Microsoft experienced issues with their DNS service as summarised - https://azure.microsoft.com/en-gb/status/history/. This issue caused internal routing issues between Microsoft 365 and Microsoft Azure services and therefore Exclaimer Cloud was not able to route messages back to 365 correctly. Mitigation: Microsoft resolved their global issue and service was restored. Next Steps: We are working with Microsoft to understand the issue in more detail and if Exclaimer Cloud can be more resilient to these sorts of issues in the future.
Resolved Global: Issues with the Exclaimer Cloud Portal
Summary of Impact: Between 07:30 and 11:15 UTC the Exclaimer Cloud Portal was down, affecting users ability to access templates. All mail flow/imprinting services were unaffected. Root Cause: Our back end data service was experiencing an issue with the continuous replication from primary to secondary sites. Engineers took the portal offline as a precaution to ensure that there was no data loss or corruption. Mitigation: Engineers resolved the issue and brought the portal back online. Next Steps: A project to change the way that our replication works has been ongoing for some time, this work requires a significant redesign of the Portal infrastructure and is nearing the final stages of completion. This should make the Portal more resilient and reliable.
Resolved Global: 3rd Party Issues affecting onboarding
Summary of Impact: Between 09:30 and 11:45 UTC new customers would have been unable to sign up to the Exclaimer Cloud service. Root Cause: Our 3rd party certificate supplier was experiencing issues with certificate issuance. Exclaimer Cloud requires each customer to have a unique certificate to ensure secure email delivery between O365 and Exclaimer Cloud. Mitigation: Engineers at the 3rd party worked to resolve the issue. Next Steps: We will continue to work with our 3rd party to understand the issue in more detail.
Resolved Global: Issues with mail processing delays
Summary of impact: Between 00:00 - 05:30 UTC customers mail flow may have been delayed due to issues with the Exclaimer Cloud service. Root cause: Preliminary investigation showed that our front-door proxy certificate that controls TLS connections between Office 365 and Exclaimer Cloud had expired. Mitigation: Once our engineers had been made aware of the issue, an updated certificate was deployed to all regions between the hours of 04:00 - 05:30 UTC. Next steps: We are currently investigating how this was not picked up by our SSL life-cycle management processes and more importantly, why our automated alert monitoring did not trigger our on-call NOC team at point of expiry.
0 incidents in the last 7 days
1 incidents in the last 30 days
Follow Exclaimer and other services to get alerts about incidents in realtime.Get started now
Ctatus aggregates status page from services so you don't have to. Follow Exclaimer and hundreds of services and be the first to know when something is wrong.Get started now