2022-08-10 - EMEA - AuthPoint Firebox & Azure Integration Authentication Issues
Incident Report for WatchGuard Technologies
Postmortem


Event Summary: The AuthPoint Authentication service experienced a partial service interruption in the European WatchGuard Cloud region between approximately August 10th 06:20 UTC and August 10th 12:17 UTC. This resulted in push notification delivery failures for all authentications using our Firebox integration and 20% of authentications using our Azure integration. Only customers registered to the European WGC region with authentications using the AuthPoint Firebox or Azure integration were impacted. The event is now resolved, and all AuthPoint Authentication services are operating normally for all users.

Event Findings: At approximately 06:20 UTC on August 10th, 2022, two core Authentication components began experiencing communication issues resulting in a slow degradation of service availability for the Firebox and Azure integrations. At 08:52 UTC, our on-call engineers were alerted to a potential issue in the Authentication Services, reporting that Push Notifications were failing in our European WGC region. They found it was affecting all authentications using our Firebox integration, affecting 20% of authentications using our Azure integration, and they engaged subject matter experts from across WatchGuard to assist. At 10:46 UTC initial mitigation actions were performed, and all authentications for Azure users began operating normally. We continued to perform additional actions to resolve issues with Firebox integration authentications, and all operations returned to normal at 12:17 UTC.

We sincerely apologize for the impact on our affected customers, and we know the stability of the WatchGuard Cloud is important to you and your business.

Posted Aug 11, 2022 - 00:15 UTC

Resolved
We are no longer experiencing errors on push notifications for Firebox and this incident is now resolved. The AuthPoint Authentication Service is operating normally at this time.

We'll post a summary in the details of this event within 24hrs. Again, we apologize for any impact this may have had on you or your customers.
Posted Aug 10, 2022 - 12:47 UTC
Monitoring
Our systems show reports of push notification errors are returning to normal and we're monitoring to ensure system stability. We'll post our next update in 30 min, if not sooner.
Posted Aug 10, 2022 - 12:36 UTC
Update
We are continuing to investigate this issue.
Posted Aug 10, 2022 - 10:49 UTC
Update
We are continuing to investigate this issue.
Posted Aug 10, 2022 - 10:47 UTC
Update
We are continuing to investigate this issue.
Posted Aug 10, 2022 - 09:19 UTC
Investigating
We are experiencing that some customers are not receiving push notifications in the EMEA region. We are investigating the issue.
Posted Aug 10, 2022 - 09:19 UTC
This incident affected: AuthPoint Authentication:::EMEA (Logon Agents:::EMEA, Firebox:::EMEA).