2023-03-20 AMER WGC Login Issues
Incident Report for WatchGuard Technologies
Postmortem


Event Summary:
On March 20th around 18:20 UTC, we experienced service interruptions in Americas (AMER) region of the WatchGuard Cloud Login (WGC). This event, which lasted 20 minutes, is now resolved and the WG Cloud Login service is operating normally.

For customers registered to the AMER WGC region, this resulted in an inability to log into the Web UI. However, customers who were already logged into the AMER WGC region were able to access all cloud-based services, such as AuthPoint or Administration services, which continued to operate normally.

 

Event Findings:
At approximately 18:20 UTC on March 20th, 2023, we experienced increased timeouts during WatchGuard Cloud (WGC) login attempts in the AMER region. By 18:33 UTC we were able to isolate the cause as an internal component of the WGC Login service triggering some login requests to time out. At 18:40 UTC, we identified a solution to mitigate the problem but our self-healing architecture restored the WGC Login service to normal operation before the mitigation was deployed. After our systems fully recovered, we proceed to deploy our identified fix in order to prevent future occurrences. We are also investigating the root cause that triggered this event and delayed our systems from self-healing faster. Throughout this event, customer logins in the European (EMEA) or Asia-Pacific (APAC) WGC regions were unaffected, customers already logged in to the AMER region were able to access all cloud-based services without interruptions, and all other services, such as AuthPoint Authentications, continued to operate normally.

 

We sincerely apologize for the impact on our affected customers, and we know the stability of the WatchGuard Cloud is important to you and your business.

Posted Mar 21, 2023 - 21:24 UTC

Resolved
Our systems show WGC Web UI Logins have returning to normal. This issue is now resolved. We apologies for any impact this event may have caused you.
Posted Mar 20, 2023 - 19:17 UTC
Update
We have identified the issue and mitigated impact. Our systems show WGC Web UI Logins are returning to normal and we're monitoring to ensure system stability. We'll post our next update in 30min, if not sooner.
Posted Mar 20, 2023 - 18:54 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Mar 20, 2023 - 18:54 UTC
Update
We are continuing to investigate this issue.
Posted Mar 20, 2023 - 18:53 UTC
Update
We are continuing to investigate this issue.
Posted Mar 20, 2023 - 18:36 UTC
Investigating
We are currently experiencing issues with customers logging into the America's region of our WatchGuard cloud management UI. We apologize for the inconvenience and are currently researching the issue. A team has assembled and an updated will be posted as soon as we have more information.
Posted Mar 20, 2023 - 18:32 UTC
This incident affected: WatchGuard Cloud Platform:::AMER (Web UI Login:::AMER, Account Administration:::AMER, Operator Administration:::AMER, Inventory Administration:::AMER).