2023-01-25 - Azure Networking. Several issues in Watchguard cloud.
Incident Report for WatchGuard Technologies
Postmortem


Event Summary:
On January 25th between approximately 8:30 UTC and 9:30 UTC, we experienced widespread service interruptions in all regions of our Cloud services. The event is now resolved, and all WatchGuard Cloud services are operating normally for all users in all regions.

For customers this resulted in an inability to log in, administer accounts or operators, configure products, activate products, manage accounts, view dashboards, and generate reports for products through the WatchGuard portal and cloud platform.

Event Findings:
At approximately 8:30  UTC on January 25th, 2023, an alarm was received indicating networking issues within our cloud infrastructure. A team was assembled and upon closer investigation, it was discovered that our partner was experiencing a global WAN outage that affected all of it’s regions. The incident team began exploring all services affected and researching workarounds. At approximately 9:30 UTC, all services began to normalize and our partner posted a notice that a WAN update was being rolled back. After monitoring the situation for a few hours it was decided that WatchGuard would close the incident and call it resolved. A ticket is open with our partner and they are researching the root cause of this outage.

We sincerely apologize for the impact on our affected customers, and we know the stability of the WatchGuard Cloud is important to you and your business.

Posted Jan 25, 2023 - 17:53 UTC

Resolved
We are no longer experiencing problems with our 3rd party infrastructure provider and this incident is now resolved. WatchGuard Cloud Platform and WatchGuard.com are operating normally at this time.

Again, we apologize for any impact this may have had on you or your customers.
Posted Jan 25, 2023 - 11:20 UTC
Update
Our systems show reports of 3rd party service provider connectivity issues are returning to normal and we're monitoring to ensure system stability. We'll post our next update in 30 minutes, if not sooner.
Posted Jan 25, 2023 - 10:22 UTC
Monitoring
Our systems show reports of 3rd party service provider connectivity issues are returning to normal and we're monitoring to ensure system stability. We'll post our next update in 30 minutes, if not sooner.
Posted Jan 25, 2023 - 09:44 UTC
Update
The cause of the outage has been identified. It was triggered by a known networking outage. With our third party cloud vendor.Their status page has not been updated as of yet to however services appear to be coming back online. We are continuing to test and discover if any other services might still be down. We will update again in 30 mins or earlier.
Posted Jan 25, 2023 - 09:30 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Jan 25, 2023 - 09:18 UTC
Update
We are continuing to investigate this issue.
Posted Jan 25, 2023 - 09:07 UTC
Investigating
One of our 3rd party infrastructure providers is experiencing a global networking issue causing several Watchguard cloud services to be degraded, we are working with them to resolve the issue. We will post further updates in the next 30 minutes or earlier.
Posted Jan 25, 2023 - 09:03 UTC
This incident affected: WatchGuard Cloud Platform:::AMER (Web UI Login:::AMER, Account Administration:::AMER, Operator Administration:::AMER, Inventory Administration:::AMER), WatchGuard.com:::CORE (Main Website:::CORE), WatchGuard Cloud Platform:::EMEA (Web UI Login:::EMEA, Account Administration:::EMEA, Operator Administration:::EMEA, Inventory Administration:::EMEA), and WatchGuard Cloud Platform:::APAC (Web UI Login:::APAC, Account Administration:::APAC, Operator Administration:::APAC, Inventory Administration:::APAC).