Initial Event Summary: The WatchGuard DNSWatch DNS Resolution service experienced a service disruption in the Americas and Europe regions between approximately March 18th 09:55 UTC and March 18th 14:49 UTC. This resulted in the DNSWatch DNS Resolution service failing to process DNS requests. The event is now resolved, and all DNSWatch services are now operating normally for all users.
Initial Event Findings: At approximately 09:55 UTC on March 18th, 2024, our on-call engineers were alerted to a potential issue in the EMEA Gateway service and found that the resolvers in our Americas region were affected as well. During our investigation, high CPU usage was observed on several backend servers resulting in an inability to process DNS requests. The high CPU load was related to a recently updated service component. Our teams worked to mitigate the impact caused and immediately began to rollback the changes, and at approximately 13:40 UTC, the Europe DNS Resolution service was restored, and at 14:50 UTC, the Americas DNS Resolution service was restored.
We're working on completing an in-depth analysis of this event.
We sincerely apologize for the impact on our affected customers, and we know the stability of the DNSWatch is important to you and your business. At WatchGuard, we will never be satisfied with anything less than perfect operational performance and will continue to do everything we can to drive improvements across our services.