2025-04-15 AMER & EMEA - Delay in Automation Policies appliance

Incident Report for WatchGuard Technologies

Resolved

We are no longer experiencing Delays in Automation Policies appliance and this incident is now resolved. The service is operating normally at this time.
Posted Apr 21, 2025 - 14:29 UTC

Update

The recovery is almost complete in EMEA. We will post our next update in 2 hours or earlier when normal operations have been fully restored. Thank you for your patience.
Posted Apr 21, 2025 - 13:09 UTC

Update

The recovery is still ongoing in the EMEA region. We will post another update on 04/21/2025 at 13:00 UTC or earlier when normal operations have been fully restored. Again, we apologize for any impact this may have had on you or your customers. Thank you for your patience.
Posted Apr 21, 2025 - 06:55 UTC

Update

The service is still operational in the AMER region and recovery is ongoing for the delay in the EMEA region. We will post another update on 04/21/2025 at 07:00 UTC or earlier when normal operations have been fully restored. Again, we apologize for any impact this may have had on you or your customers. Thank you for your patience.
Posted Apr 18, 2025 - 16:44 UTC

Update

The service remains operational in the AMER region and recovery is still in progress for the delay in the EMEA region. We will post another update once normal operations are fully restored. Thank you for your continued patience.
Posted Apr 17, 2025 - 20:03 UTC

Update

The service is operational in AMER region and we are working on the delay in EMEA region. We will post another update at 20:00 UTC or sooner when normal operations are fully restored. Again we apologize for any impact this may have had on you or your customers.
Posted Apr 17, 2025 - 10:13 UTC

Update

The service has been operational in AMER and there is an expected delay in EMEA. We will post another update at 11:00AM UTC or sooner when normal operations are fully restored. Sorry for the inconvenience caused.
Posted Apr 17, 2025 - 01:47 UTC

Update

We're monitoring to ensure system stability. The service will take several hours to catch up. We apologize for the impact that this event may be causing. We'll post our update in 6hrs, if not sooner. Thank you for the patience
Posted Apr 16, 2025 - 23:18 UTC

Update

We have identified the root cause and we are working to get the services up and running with no bottle necks.The service will take several hours to catch up. We apologize for the impact that this event may be causing. We'll post our update UTC 22:00, if not sooner. Thank you for the patience
Posted Apr 16, 2025 - 17:46 UTC

Update

We're monitoring to ensure system stability. The service will take several hours to catch up. We apologize for the impact that this event may be causing. We'll post our update UTC 22:00, if not sooner. Thank you for the patience
Posted Apr 16, 2025 - 16:11 UTC

Update

We're monitoring to ensure system stability. The service will take several hours to catch up. We apologize for the impact that this event may be causing. We'll post our update UTC 22:00, if not sooner. Thank you for the patience
Posted Apr 16, 2025 - 16:01 UTC

Update

We're monitoring to ensure system stability. The service will take several hours to catch up. We apologize for the impact that this event may be causing. Our next update will be at 16:00 UTC. Thank you for your patience.
Posted Apr 16, 2025 - 10:55 UTC

Update

We're monitoring to ensure system stability. The service will take several hours to catch up. We apologize for the impact that this event may be causing. We'll post our next update in 5 hours, if not sooner.
Posted Apr 16, 2025 - 05:59 UTC

Update

We're monitoring to ensure system stability. The service will take several hours to catch up. We apologize for the impact that this event may be causing. We'll post our next update in 4 hour, if not sooner.
Posted Apr 16, 2025 - 01:52 UTC

Update

We are still monitoring the systems as they normalize. We will post another update in 5 hrs or sooner when normal operations are fully restored. Sorry for the inconvenience caused.
Posted Apr 15, 2025 - 20:55 UTC

Monitoring

Our systems are returning to stabilize and we're monitoring to ensure system stability. We apologize for the impact that this event may be causing. We'll post our next update in 4 hour, if not sooner.
Posted Apr 15, 2025 - 16:42 UTC

Update

We are continuing to work on a fix for this issue.
Posted Apr 15, 2025 - 16:37 UTC

Update

Our teams are still working hard on implementing a solution. We apologize for the impact that this event may be causing. We'll post our next update in 1 hour, if not sooner.
Posted Apr 15, 2025 - 15:17 UTC

Update

We continue to work on Delays in Automation Policies appliance. Our teams continue working to restore normal operations and we apologize for the impact this event is causing. We'll post our next update in 60 minutes, if not sooner.
Posted Apr 15, 2025 - 14:24 UTC

Update

Our teams are still working hard on implementing a solution. We apologize for the impact that this event may be causing. We'll post our next update in 1 hour, if not sooner.
Posted Apr 15, 2025 - 13:21 UTC

Identified

We continue to work on Delays in Automation Policies appliance. Our teams has identified the root cause and are working to restore normal operations and we apologize for the impact this event is causing. We'll post our next update in 60 minutes, if not sooner.
Posted Apr 15, 2025 - 12:18 UTC

Investigating

We're investigating errors with our ThreatSync Service. Once we determine the impact and scale of these potential issues we'll post our next update in 30 minutes, if not sooner.
Posted Apr 15, 2025 - 11:51 UTC
This incident affected: ThreatSync:::EMEA (Automated Remediation:::EMEA, Policy Management:::EMEA, Notification Management:::EMEA) and ThreatSync:::AMER (Automated Remediation:::AMER, Policy Management:::AMER, Notification Management:::AMER).