AI
When the Fix Causes the Outage
How to best increase speed of patching VS teams rushing to just trade a breach for an outage.
Rushing patches out to try and beat AI threats uncovered that teams simply traded a breach for an outage. This article seeks to provide a better path. To sort first, contain the few true emergencies in hours, and ship the real fix in rings.
Over the last 7 days a string of outages impacted American life. Microsoft 365 and Outlook went down. The FAA stopped flights for more than four hours. JPMorgan Chase customers could not reach their accounts, and Zelle payments slowed at other banks. The NSA's classified network failed. Apple services dropped for part of a day.
On November 17, the Washington Sun reported a startling cause. Bank executives and NSA officials had privately told members of Congress that a rush to patch AI found flaws was to blame. One congressional official said the patches were pushed "more than the system was designed for." If that is right, the cure caused the disease or maybe we never fully developed the cure in the first place?
This article tests that claim, then turns it into choices a CIO can make in the next 30 days. Our bias is of course leverging open tools where possible, this doesn't imply building yourself with opensource just making vendor choices based on adhering to industry open standars. We'll discuss where they fit and where the bias to open has limits.
What we know and what we do not
The claim rests on three anonymous sources, and the public record tells a messier story. Microsoft's early report blamed a bug from a July update that exposed itself under extra traffic while some capacity was offline. Officials traced the FAA failure to a router that flooded the network during a copper to fiber upgrade in Arkansas. The FAA administrator said he saw no evidence of anything AI related.
JPMorgan pointed to a software bug in on premise hardware. The NSA cited scheduled maintenance, while the Director of National Intelligence blamed patches for flaws in PHP. Apple blamed an internal update. Implicator has the full timeline. FourWeekMBA counted public causes for seven of nine incidents, and none named AI found flaws.
Rapid7's Christiaan Beek put it best. Systems "fail in boring ways."
Now read that list again from first principles. The process to fix was the root cause: an update, a network upgrade, maintenance, a patch and another update. Whatever the label, almost every outage followed a change someone made on purpose. AI may or may not have triggered these particular changes. It is certainly multiplying the number of changes everyone must make so the lesson holds either way.
To add, regulators saw this coming. In May, Sam Woods, who runs the Bank of England's Prudential Regulation Authority, called patching "the main driver of outages" in the financial system. UK Finance warned that shorter fix windows could bring change fatigue, testing operational strain and outages.
The Flood is Real
AI is finding flaws faster than teams can fix them, and regulators answer with one word. go faster. Microsoft's July and August Patch Tuesday releases carried 570 and 398 fixes, the two largest in the program's history. Microsoft tied the surge to AI assisted discovery.
In October, two critical PHP flaws found with AI help pushed governments in Europe and beyond to take websites offline as a precaution. PHP runs roughly 70 percent of websites, and about half of internet connected PHP software runs older versions with no patch at all.
CISA's new directive gives federal agencies as little as three days for the worst flaws. The UK NCSC tells every organization to prepare to patch quickly, more often and at scale. A BIS paper cites industry guidance that urges fixes outside normal maintenance windows and more acceptance of planned downtime.
That advice is right but it is also incomplete. Speed alone fills your change calendar faster than your teams can empty it.
First Principles. You Manage Exposure, Not Patches
The goal is to remove danger, not to apply patches as fast as they arrive. A flaw is dangerous only when an attacker can reach it and use it. A patch is one way to remove that danger. It is also a changes, especially incorrect ones are what break systems. So every patch trades one risk for another.
That gives you two clocks. The exposure clock starts when a flaw is identified and an attacker could use it. The change risk clock starts when you touch production. Rushing the patch to beat the first clock starts the second one early. The better move is to stop the exposure clock some other way, then run the patch through a calmer process.
Think of an emergency room. Doctors do not treat patients in the order they arrive. They sort by who is in danger now. They stabilize the critical few with a tourniquet and schedule surgery when the team is ready. A rushed operation can kill a patient that a tourniquet would have saved.
CISA's own numbers show how few patients are critical. Its directive asks four questions of every flaw. Is the system exposed to the internet? Can the attack be fully automated? Does it give the attacker control? Is anyone exploiting it now? A flaw that meets all four gets three days and a check for signs of breach. When CISA applied this test to one large agency, only 1 percent of flaws landed in the three day bucket. Sixty percent could wait for the next planned upgrade.
That is the point most coverage missed. The flood is real, but the emergency is small. Treat the 1 percent like a fire and treat the 60 percent like maintenance. A program that treats every flaw as a fire burns out its people and breaks its systems.

Contain In Hours, Fix in Days
Containment is the tourniquet. It closes the exposure without changing the code, and four kinds work fast.
A virtual patch puts a rule in front of the flaw. OWASP describes it as a policy layer that blocks attempts to exploit a known flaw. Picture a guard at the door who checks every visitor for one specific trick while the locksmith replaces the lock. Web application firewalls do this well for web flaws such as injection attacks. They do less for other kinds. Open source tools such as ModSecurity can carry these rules. Vendors such as Palo Alto Networks and F5 now sell AI driven versions. Palo Alto says customers of its service were shielded from the PHP flaws before details went public. Although that is a vendor claim, it carries a higher probability as the capability exists with them.
A switch off removes the feature or module that holds the flaw. A network rule cuts the system off from the internet. And sometimes you take the site down, as European governments did in October. The FBI advised that step for PHP sites that could not be patched at once. Downtime you choose beats downtime an attacker chooses.
Automation makes containment safe at speed. Red Hat describes event driven playbooks, approved in advance, that isolate a host, tighten a security group or revoke a key when a scanner flags a critical exposure. The same system watches health during the change and rolls it back if production suffers. Higher risk steps wait for a person. Red Hat is clear that this buys time and does not replace testing the real fix.
Containment has limits, and the objective view matters here. Sophos's CISO argues that compensating controls cannot keep pace with AI built exploits on old perimeter devices. Firewalls, VPNs and gateways that no longer get updates need replacing, not shielding. A tourniquet saves a limb for hours, not for months.
Ship the Real Fix in Rings
Once the exposure is contained, the patch can move at a safe pace. Bear in mind safe does not mean slow, It means to do so in rings or concentric blast radius.
The model comes from the worst update failure on record. On July 19, 2024, CrowdStrike pushed a faulty content update to every Windows sensor at once. It was live for about 78 minutes, long enough to crash more than eight million machines. CrowdStrike's fix was to stage future updates, starting with a small canary group and widening only when the data looks clean. Microsoft and other vendors already worked that way, in fact this is how most saavy software engineering shops released code as far back as early 2000s (I Know, I did).
So what are rings? A canary group or ring works like the bird miners once carried underground. If the air turns bad, the canary shows it first and the miners still get out. Ring zero is a handful of test systems. Ring one is a small share of real users who can tolerate a hiccup. Each later ring is bigger. A health check gates each step, and a failed check triggers rollback on its own.
Open tools do this well. Argo Rollouts, an open source Kubernetes controller, shifts traffic to a new version in steps and rolls back when error or latency metrics go bad. On OpenShift it pairs with the GitOps tooling many teams already run. For servers, Ansible can patch in batches and stop when a health check fails. On Red Hat Enterprise Linux, live kernel patching fixes select critical and important kernel flaws with no reboot. Think of changing a light bulb without cutting power to the house. It has limits. Each live patch stream lasts six months, so systems still need a full reboot at least twice a year.
Rings need one more rule so keep headroom. Microsoft's early report says its hidden bug woke up under extra traffic while some capacity was offline. The report does not say why that capacity was down, but the lesson stands. Patching takes servers out of service. If you patch too much of a pool at once, or at peak hours, the servers left standing carry the load and hidden faults appear. Patch no more of a pool than you could afford to lose on a bad day.

Read it top down. Each ring is a small bet. A bad patch stops at the ring where it fails, not across the whole estate.
Where AI helps and where it hurts
AI can drain the queue it floods, but it should not write fixes that skip review. Models can read advisories, match them to your software list, find which systems actually run the flawed code and draft test cases. That is real help on the slow part of the job, which is knowing what you run and proving a fix works.
Be careful with AI written fixes. The Cloud Security Alliance cites research showing that AI generated patches miss the flaw or add new defects more than half the time. Google's DORA research found that AI adoption is linked to more instability in software delivery, not less. AI speeds the typing. It does not yet make changes safer. Keep a person and a test suite between any AI written fix and production.
Here our bias toward open models applies. Your list of unpatched flaws is a map of every unlocked window in your house. Sending it to an outside AI service creates a new target. An open model on your own hardware can do the triage and matching while the map stays home. The honest tradeoff is that open models take GPUs and skill to run, and the best closed models still lead on the hardest reasoning. For most triage work, that gap matters less than where the data sits.
Your Options, and Where Open Fits
Every CIO now needs a fast lane for the few real emergencies and a safe lane for everything else. Five routes get there, and each carries risk.
| Route | Speed to start | Outage risk | Who holds your flaw map | Main risk |
|---|---|---|---|---|
| Patch everything as fast as it arrives | Fast | High | You | Change volume breaks systems and burns out staff |
| Vendor virtual patching service | Fast | Low | The vendor | Best for web and network flaws, and you rent the shield |
| Managed patching service | Medium | Medium | The provider | Their test schedule, not yours |
| Open stack you run, with WAF rules, Argo Rollouts, Ansible, live patching and open model triage | Slow | Low once built | You | Needs skills and time you may not have this month |
| Mix, open at the core, buy the shield where skills run thin | Medium | Low | You, shared by choice | More parts and contracts to manage |
Our bias is the mix. Build the triage, the rings and the rollback on open tools you control, because those encode how your systems behave. Buy a virtual patch feed where your team cannot write rules fast enough.
The objective view cuts both ways. A vendor feed may see flaws before they go public, which an open rule set cannot. And an open stack built in a hurry is its own outage risk. With 30 days, do not start by building a new platform. Start with the process, then add tools.
Which Venors and Technologies Can Help
Four common platforms cover clear parts of this plan, and a fifth fits if you already run it. None covers every step, and none replaces the decisions in week one. Most help you sort and watch. Few help you contain, which is the step that buys the most time.

Read it left to right. The products crowd into sorting and watching. The containment column, the one that buys time, is nearly empty.
Wiz sorts cloud flaws the way CISA asks. Wiz describes a mapping to all four CISA questions. It checks new KEV listings against your cloud inventory and tests whether a flaw can be reached from the internet. It pulls the other two signals from CISA's own enrichment data to set a fix window. Wiz Defend adds the check for signs of breach. The limit is scope. The coverage Wiz describes is cloud and cloud connected hybrid systems, so check how far it reaches into your data center before you count on it.
GitLab sorts flaws in the code you build, then measures the result. Its dependency scanning flags flaws on the KEV list and shows EPSS scores. EPSS is a forecast of how likely a flaw is to be exploited in the next 30 days, like a weather forecast for attacks. Static reachability analysis shows whether your code actually imports the flawed package. Feature flags can turn a change on for a percentage of users, which gives you rings for your own applications. And GitLab reports change failure rate as incidents divided by production deployments. The limits are tier and reach. Reachability and KEV flags sit in the Ultimate tier. The failure rate counts only incidents you log in GitLab. None of it sees vendor software you did not build.
Elastic watches the gap and gates the rings. During the PHP scare, Elastic Security Labs published detections built on what a hijacked web server does next. Examples include a PHP process opening a shell or downloading a file. That catches an attack in the window before your fix lands, even with no exploit sample in hand. On the health side, Elastic's burn rate alerts fire when a service uses up its error budget too fast. The error budget is the amount of failure you agreed to tolerate, so a burn rate alert works like a fuel gauge for reliability. That gauge can serve as the check between rings. The limit is that Elastic watches and alerts. It does not push patches or roll them back, so you wire its alerts into the tool that does.
AWS covers the most steps for systems that run on AWS. Amazon Inspector scores each flaw using exploit data and whether the machine can be reached over the network, and it shows EPSS scores. During PHP2Shell, AWS published WAF rules the same day as the patch and added one to its managed PHP rule set. That is the tourniquet, shipped by the vendor. Systems Manager patches fleets with rate controls. You set how many machines patch at once and how many failures stop the run. With a 10 percent error limit across 50 machines, it stops after the fifth failure. For applications, CodeDeploy rolls back on its own when an alarm fires during a rollout.
The AWS limits are real. AWS itself says patching is the only complete fix. Behind a load balancer, AWS WAF reads only the first 8 KB of a request body, so the interim rule can miss an attack or block real uploads. And AWS does not test patches before it offers them in Patch Manager. The testing is still yours.
Databricks fits only if you build on it. Its own security team built VulnWatch on the platform. It joins CVE feeds, EPSS scores and exploit data, then uses AI to match each flaw to the libraries the company runs. The team reports cutting manual triage work by over 95 percent. That is the sorting step, built in house, with the flaw map kept on your own platform. The limits are effort and model choice. This is a build, not a patching product. Its new security product, Lakewatch, launched in private preview in March and names Anthropic's Claude among the models behind it. If open models matter to you, ask how much model choice you get.
Snapshots under XFS give you an open undo button. XFS, the default file system on RHEL, has no built in snapshots. It can pause writes for a clean copy and clone single files cheaply. The system snapshot comes from LVM, the layer below it. Red Hat ships open tools that turn this into a save point, like one in a video game. Snapshot Manager and Boom take one snapshot of / and /var before a patch. You can then boot the old state from the boot menu or revert to it. In the contain step, a snapshot before a switch off lets you undo it fast if it breaks something. In the rings, the snapshot system role does the same across a fleet with Ansible. Image mode goes further. The OS ships as an image, and bootc rollback returns to the last one on the next boot. That is the cleanest toggle, but too big a change for 30 days.
The snapshot limits are real. A revert also rewinds data written after the snapshot, so consider databases and app data on separate volumes. A snapshot on the same disk is not a backup. A thin pool that fills up can take the system down, so monitor its capacity.
Notice what is missing. No product decides who can stop a rollout, and only one shipped the tourniquet on patch day. Those gaps are why the plan starts with decisions, not tools.
A Plan For the Next 30 days
Start with decisions and inventory, then add rings and measurement. Tools come last.
1) Week 1 is about decisions, not tools. Write down who can stop a rollout and who can take a public site offline, and give them that authority before a crisis. GovTech's interviews after the PHP scare made the same point. Adopt CISA's four questions as your triage rule, even if you are not a federal agency. During a patch wave, pause other releases so test capacity goes to fixes.
2) Week 2 is about knowing what you run. Governments that had cataloged their software, including vendor run systems, knew fast whether PHP exposed them. Others learned PHP was hiding inside WordPress. Start with internet facing systems. List what runs there and who owns it.
3) Week 3 is about rings and rollback. Use groups you already have, such as test servers, one region or one office, as your first rings. Then rehearse a rollback for real. An untested rollback is a plan written in pencil. Turn on live kernel patching where your support covers it, and put a virtual patch layer in front of your top public web applications.
4) Week 4 is about measurement. Track how often a security patch causes an incident. DORA calls this the change failure rate. Think of it as how often a mechanic returns your car with a new rattle. Run one tabletop exercise on a flaw like PHP, from triage to rollback. Then fix the slowest step you find.
4 Questions for Your Team This Week
- Of the flaws in our queue today, which meet all four of CISA's tests, and how many is that really?
- Who can stop a bad patch at the first ring, and do they know it?
- When did we last roll back a patch in production on purpose?
- If we had to take our main public site offline tonight, who decides and what do we tell the public?
A team that can answer all four will patch fast without patching blind. A team that cannot will keep trading breaches for outages.
References
- Bank CEOs, NSA Tell Congress That AI Driven Patches Are Behind Outages, Jeff Stein (The Washington Sun)
- US Tech Outages, FAA, Microsoft, NSA, Apple and AI Patching, Implicator.ai
- NSA, Banks Told Congress AI Flaw Patches Drove the Outages, FourWeekMBA
- Bank of England flags financial disruption from advanced AI, Retail Banker International (reporting Reuters)
- Frontier AI implications for financial stability, UK Finance
- The Widening Gap, AI Discovery Outpaces Patch Capacity, Cloud Security Alliance AI Safety Initiative
- For Government, PHP Scare Points to an AI Fueled New Normal, Jule Pattison-Gordon (GovTech)
- CISA directive revamps how agencies prioritize vulnerable systems, Nextgov
- CISA vulnerability remediation directive BOD 26-04, CyberScoop
- Preparing for a vulnerability patch wave, Ollie Whitehouse (UK NCSC)
- Banks Have Minutes, Not Weeks, to Fix Flaws as AI Speeds Up Attacks, Jason Nelson (Decrypt, reporting the BIS)
- Virtual Patching Cheat Sheet, OWASP Cheat Sheet Series
- Critical PHP Vulnerabilities, Frontier Virtual Patching Customers Were Protected Before Day One, Palo Alto Networks
- The vulnerability flood is here, Ross McKerchar (Sophos)
- Managing IT operations when AI outpaces your patching cycle, Richard Henshall (Red Hat)
- CrowdStrike blames outage on content configuration update, Computer Weekly
- CrowdStrike promises more testing of security content updates, iTnews
- CrowdStrike Says Code Testing Bugs Failed to Prevent Outage, Bank Info Security
- Argo Rollouts documentation, Argo Project
- Is live kernel patch (kpatch) supported in Red Hat Enterprise Linux, Red Hat Customer Portal
- DORA 2025, Measuring Software Delivery After AI, Rachel Stephens (RedMonk)
- Accelerating CISA BOD 26-04 Vulnerability and Triage Activities through Wiz, Aviram Ben Kalifa and Bryan Rosensteel (Wiz)
- Vulnerability risk prioritization made simple with GitLab, GitLab
- Reduce supply chain risk with smarter vulnerability prioritization, Salman Ladha (GitLab)
- Static reachability analysis, GitLab Docs
- Feature flags, GitLab Docs
- DORA metrics in GitLab, GitLab Docs
- php2shell, Assessing Elastic detection coverage, Ruben Groenewoud and Bryan Porras Blanch (Elastic Security Labs)
- Create an SLO burn rate rule, Elastic Docs
- Viewing the Amazon Inspector score, AWS Docs
- PHP web runtimes, Security Bulletin 2026-143-AWS, AWS
- Comparing the AWS WAF Managed Rule PHP2Shell_BODY with a Custom Rule, Classmethod
- PHP2Shell on AWS, Start on October 17, and Mind the 8 KB WAF Gap, FactualMinds
- Control automations at scale, AWS Systems Manager Docs
- AWS Systems Manager Patch Manager, AWS Docs
- Automate rollbacks for Amazon ECS rolling deployments with CloudWatch alarms, AWS Containers Blog
- VulnWatch, AI Enhanced Prioritization of Vulnerabilities, Anirudh Kondaveeti (Databricks)
- Databricks Enters Security Market with Launch of Lakewatch, Databricks
- xfs_freeze(8), Linux manual page, xfsprogs project
- Managing system upgrades with snapshots, Red Hat Docs
- snapshot system role, Linux System Roles
- Managing RHEL bootc images, Red Hat Docs
© 2026 Symvoulos LLP. You may quote or excerpt this article with attribution and link to the original URL as well as referenced material. Republication in full, adaptation, translation, or use as the basis of derivative or secondary works requires attribution. Removal or alteration of attribution is prohibited.
Stathy Touloumis is President of Symvoulos (Symv), a boutique technology consultancy serving all sectors of business and government entities including regulated enterprises. His experience spans over 30 years with most recent to state, local, and public entities across infrastructure modernization, security, service delivery strategy, and the governance questions raised by AI adoption. Symvoulos’ expertise starts where most stall: multi-year cost models, risk vectors and staffing plans an agency can defend in a budget hearing. Be sure to follow us at symv.ai.
Want to talk about your own case?
Share a few details. A person on our team will reply.
Start a conversation