This reads like bullshit. “Pushing too many patches”? What is that supposed to mean? It’s not like they are sending each patch separately. From what we know FAA outage was because they deployed a new tool, not because of vulnerabilities in the old one. Outlook outage was supposedly caused by an existing bug, not by a new fix. That leaves us with banks and NSA and the article says “But even entities with large resources, such as the nation’s top banks and the intelligence community” which again is bullshit. Everyone knows banks don’t like spending their money on IT. There might be something to this story. It’s possible that NSA and banking system outage were caused by AI but definitely not in the way this article is describing.
“Pushing too many patches”? What is that supposed to mean? It’s not like they are sending each patch separately.
I’ll have to throw in a “yeah, but.” Nobody in their right mind would bundle (say) 50 patches together into a single upgrade. The chances of regression are too great, and if it happens, it makes it harder to isolate which patch caused the problem. You can mitigate that by proper testing, but in complex systems, that takes significant time and resources, and there’s countervailing time pressure to immediately deal with zero-days.
But they were also talking about scheduled maintenance, which might suggest that they still make provision to shut the system down to apply changes, rather than having spent the money and technical effort to allow them to maintain system operations when applying changes (for example, by blue/green deployments). At very least, it’s good practice to minimize downtime even when you can’t maintain operational continuity, by applying changes in the smallest possible increments.
Minimizing downtime might matter less in banks, where back-end systems often still run in batch mode, due to the banks’ reluctance to invest properly in IT (as you mentioned) and because some of the external systems they have to interface with also run in batch (e.g., overnight clearing, which is instantaneous in most of the rest of the developed world).
Tech debt from the mainframe era is the world’s biggest submerged iceberg.
The NSA outage sounded like it was a failed comms upgrade, which is probably for different causes.
Yes, but the exact quote is “They’re trying to push through a lot of patches all at once — more than the system was designed for.”
The process can have bottlenecks and can not handle enough patches properly (as you describe, issues with testing, regressions, downtimes etc.). The system doesn’t handle patches like that. It doesn’t process each patch separately. It’s not like every dev or AI agent send a patch at once and the system fails to process them.
This reads like bullshit. “Pushing too many patches”? What is that supposed to mean? It’s not like they are sending each patch separately. From what we know FAA outage was because they deployed a new tool, not because of vulnerabilities in the old one. Outlook outage was supposedly caused by an existing bug, not by a new fix. That leaves us with banks and NSA and the article says “But even entities with large resources, such as the nation’s top banks and the intelligence community” which again is bullshit. Everyone knows banks don’t like spending their money on IT. There might be something to this story. It’s possible that NSA and banking system outage were caused by AI but definitely not in the way this article is describing.
It would be just like a big bank to try to turn a worldwide hacking crisis into a pitch for AI.
I’ll have to throw in a “yeah, but.” Nobody in their right mind would bundle (say) 50 patches together into a single upgrade. The chances of regression are too great, and if it happens, it makes it harder to isolate which patch caused the problem. You can mitigate that by proper testing, but in complex systems, that takes significant time and resources, and there’s countervailing time pressure to immediately deal with zero-days.
But they were also talking about scheduled maintenance, which might suggest that they still make provision to shut the system down to apply changes, rather than having spent the money and technical effort to allow them to maintain system operations when applying changes (for example, by blue/green deployments). At very least, it’s good practice to minimize downtime even when you can’t maintain operational continuity, by applying changes in the smallest possible increments.
Minimizing downtime might matter less in banks, where back-end systems often still run in batch mode, due to the banks’ reluctance to invest properly in IT (as you mentioned) and because some of the external systems they have to interface with also run in batch (e.g., overnight clearing, which is instantaneous in most of the rest of the developed world).
Tech debt from the mainframe era is the world’s biggest submerged iceberg.
The NSA outage sounded like it was a failed comms upgrade, which is probably for different causes.
Yes, but the exact quote is “They’re trying to push through a lot of patches all at once — more than the system was designed for.”
The process can have bottlenecks and can not handle enough patches properly (as you describe, issues with testing, regressions, downtimes etc.). The system doesn’t handle patches like that. It doesn’t process each patch separately. It’s not like every dev or AI agent send a patch at once and the system fails to process them.
Bank CEOs are well known for understanding how coding works.