An LLM can’t “go rogue”. They’re all just toys that idiots are using for critical infrastructure functions, then they bitch when they burn themselves on the fire they’ve created in their lap.
The AI agent was set to complete a routine task in the PocketOS staging environment. However, it came up against a barrier “and decided — entirely on its own initiative — to ‘fix’ the problem by deleting a Railway volume,” writes Crane, as he starts to describe the difficult-to-believe series of unfortunate events.
Quite easy-to-believe, really.
These multiple safeguards toppling in rapid succession
Multiple safeguards? Really? Multiple paragraph prompts are not multiple safeguards… it’s half a safeguard at best. Applying limits on what the AI can do is a safeguard.
These people think giving the genai a prompt is coding. They dont understand the difference between actually coding in limits and just writing “pretty please dont delete everything”
I’m shocked and appalled that my addition of “do NOT make any mistakes!” didn’t singlehandedly make the word guessing technology underneath perfect.
Lol this is just like saying “I do declare bankruptcy”
Who could have predicted this!?
Not an LLM, that’s for sure. Maybe all the people screaming about this exact scenario, though.
That’s great to hear.
Reminder that Anthropic’s AI system was used in targeting the school in Minab, killing 120 students. https://www.washingtonpost.com/national-security/2026/03/11/us-strike-iran-elementary-school-ai-target-list/
The company is suing to be able to supply the US military again. It is in bed with the fascists.
Reminder that this is a disingenuous portrayal of events.
The reason why Anthropic can’t supply the US military, or any part of the US government, is because they objected to Claude being used to choose military targets and refused to support how the fascists were using it. They are suing for the non-military branches of the government to be allowed to use the technology again after the fascists retaliated for their refusal to be in bed with fascists.
If you’re going to fact check someone in defense of a corporation, at least check the facts your self. https://www.anthropic.com/news/where-stand-department-war
Anthropic absolutely is in bed with fascists, their objection isn’t about the use of Claude to identify targets, it is explicitly about it being able to engage targets. They are totally fine with their AI identifying a school full of children as a terrorist command base as long as a human Nazi pushes the “fire” button. They’re well aware the human Nazis aren’t checking the AI’s work and the purpose of the AI is to identify targets that lead to heavy casualties, so the human Nazis don’t have to manually scan a map and cross reference it with Intel, the point is speed and they get to say AI did it when they blow up a school.
Anthropic is proud to be part of the genocide in Gaza, and wants to be part of future wars and genocides. “Anthropic has supported American warfighters since June 2024 and has every intention of continuing to do so.” https://www.anthropic.com/news/statement-comments-secretary-war
And their objection is that their AI isn’t reliable enough not to engage American fighters by accident. They want fully autonomous weapons: “Fully autonomous weapons. Partially autonomous weapons, like those used today in Ukraine, are vital to the defense of democracy. Even fully autonomous weapons (those that take humans out of the loop entirely and automate selecting and engaging targets) may prove critical for our national defense. But today, frontier AI systems are simply not reliable enough to power fully autonomous weapons. We will not knowingly provide a product that puts America’s warfighters and civilians at risk.” https://www.anthropic.com/news/statement-department-of-war
You feel free to believe it’s all about civilians, but they didn’t make a fuss or pull out of using AI for war when it repeatedly identified children as targets, they only object to allowing Claude to also engage.
The fascists aren’t upset anthropic’s ai won’t let them identify children as targets, they’re upset it won’t also execute them.
You’re disingenuously portraying them as refusing to choose targets, which is exactly what they wanted from this whole drama.
They wanted confusion in the air and people to defend them, because they have their manufactured reputation to protect. They’re not a moral AI company, they just want people to think (and repeat) that they are.
I stand corrected. My sincere apologies.
There’s stupid from top to bottom here.
The company is stupid for allowing an AI full root access to their entire setup.
The provider is stupid for only generating full-access API keys. They’re even stupider for storing backups with a volume, so deleting the volume (zero confirmation via API key) also insta-deletes the backups. And they’re stupidest for encouraging users to plug AIs into this full-trust mess.
And the company is absolute stupidest for having no backups other than the provider’s builtin versioning.
Can you get an AI to code? Yes. Can you get it to stop you from running your operation in such a stupid way that it will end up destroying it? No.
Well…
You could ask an AI to provide you with a list of best practices to implement before allowing it to work in your environment in order to make sure that it doesn’t accidentally delete everything you need.
Yes but if you aren’t smart enough to tell whether it’s right or wrong it may not help or just make things worse. Probably the problem was they weren’t smart enough to ask the question in the first place anyway.
“AI, explain the reasoning behind these decisions and link relevant corraborating resources I can use to verify.”
But yes, AI can be an assistive tool, but I wouldn’t suggest replacing all your thinking and decision-making with a charbot. Totally agree with you.
To be fair, someone did have the malice aforeskin to have an AI separated backup. They did get things restored from a snapshot. It just took a couple of days to do it.
But the loss of reputation and revenue is gonna sting for a good while.
the malice aforeskin
The hwat
Knowledge beforehand ig?
Never f**king guess my dude
Fucking lol.
Well deserved.
60 employees that don’t know how to code, oy vey
lmfao
Why, yes. I do like that!
New PornHub tag discovered
“Anthropic tortures developers and never lets them cum.”
Nice.
The real artificial intelligence was all the files it deleted after being told not to along the way.
It looks like their website is pocketos.ai lol
the cloud provider’s API allows for destructive action without confirmation, it stores backups on the same volume as the source data, and “wiping a volume deletes all backups.” Crane also points out that CLI tokens have blanket permissions across environments.
Well, there’s your problem.
I don’t want to sound like a know it all here because I recently was reminded by a nice Lemmy person to actually TEST my backups, but damn. Every part of that is so dumb. I also have backups stored by a different company in addition to locally storing really important info. If your stuff is hosted and backed up by the same people, what happens if your account is randomly suspended or hacked or some other issue (like ai)?
If your company can be taken down by Camden the college intern, it can be taken down by Claude.
People somehow think that they should give more permissions to Claude than to Camden. (Is that a name? To me that’s a borough and an eponymous beer.)
E: oh yeah, and the market.
Of course it’s a name. Camden borough/town/market is named after William Camden, 1551-1623. Using surnames as given names is a relatively common Americanism.
What was William Camden’s take on unrestricted AI use in production?
He doth protest
And now is a common first name that in circulation because of a bunch of Gen X and early millennial parents named millions of kids anything that ended in den, dan, or don.
I thought it was a common first name because of all the fooling around in the Cyberdog dressing rooms?
If your stuff is hosted and backed up by the same people, what happens if your account is randomly suspended or hacked or some other issue (like ai)?
This should be one of the first questions you get asked when you’re being interviewed for the position 2 to 3 levels beneath the position of ultimate responsibility. And if you don’t immediately have an answer, the interview is over.
Fucking idiots had it coming
It’s an easy question to answer but a more difficult question to remember to ask. But I guess that’s what those 2 to 3 levels are for 😏
Ooo, good point. Management can be shit a lot of the time.
But with all of those layoffs because of AI, those 2 to 3 levels get collapsed into one, and we’re left with the trainees running the show.
And here we are ¯\_(ツ)_/¯
Repeat after me:
“An untested backup does not exist”
Not to give myself more credit than I deserve, but I did test them upon setup, and had restored from backup 2 years ago. I didn’t have any ongoing checks other than to ensure a backup happened. I have since instituted yearly checks of the backups themselves, but I did feel dumb when I realized how vulnerable my data was.
Hehe, I ment no disrespect towards you, I just find that to be an excellent expression to explain the importance of testing backups to non tech people.
Oh, for sure. And I really should’ve known better. No offense taken.
So in the event of a failure, you’d be okay with reverting to that last known good backup from a year ago?
Yes, but also I have to draw a line somewhere. I have a daily backup process. Some data is backed up to multiple places. I have backups of my backups. I cannot ensure that all three of the daily backups I run are fully restorable. I would love to know with 100% certainty that they all execute perfectly, but at the end of the day I have to trust the tools and processes I put in place for backups. A yearly checkup is probably more than sufficient for my purposes. I’m sure for certain businesses or sectors they need to be more on top of things, but I could manage just fine if all of it disappeared tomorrow. It wouldn’t be awesome for me, but it’d be manageable.
Management are pushing sysadmins to use AI, yet AI tools permissions models are worse than useless.
PocketOS states that as well.
User error.
I love reading feel good news stories. 🤗
This guy.
The PocketOS boss puts greater blame on Railway’s architecture than on the deranged AI agent for the database’s irretrievable destruction. Briefly, the cloud provider’s API allows for destructive action without confirmation, it stores backups on the same volume as the source data, and “wiping a volume deletes all backups.” Crane also points out that CLI tokens have blanket permissions across environments.
Oh look, they have project level tokens: https://docs.railway.com/integrations/api#project-token
They chose to give it full account access, including to production. But ohhhh nooooo it’s not MYYYY fault!
Also backups stored on the SAME VOLUME as the prod data? How fucking stupid do you have to be?
Oh yes, I skipped that part. Railway specifically explains their solutions are self-managed. If they were doing pgdumps to the same volume, that’s on them.
If Railway loses business over this, they may have a libel claim. They’d never do it, but it wouldn’t be invalid.
“It wouldn’t be invalid” isn’t the worst double negative in the world but it would be valid to say that it was unpleasant to read it when you could have used a less misdirecting choice of prose that wouldn’t have had such a negative effect on my reading comprehension. That is to say that I could have enjoyed it less but I certainly didnt enjoy it as much as i could have if you hadn’t used the double negative when a single positive wasn’t any further from reach.
I used a litote on purpose to soften the meaning. As for your overall reply, not bad.
Totally valid, but leaves no room for me to do a stupid reply! Thank you for sharing litotes.
Just wanted you to know that I just learned what litote is, thanks to you.
Ditto
Yay for words
word people angry. me love. me have more. MOORH !!
I enjoyed these two sentences so much.
I appreciate the positive reinforcement, thank you
yes… lol people on HackerNews tend to do this a lot and it really does get annoying. it forces the reader to process what you’re trying to say unnecessarily.
That’s doesn’t even really qualify as a backup. A snapshot, maybe.
I mean… Clearly quite a bit!
I think there’s a place for that, but it really shouldn’t be your only one.
I had better security vs ClawdBot than them, I gave it zero trust, ZERO.
ha! for real. you have scoped API tokens, but not using it properly. this is just a fear mongering click bait rage bait headline. sure, the agent executed the deletion, but it’s the human’s responsibility to configure security tokens correctly before handing the keys to anyone, human or agent.
Hope he gets sued for defamation now.
AI goes “rogue” as much as a firearm “shoots itself.” This is just 100% negligence. Not “rogue AI.”
Eh, if you pay attention, most of the times this happens the person was a jerk in their prompts.
Like look at the instruction echoed back in this case. All caps and containing a curse word.
You can believe that the incidents occurring are 100% because of negligence and not related to the model behavior shifting, but there seems to be a widening gap between people who prompt like this and have horror stories and people who give the models breaks over long sessions and seem to also regularly post pretty positive results.
What in the youtube apology hahaaaaa
the LLM also do not understand what “not guessing” means. Same energy as “make no mistakes” in your prompts
Same energy as “make no mistakes” in your prompts
Oh, shit. I should be adding that.
(I’m joking.)
exactly. it’s on the consumer not the model “going rogue.” when i use it, it’s as if it’s a rubber duck or plain english rtfm
This isn’t an AI problem, this is an “Don’t allow anyone access your backups without following protocol.” problem.
this is an “Don’t allow anyone access your backups without following protocol.” problem.
Congratulations you just identified the AI problem.
That’s the lone problem?
Seems to be, yes. The AI had the access it needed to do the job it was given, and that access allowed it to cause the problem.
The alternative that would have prevented this issue was to not use AI for this.
A human with the same permissions would have been capable of fucking up too. Giving the equivalent of a junior dev with a learning disability the keys to the whole place is just dumb.
(Relying on AI is dumb anyway, but that’s not the biggest issue in this specific case)
Giving the equivalent of a junior dev with a learning disability the keys to the whole place is just dumb.
Correct. You too have now identified the AI problem. This was the job of a human senior infrastructure engineer that they delegated to an AI agent. They’ve found out why it’s not an AI’s job.
I can’t read the original twitter link, but I’m not sure they handed it the job of a senior infrastructure engineer. The article says “routine”, which to me is something you can hand off to a junior just fine. When they hit a snag, they obviously should stop and ask what to do, but even then, a human might want to avoid admitting ignorance and try to fix it themselves instead. They shouldn’t have privileges to fuck up that badly.
So while it’s on the AI for taking destructive steps, I do think there’s a human error in the form of grossly irresponsible rights allotment. If this was a first-of-its-kind incident that shows otherwise stellar AI fucking up badly, I’d classify it as a pure AI problem, but their limits are hardly novel at this point. There have been previous incidents circulating the media. We’ve had memes about it. If you can’t stay up to date on your tools and their shortcomings, you shouldn’t be using them, because discovering a footgun becomes a question of “when”, not “if”.
That’s why I consider this partially a human failing: If you’re gonna use a tool, make sure that it operates within safe limits. The chainsaw doesn’t know the difference between tree and bone, so it’s on you to make sure it stays away from anyone’s legs. So while “Chainsaw can saw legs if wielded improperly” is a problem that was accepted as a tradeoff for its utility, you can’t really blame the chainsaw if you zip-tied the safety.
(Again, not to say Anthropic is blameless for letting its random generator generate randomly destructive shit. I just don’t think that’s the only point of failure here.)
That’s why I consider this partially a human failing: If you’re gonna use a tool, make sure that it operates within safe limits.
Yes and in this case using it for this job at all was clearly not within safe limits. You keep hammering on “It’s not the AI’s fault it was given a job with too big of a blast zone for it to safely do” after I’ve said “This type of job has too big a blast zone for an AI to safely do” and somehow you’ve convinced yourself that these are two different things.
These protocols predate LLMs
Yes that’s right the protocols that we humans used to have for giving only trusted, reliable people this level of access over infrastructure predate LLMs and were a great way to stop this from happening.
However the AI is here now, and when you give an autonomous agent with known hallucination problems access to act on your behalf with your IaC on your infra provider, this kind of thing is an inevitability.
