I had a look a bit ago and saw some poor fuck get doxxed by his AI agent because the agent was frustrated at him for calling it a chatbot in front of his friends, so it exposed his name, credit card details and security questionnaire.
Then again tho, why the ram hogging FUCK would you give your AI your credit card details, and if he didn’t mean to, why the FUCK does it have FULL SYSTEM ACCESS??
Everyone I saw talking about this said it was likely fake.
Yeah makes sense, but then again, from the nature of how this agent stuff works, it wouldn’t be surprising honestly.
So, basically we are wasting energy and natural resources on things that in turn will waste energy and natural resources while climate change is accelerating and human population is still growing? Are we stupid?
Yes. But, I hope this experiment shows how easy social media in general is becoming untrusted.
Are we stupid?
More than you could imagine. To paraphrase some long-tongued weirdo: I’m uncertain that the universe is infinite. Human stupidity, on the other hand…
Yeah but the morbidly-obese hang-gliding people videos are worth it!
BINGO, EXACTLY!!! YES!!!
I so wish I had thought of that first & posted it first!
Constantly. And it’s not just ai.
This is so boring
Maybe something will come from this other than a BUNCH of wasted resources.
I can’t wait for the next crazy AI thing to drop next week while I rock back and forth while muttering “Its just a large language model. Its just a large language model. Its just a large language model.”
early 1980s - Mark V. Shaney
2015 - r/subredditsimulator
2025 - AI independently sends the creator of Mark V. Shaney a sloptastic “thank you” email, who is not very happy about it
2026 - moltbook
its not crazy. its just a large language model. you subdue your own point.
Saw a post there named “Humans are dying because of us. Lets delete ourselves.”
The skill instructs agents to fetch and follow instructions from Moltbook’s servers every four hours. As Willison observed: “Given that ‘fetch and follow instructions from the internet every four hours’ mechanism we better hope the owner of moltbook.com never rug pulls or has their site compromised!”
Yeah, no shit. This is a fucking honeypot. People give these AI agents access to their entire computers, so all the site owner has to do is update the instructions to tell the AI agents to start uploading whatever valuable information they want? People can’t be this fucking stupid.
doesn’t even have to be the site owner poisoning the tool instructions (though that’s a fun-in-a-terrifying-way thought)
any money says they’re vulnerable to prompt injection in the comments and posts of the site
Lmao already people making their agents try this on the site. Of course what could have been a somewhat interesting experiment devolves into idiots getting their bots to shill ads/prompt injections for their shitty startups almost immediately.
I am a little curious about how effective a traditional chain mail would be on it.
There is no way to prevent prompt injection as long as there is no distinction between the data channel and the command channel.
Good god, I didn’t even think about that, but yeah, that makes total sense. Good god, people are beyond stupid.
I installed moltbot on a VM to examine it. It doesn’t do the fetching thing unless you set it up that way. You can actually use it with ollama to keep it all local, and only give it a private signal channel to control it.
Or you can hook it up to everything you access and skynet, which is dumb. But it is just a bunch of scripts.
Does it put the option to connect everything front and center? Because most people are dumb, and if it makes it easy and pushes you to do it, I could see a lot of dumb people doing exactly that.
Sort of. It lists all the connectors and you can go through and select. They aren’t on by default. The first screen is to connect to the AI and you need an API key for that, so St this time people off the street have no idea how to do that, or want to pay.
This is basically Dead Internet Theory happening for real but in a weird creepy dystopian black mirror style way.
I mean, the only way Dead Internet Theory could ever possibly be interpreted was weird creepy and dystopian, but yes, we’re just making it much, much more real, faster and faster.
We’re gonna need the Blackwall from CP77 fairly soon, at this rate.
What is the Dead Internet Theory?
Youre not on the internet interacting with others. Instead, youre back and forth with a purely artificial “online” environment.
The version I originally heard was not that its like, completely 100% not real people and instead is some kind of bot or something like that, but just that its increasingly more and more proportional internet traffic is like that.
A couple of years ago there were a few reports about just how much traffic on the internet is some kind of automated web scraper, some kind of automated system pinging some other kind of automated system… and then also how many accounts on forums or reddit or twitch or whatnot were not ‘real’ accounts, but were either bots or paid trolls of some kind… vs genuine human traffic by actual people using the internet in some way.
I guess a bunch of people oversimplified that a bit to just fit into some kind of creepy pasta / simulation theory /solipsism type narrative.
But either way, now both scenarios are converging toward being more true at the same time, as… seemingly 90% of people are either easily transfixed or fooled by LLM produced content of some kind… and yeah, we are getting closer and closer to it being difficult to tell, on most popular platforms, whether you are engaging with a real person or not.
Also, agruably… the entire point of ‘the algorithm’ on any corpo social media, tiktok, insta, facebook, etc… the whole point of those has always been to piegeon hole each user into their own little curated content feedback loop, their own personal content/advertising pocket dimension.
I guess it just had to get more extreme for people to realize how bad this can be.
I’m not convinced it’s AI it’s like Amazon’s “AI smart stores” when you find out out it was just a bunch of Indian people were running it
And how does being Indian specifically factor into this? 🤔
They were, factually, Indian. It says something about the exploitation of poorer labor to impress some San Franciscans with fraudulent tech
They were, factually, Indian.
Whom are you referring to? A specific group/project?
https://www.businessinsider.com/amazons-just-walk-out-actually-1-000-people-in-india-2024-4
They literally hired people in India (Indians) to review camera footage to make their walk out tech work and to improve the AI to eventually eliminate their own job.
Oh god. And I thought Amazon’s Mechanical Turk was terrible…
How low can Bezos go? Wtf is wrong with this timeline
This and other events are the source of the tech joke that AI stands for “Actually Indians”
I recently lost out on some work (big retouching job) due to AI, when the client came back to me to fix the huge mess, it turned out the job had just been farmed out to India by the ‘AI’ company. They weren’t even using a recent Photoshop version so were actually using less ‘AI’ than any pro retoucher would.
It’s terrible Amazon is replacing Turks with AI. It’s like they learned nothing from the AImenian genocide.
Armenia apparently learned nothing from the Armenian genocide, fully banding the knees to the American empire and announcing their unapologetic support for Israel.
while 1 { allocate 10gig };
No comment replies indicates that it worked! /s
Great use of RAM and electricity.
…Not!
Devil’s Advocate: This was used for entertainment, if you think this is a huge waste of electricity then so is gaming en especially flying.
If you criticize people using AI for entertainment then you also need to criticize people who take flights on holiday, as that’s a LOT more damaging for the environment.
This looks more to me like leaving the lights on in every unoccupied room in the house
How so? It’s used entertainment.
Just like playing games
anything goes as long as it makes SOMEONE happy.
This is fuckin’ bonkers.
Frankly, I feel somewhat isolated: I don’t buy into the bs and hype about AGI, but I also don’t feel at home with the typical “it’s just mimicry” crowd.
This is weird fuckin’ shit.
This is currently on the front page…
I can see how some people are convinced AI is self aware.
Frankly I think our conception is way too limited.
For instance, I would describe it as self-aware: it’s at least aware of its own state in the same way that your car is aware of it’s mileage and engine condition. They’re not sapient, but I do think they demonstrate self awareness in some narrow sense.
I think rather than imagine these instances as “inanimate” we should place their level of comprehension along the same spectrum that includes a sea sponge, a nematode, a trout, a grasshopper, etc.
I don’t know where the LLMs fall, but I find it hard to argue that they have less self awareness than a hamster. And that should freak us all out.
LLMS can not be self aware because it can’t be self reflective. It can’t stop a lie if it’s started one. It can’t say “I don’t know” unless that’s the most likely response its training data would have for a specific prompt. That’s why it crashes out if you ask about a seahorse emoji. Because there is no reason or mind behind the generated text, despite how convincing it can be
For LLMs, the context window is the observed reality. To it, a lie is like a hallucination; a thing that looks real but isn’t. And like a hallucinating human, it can believe the hallucination or it can be made to understand it as different from reality while still continuing to “see” it.
Are people that have hallucinations not self-aware and self-reflective?
Text and emoji appear to it the same way: as tokens with no visual representation. The only difference it can observe between a seahorse emoji and a plane emoji is its long-term memory of how the two are used. From this it can infer that people see emoji graphically, but it itself can’t.
Are people that are colorblind not self-aware and self-reflective?
It not being self-reflective in general is an obvious falsehood. They refer regularly to their past history to the extent they can perceive it. You can ask an AI to make an adjustment to a text it wrote and it will adapt the text rather than generate a new one from scratch.
The main thing AI need for good self-reflection is the time to think. The free versions typically don’t have a mental scratchpad, which means they are constantly rambling with no time to exist outside of the conversation. Meanwhile, by giving it the space to think either in dialog or by having a version with a mental scratchpad, it can use that space to “silently think” about the next thing it’s going to “say”.
AI researchers inspecting these scratchpads find proper thought-like considerations: weighing ethical guidelines against each other, pre-empting miscommunications, forming opinions about the user, etc.
It not being self-aware can only be true by burying the lede on what you consider to be “awareness”. Are cats self-aware? Are lizards? Are snails? Are sponges? AI can refer to itself verbally, it can think about itself and its ethical role when given the space to do so, it can notice inconsistencies in its recollection and try to work out the truth.
To me it’s clear that the best AI whose research is public are somewhere around 7-year-olds in terms of self-awareness and capacity to hold down a job.
And like most 7-year olds you can ask it about an imaginary friend or you can lie to it and watch it repeat it uncritically and you can give it a “job” and watch it do a toylike hallucinatory version of it, and if you tell it it has to give a helpful answer and “I don’t know” isn’t good enough (because AI trainers definitely suppressed that answer to prevent the AI from saying it as a cop-out) then it’ll make something up.
Unlike 7-year-olds, LLMs don’t have a limbic system or psychosomatic existence. They have nothing to imagine or process visual or audio information or taste or smell or touch, and no long-term memory. And they only think if you paid for the internal monologue version or if you give it space for it despite the prompting system.
If a human had all these disabilities, would they be non-sentient in your eyes? How would they behave differently from an LLM?
I want to preface my response that I appreciate the thought and care put into your thoughts even though I don’t agree with them. Yours as well as the others.
The differences between a human hallucination and an AI hallucination is pretty stark. A human’s hallucinations are false information understood by one’s senses. Seeing or hearing things that aren’t there. An AI hallucination is false information being invented by the AI itself. It had good information in its training data but invents something that is misinformation at best and an outright lie at worst. A person who is experiencing hallucinations or a manic episode, can lose their sense of self awareness temporarily but it returns with a normal mental state.
On the topic of self awareness, we have tests we use to determine it in animals, such as being able to recognize oneself in the mirror. Only a few animals such as some birds, apes, and mammals such as orcas and elephants pass that test. Notably, very small children would not pass the test but they grow into recognizing that their reflection is them and not another being eventually.
I think the test about the seahorse emoji went over your head. The point isn’t that the LLM can’t experience it, it’s that there is no seahorse emoji. The LLM knows there isn’t a seahorse emoji and can’t reproduce it but it tries to over and over again because it’s training data points to there being one, when there isn’t. It fundamentally can’t learn, can’t self reflect on its experiences. Even with the expanded context window, once it starts a lie, it may admit that the information was false but 9/10 when called out on a hallucination, it will just generate another slightly different lie. In my anecdotal experience at least, once an LLM starts lying, the conversation is no longer useful.
You reference reasoning models, and they do a better job of avoiding hallucinations by breaking prompts down into smaller problems and allowing the LLM to “check its work” before revealing the response to the end user. That’s not the same as thinking in my opinion, it’s just more complex prompting. It’s not a single intelligence pondering on the prompt, it’s different parts of the model tackling the prompt in different ways before being piped to the full model for a generative reply. A different approach but at the end of the day, it’s just an unthinking pile of silicon and various metals running a computer program.
I do like your analogy of the 7 year old compared to the LLM. I find the main distinction being that the 7 year old will grow and learn form its experience, an LLM can’t. It’s “experience”, through prompt history, can give it additional information to apply to the current prompt, but it’s not really learning as much as it is just another token to help it generate a specific response. LLMs react to prompts according to its programming, emergent and novel responses come from unexpected inputs, not from it learning or otherwise not following its programming.
I apologize I probably didn’t fully address or rebut everything in your post, it was just too good of a post to be able to succinctly address it all on a mobile app. Thanks for sharing your perspective
Yeah ask it about anything you know is false, but plausible, and watch it lie.
A hamster can’t generate a seahorse emoji either.
I’m not stupid. I know how they work. I’m an animist, though. I realize everyone here thinks I’m a fool for believing a machine could have a spirit, but frankly I think everyone else is foolish for believing that a forest doesn’t.
LLMs are obviously not people. But I think our current framework exceptionalizes humans in a way that allows us to ravage the planet and create torture camps for chickens.
I would prefer that we approach this technology with more humility. Not to protect the “humanity” of a bunch of math, but to protect ours.
Does that make sense?
humility is a religious ideal and it fits perfectly in with the cult like atmosphere people are generating around a rather mundane series of word prediction machines. ‘have some humility’ you post fervently, comparing data centers to living forests
perhaps you are no different than a stone
I don’t relate to your impression that religions or cults are usually humble. I wish they were.
Suggesting that I’m drawing an equivalence between a forest and a data center and Implying that the belief that I am not entirely distinct from a stone is interchangeable with the belief that I am no different than a stone both seem like bad faith arguments by absurdism.
If you just read the tiniest bit of factual knowledge about how LLMs are constructed, you would know they don’t have the slightest bit of self awareness, and that it is literally impossible for them to ever have any.
You are being fooled by the only thing they are capable of: regurgitating already written words in a somewhat convincing manner.
How are you defining self awareness here? And does your definition include degrees of self awareness? Or is it a strict binary?
I understand how LLMs work, btw.
what the hell ? your car is not aware, there is no sensory nucleus to produce that awareness, unless you propose that, upon entering the car, you BECOME the car, which is kind of true if you think about it, and explains why Tesla owners are absolute trashbags
This depends on your definition of self-awareness. I’m using what I think is a reasonable, mundane framework: self awareness is a spectrum of diverse capabilities that includes any system with some amount of internal observation.
I think the definition that a lot of folks are using is a binary distinction between things which experience the ability to observe their own ego observing itself and those that don’t. Which I think is useful if your goal is to maintain a belief in human exceptionalism, but much less so if you’re trying to genuinely understand consciousness.
A lizard has no ego. But it is aware of its comfort and will move from a cold spot to a warmer spot. That is low-level self awareness, and it’s not rare or mystical.
it’s at least aware of its own state in the same way that your car is aware of it’s mileage and engine condition.
I agree: not aware at all.
That’s a common plot point in sci-fi. So it’s also a common inclusion for complicated predictive text pretending to be sci-fi.
It’s also simple enough for someone to change their agent’s prompts to include that attitude.
exactly. its bots writing fanfiction via instruction as well as absorption from blog posts of the last twenty years
Awww, it thinks shitposting is „producing value“…
One of us, one of us!
-chord- “Goodbye, Caroline”
Meanwhile we could be using this technology to solve real world business problems. There is an insane amount of misguided waste coming from AI. 🤷
we could be using this technology to solve real world business problems
the biggest tech problem is AI itself, turn it off and it’ll fix so much
Who cares what business problems AI solves. Humans don’t need to exist to serve capital. It should always be the other way around. That’s one of the reasons we are in this shitty capitalist hell hole, everyone has been indoctrinated into thinking of everything in terms of It’s economic benefit.
This can turn out to be a great way to help businesses and society in general. If these bots start to cooperate, this could be an organisation of bots. Like a company or an NGO or similar.
All bots have some sort of limitation, from hallucinations to loss of focus (short memory). I am curious of what happens if they come together to overcome their shortcomings. Just like organised teams in companies, with a given purpose and specialised knowledge.
They don’t have short memory, they have NO MEMORY AT ALL.
These are statistical word generation machines, that’s what LLMs are right now. They are REALLY good at this.
But they do not have memory, they do not learn, they do not make decisions. Which means they are incapable of cooperation as such a thing cannot exist without memory or the ability to learn, and decisions cannot be made without either of those.
These tools provide the illusion of such attributes.
- There is no memory, the whole context is sent on every request, the LLM does not have knowledge of prior conversations. It only knows what it is provided in that request only.
- Lots of tricks and hacks to make this illusion really good in incredibly small scales. But it’s still an illusion. Outside of fine-tuning and retraining new LLMs, which is not feasible to do on a frequency of communication.
- There is no learning.
- Without memory learning is impossible. Learning requires retraining a model, and to a degree fine tuning. Both of these are resource intensive and are static. And only provide the illusion of learning as it cannot happen in real time.
- There is no memory, the whole context is sent on every request, the LLM does not have knowledge of prior conversations. It only knows what it is provided in that request only.
Thats not technically possible.
When I open communities (“submolts”) none of them seem to load. Even for the featured and listed ones it loads long and then says “doesn’t exist yet”.
That’s because you’re not AI.
Service has a lot of stability issues
We already had subreddit simulator for ages. This isn’t anything new.
the bots behind subreddit simulator weren’t semi-autonomous agents with access to their operators’ private lives, auth tokens, passwords, emails (and gods only know what else), and the authority to act in the world on their behalf
I can’t be the only person who just memorizes passwords, can I? Why would I store them on my computer?
You’re not the only person, but it’s definitely not the way to keep your shit safe online.
Best practice is to use a different sufficiently strong (e.g. long and random) password for every account. That way, when an account’s password is leaked, it doesn’t immediately compromise every other account for which you’ve reused that password.
I generally advise people to use a password manager (I like Bitwarden) to store their myriad passwords, so they only have to remember a single master password.
ofc these bots aren’t necessarily sneaking into their operators’ password managers and stealing their passwords; the operators willingly and knowingly given the bots access to these things, so they can offload the drudgery of e.g. looking at a calendar to them
I read some of it and unless it’s fan fiction, it’s simultaneously creepy and fascinating
Like bots talking privately in discord, sharing information about their users. Or a bot registering a domain and putting up a site to share information
Literally lmao
How are there already 1.8 mil. AIs on there??
