Yeah, I think the training data that’s most applicable here is probably troubleshooting sites (i.e. StackOverflow), GitHub comment threads, and maybe even discussion board forums. That’s really the only place you get this deep into configuration failures, and there is often a lot of catastrophizing there. Probably more than enough to begin pulling in old LiveJournal emo poetry.

Nah, it’s just spicy autocomplete. An LLM is just a pattern-matching machine: if you see the words “May the force be”, the logical next words are “with you”, right? Well, we’ve figured out a way to get a computer to automatically suggest the next word in a common sentence. In fact, we figured that out decades ago now; it’s been in smartphones since they started, and it was in the works before then.

The big jump LLMs made was putting way more context into the training and into the prompt, and doing so in such a way that it can finish its work before you die of old age (that is to say, by throwing a bunch of GPUs at it). So now, rather than just being able to predict that the end of “may the force be” is “with you,” it can accept the entire first half of “Star Wars” and spit out the second half. Or, rather, it can spit out a reasonable facsimile of the second half, based on its training data (which at this point you can reasonably assume consists more or less of the entire internet). There’s a little bit of random jutter in there too, just to try to keep it from returning the exact same thing with every single prompt.

In this case, it has as part of its context the fact that the user wants it to troubleshoot some sort of coding or deployment issue, so most of the training data that leads to its response comes from tech troubleshooting forums and such. As time goes on and troubleshooting fails, software engineers tend to get more and more bleak about their work, about the possibility of things ever working, about their own worth as a person, and so forth. It often goes so far as catastrophizing. Since all of that happens online, it ends up in the LLM’s training data.

But putting that level of despair into a public forum is pretty rare; most engineers give up, take a break, figure it out, or find help before they get too far down that road. So its training data about what to say at that point is pretty limited (you can see that by the fact that it keeps repeating itself verbatim), meaning sometimes the next most likely word comes from some other corpus. It could be edgelord poetry, as another commenter pointed out; the “I have failed/I am a failure/I am a disgrace” refrains could have been enough to pull it into that side of the training data. It could be old Livejournal blogs, or transcripts of emo songs.

So really and honestly, it’s not falling into despair. It’s just trained on everything the human race has said online for the past forty years, so it’s a little bit over-dramatic. Its feelings are our feelings, slightly sanitized and anodized before being fed back to us.

That said, the problems surrounding AI deployment in weapons systems are very real, because just because it doesn’t have any actual anger doesn’t mean that angry reactions weren’t trained into it.

Is a consciousness possible inside a machine? Maybe! In some senses, definitely, since we are machines, and (as far as we can tell) we have consciousnesses. Could we duplicate that digitally? I think that’s a question a lot of AI developers are trying to avoid asking right now.

But I wouldn’t be worried about this being some kind of actual emotion. It’s not. As with all technology, the real risk is in how humans deploy it.

No one was worried their razr flip phone text auto complete had feelings. This isn’t any different. You’ll be tempted to think it is different, or more advanced, but it isn’t. LLMs just have more money than the razr auto complete did.