At home, she has forbidden her 10-year-old daughter from using chatbots. “She has to learn critical thinking skills first or she won’t be able to tell if the output is any good,” the rater said.
And this is why the vast majority of people, particularly in the USA, should not be using AI. Critical thinking has been a weakness in the USA for a very long time and is essentially a now four-letter word politically. The conservatives in the USA have been undermining the education system in red states because people with critical thinking skills are harder to trick into supporting their policies. In 2012, the Texas Republican Party platform publicly came out as opposed to the teaching of critical thinking skills.
We oppose the teaching of Higher Order Thinking Skills (HOTS) (values clarification), critical thinking skills and similar programs that are simply a relabeling of Outcome-Based Education (OBE) (mastery learning) which focus on behavior modification and have the purpose of challenging the student’s fixed beliefs and undermining parental authority.
This has been going on at some level for more than 4 decades. The majority of people in those states have never been taught the skills and knowledge to safely use these tools safely. In fact, their education has, by design, left them easily manipulated by those in power, and now, by LLMs too.
In other news:
Meet the Drug Dealers That Won’t Let Their Friends Do Their Drugs
Meet the Pimps That Won’t Let Their Kids Become Prostitutes
Meet the People Dismantling Public Education While Their Families All Attend Private Schools
I work with AI and use it personally, but I have my own servers running local models which solves tons of privacy concerns. The inaccuracy is another problem but not a big one for me as I know it and will simply fact check. Also, I don’t really use it for knowledge anyway. Just to filter news to my interest, help with summaries and translation etc.
People use AI as some all-knowing oracle but an LLM is not meant for that at all.
This is the correct way to use it. In a field you are already very knowledgeable in, so you can do your own fact-checking. This is absolutely paramount. But most people are content to just copy-paste and don’t even ask the llm for sources.
Could you elaborate a little on your setup? Sounds interesting
I have one server with a cheap MI50 instinct. Those come for really cheap on eBay. And it’s got really good memory bandwidth with HBM2. They worked ok with ollama until recently when they dropped support for some weird reason but a lot of other software still works fine. Also older models work fine on old ollama.
The other one runs an RTX 3060 12GB. I use this for models that only work on nvidia like whisper speech recognition.
I tend to use the same models for everything so I don’t have the delay of loading the model. Mainly uncensored ones so it doesn’t choke when someone says something slightly sexual. I’m in some very open communities so standard models are pretty useless with all their prudeness.
For frontend i use OpenWebUI and i also run stuff directly against the models like scripts.
This is the way.
…Except for ollama. It’s starting to enshittify and I would not recommend it.
Agreed. The way they just dumped support for my card in some update with some vague reason also irked me (we need a newer rocm they said but my card works fine with all current rocm versions)
Also the way they’re now trying to sell cloud AI means their original local service is in competition to the product they sell.
I’m looking to use something new but I don’t know what yet.
I’ll save you the searching!
For max speed when making parallel calls, vllm: https://hub.docker.com/r/btbtyler09/vllm-rocm-gcn5
Generally, the built in llama.cpp server is the best for GGUF models! It has a great built in web UI as well.
For a more one-click RP focused UI, and API server, kobold.cpp rocm is sublime: https://github.com/YellowRoseCx/koboldcpp-rocm/
If you are running big MoE models that need some CPU offloading, check out ik_llama.cpp. It’s specifically optimized for MoE hybrid inference, but the caveat is that its vulkan backend isn’t well tested. They will fix issues if you find any, though: https://github.com/ikawrakow/ik_llama.cpp/
mlc-llm also has a Vulcan runtime, but it’s one of the more… exotic LLM backends out there. I’d try the other ones first.
Thank you so much!! I have been putting it off because what I have works but a time will soon come when I’ll want to test new models.
I’m looking for a server but not many parallel calls because I would like to use as much context as I can. When making space for e.g. 4 threads, the context is split and thus 4x as small. With llama 3.1 8b I managed to get 47104 context on the 16GB card (though actually using that much is pretty slow). That’s with KV quant to 8b too. But sometimes I just need that much.
I’ve never tried the llama.cpp directly, thanks for the tip!
Kobold sounds good too but I have some scripts talking to it directly. I’ll read up on that too see if it can do that. I don’t have time now but I’ll do it in the coming days. Thank you!
Vllm is a bit better with parallelization. All the kv cache sits in a single “pool”, and it uses as many slots as will fit. If it gets a bunch of short requests, it does many in parallel. If it gets a long context request, it kinda just does that one.
You still have to specify a maximum context though, and it is best to set that as low as possible.
…The catch is it’s quite vram inefficient. But it can split over multiple cards reasonably well, better than llama.cpp can, depending on your PCIe speeds.
You might try TabbyAPI exl2s as well. It’s very good with parallel calls, thoughts I’m not sure how well it supports MI50s.
Another thing to tweak is batch size. If you are actually making a bunch of 47K context calls, you can increase the prompt processing batch size a ton to load the MI50 better, and get it to process the prompt faster.
EDIT: Also, now that I think about it, I’m pretty sure ollama is really dumb with parallelization. Does it even support paged attention batching?
The llama.cpp server should be much better, eg use less VRAM for each of the “slots” it can utilize.
Bloefz has a great setup. Used Mi50s are cheap.
An RTX 3090 + a cheap HEDT/Server CPU is another popular homelab config. Newer models run reasonably quickly on them, with the attention/dense layers on the GPU and sparse parts on the CPU.
How do you know it’s doing any of this correctly, especially filtering and translations?
I mainly use it for Spanish which I have a basic proficiency in. It just accompanies me on my learning journey. It may be wrong sometime but not often. Like the other reply said, LLMs are good at languages, it’s what they were originally designed for until people found out they could do more (but not quite as well).
And as for filtering, I just use it as a news feed sanitizer with a whole bunch of rules. It will miss things sometimes but it’s also my ruleset that’s not perfect. I often come across the unfiltered sources anyway and even if it misses something, it’s only news. Nothing really important to me.
It’s funny, I had half been avoiding it for languages. I had lots of foreign friends and they often lived together in houses and those houses would almost have this creole. They came to learn English and were reinforcing their own mistakes but it was mutually intelligible so the mistakes were reinforced and not caught. I suspect LLMs would be amazing at doing that to people and their main use case along these lines seems like it would be to practice at a slightly higher level than you so I suspect some of those errors would be hard to catch / really easy to take as correct instead of validating
Anyone learning a new language massively benefits from being able to speak with native speakers.
That being said, LLMs are better at languages and translation tasks than any pretty much anything else. If you need vocabulary help or have difficulty with grammar they’re incredibly helpful (vs Googling and hoping someone had the same issue and posted about it on Reddit).
I mean, if you can afford a native speaker tutor that is the superior choice. But, for the average person, an LLM is a massive improvement over trying to learn via YouTube or apps.
And the problem with Reddit–especially with certain language communities–is you’ll get a hallucination rate higher than current LLMs because learners can either overestimate their knowledge or sound off just because they want to show off.
I don’t recommend LLM use for beginners at languages but once they get a semester or two (or the equivalent) under their belt, the instant access to an answer that’s right most of the time is invaluable. Just first get to the point where you can start to recognize “maybe that’s not quite right…” first, and check sources. And definitely check in with natives as much as possible.
I don’t think that’s a problem. I live in Spain and speak Spanish daily with real people, many of them my friends. They’ll correct me if needed, they often do. Though most are my own mistakes.
Don’t forget people give wrong answers too. But people aren’t available 24/7 to help me.
This has a strong whiff of the former Facebook engineers that forbade their families from using the platforms they built.
We’re living in a Black Mirror episode.
we’re living in multiple concurrent Black Mirror episodes… a multifaceted dystopia
AI companies want to sell their product. They tell us all the good stuff. Consumers take the bait. Normal capitalism today.
