What tools and projects do you use for your local LLM chats and agents?
I’m trying to sift through the mess of LLM harness and tools to find how to really get the most out of local LLMs.
You must log in or register to comment.
The anti-AI sentiment here is ridiculous. They were asking a legitimate question!
I just use ollama locally but I keep meaning to setup OpenRouter because I like to play with larger models that my hardware can’t handle and I’d like to be able to pick the best deals 🤷♀️
AI effectively raped and pillaged all human information for the benefit of a handful of corporations. It’s use has been shown to reduce the skills of school children and even deskill those with established knowledge. Then there’s the environmental considerations in a time where the world is decades behind appropriate climate change management… Id say it’s pretty understandable that it evokes passionate responses.
I’m sure OPs RTX 3060 will have a tolerable effect on the school children and the rainforest.
If OP is doing orbital dynamics in their day job, they probably live in California. JPL, Raytheon, LLNL, LBL, and Berkeley are all here.
Even if they don’t have rooftop solar, most of our grid is renewable. Their local GPUs have a decent chance of being more environmentally friendly than charging your cell phone, depending on where you live.
And using it for science and programming is really different than using ChatGPT to write you an email. I’m home right now, but I have two agents at work running debug and trial experiments, and they’ll be running it all night until I get in in the morning. I’ll review their work and steer them all day while I’m in the office; and while those two completely different projects are running, I’ll be personally working on a third. They are enormous productivity multipliers when you have a well-defined and deterministic target.
My grievance isn’t so much with current household models with PCs sitting there running off solar (that’s starting to get very bloody specific though). But how did you get to a point where you could get a local model on your PC. All the pillaging of content, hundreds of Billions to Trillion(s) of dollars spend on a product that didn’t renumerate anyone who created their learning data.
We involuntarily contributed to their creation, the least we get to do is enjoy the fruits of that contribution. More should be given, financially reimbursement through taxation, but until that happens, free use of the technology is consolation.
Besides, following through with your analogy, if a baby is born from rape - does that baby deserve hate for who their father is?
Unpopular opinion time: I’m torn as to where that theft line is.
If a turn of phrase in Lord of the Rings inspires something new in me, do I owe royalties to the Tolkien estate?
The stories on FanFiction.net are derivative by definition — is it theft? I love “Harry Potter and the Natural 20”, a 4th-wall breaking D&D and Potter universe mashup story (seriously, it’s hilarious). Does that story owe royalties to Wizards of the Coast and (ugh) JKR?
Maybe those are super transformative. What about opinion pieces where the author redigests information from all over, many paid writings? Are they selling the original authors’ product, even though restated?
Maybe it’s ok if they paid for the subs. But what if it came from free library books?
My wall has a bunch of art from artists at SDCC. I’m looking at Spider Gwen; Thor, Iron Man, and Mjolnir Cap charging into battle; Kaladin and Syl before the Stormfather; and a map of all the places the Gaang went in ATLA. Those artists all ingested the original work to make those derivatives; were they not allowed to make them if they borrowed their friends’ copy?
It feels kinda wrong to vacuum up public data to train models, kinda? But I don’t know that I can fairly call it immoral without condemning all these other things I think aren’t immoral. I kind of want to, but that’d be really inconsistent, and that’s profoundly frustrating in its own way.
It’s messy and complicated and I think the gut response of most people isn’t actually a reasoned one.
It’s a standing on the shoulders of giants thing. That’s how humanity have been making progress historically.
People running local LLMs with open weights abliterated models, especially with solar PV power (perhaps not as rare as you think, I’m upgrading my solar PV system in the next couple weeks for that explicit purpose), is about freedom and privacy. At some point there should be enough of them to link up into a grid over the Internet, collaborate or donate resources to distributed community projects. This is a way to allow for the “pillaged” content to pay back for itself.
Do you have qualms with public pirate libraries, like SciHub and LibGen, too? These have also been used for training LLMs.
My grievance isn’t so much with current household models with PCs sitting there running off solar (that’s starting to get very bloody specific though).
It’s actually my setup.
AI effectively raped and pillaged all human information
Copying information is not theft. Learning from copied information is not theft.
Theft is when you take something and deprive the owner of what you took. If I make a copy of your car, I am not stealing your car, you still have it and can drive it, wreck it, whatever.
Theft is when you take something and deprive the owner of what you took
So google ai summary now taking content from the web and putting at the top of their search, thus generally stopping people clicking on websites and taking their revenue -> theft.
So people should get free referrals to their site from google? Is that a right now?
Free referrals to their site? What are you talking about? Google is a search engine, an indexer, its role is to point people to websites based on their search terms. It now scrapes all of the websites information and people no longer visit those websites???
Whoosh!
Isn’t ollama made by Meta?
I just use llama.cpp with Zoo Code, the VSCode extension
join us !localllama@sh.itjust.works
php server and a local certificate authority.
Literally just having ollama installed and some models that work well with the amount of VRAM I have. I don’t auto-start the daemon because I rarely use it. Occasionally I use it for proof-reading/feedback on things like cover letters, or debugging. I don’t let it write anything. If it has something useful to say it can tell me in the chat and I can implement it if it’s good. Usually I don’t need it though so it just sits on my disk for the most part.
- Pi - plain and simple - run inside firejail.
- lamma.cpp w/ vulkan backend
That’s it.
This community won’t be much helpful but there is one specific for local LLMs. I think it was ai_ and some other…
Ollama on an N150 mini PC, 16GB, no GPU, Open WebUI on top. Being honest, 1-3B models are fine for searching my offline Kiwix stuff and 8B works if you don’t mind waiting. I’m turning this into a small box (heyeddi.com/projects/doomsday-box) and I keep going back and forth on whether people even want AI on it, or just the media and offline archives. Would you?
It depends on how complex you want to make it.
The simplest way would be to use a GUI tool like LM Studio, it has the benefit of selecting the model quants that are likely to work on your hardware and having sane defaults for the various parameters that you can work from.
Typically you’ll want to increase the context window, the default is good for short one-off questions but if you’re going to do RAG or coding tasks you will need a much larger context window. You will typically find this through trial and error, increasing until you hit memory limits or notice a large slowdown on tokens/second.
You can also have the model respond to an OpenAI-like API interface on a designated port/interface if you want to run non-chat tools through the same model.
Then, depending on the task that you’re trying to solve, you may want to find an MCP server built for that task which can provide the ability for the AI to interact with tools in a way that’s more controlled (giving an AI raw terminal access on a user’s machine is typically a bad idea).
If you need something more flexible that you can incorporate into scripts and automation, llama.cpp has been popular for a while. Like Linux distros, everyone is going to have an opinion on the best and whatever you decide on will be the wrong choice. There’s also Ollama and vLLM if you want to run a datacenter.
Model-wise it completely depends on the task. Do you need a multi-modal model that supports vision, is it going to be coding, is it trained to use tools/MCP servers, does it use reasoning, etc. The Qwen models are open weight and available in many different quants. The version 3.5 models run well on consumer hardware and can use vision, reasoning and tools.
I’m running it through Llama.cpp but it will likely be available via LM Studio as well.
vLLM to host and openwebui for a frontend that I really only use to demo models to friends.
harness
Opencode running with full access inside a rootless podman container with only the directories I want mounted (usually a single project, sometimes read only).
So far though, I’ve only used it to document and report on project repos because Qwen is not that good at actually coding (although it can give pretty decent conf file examples, but not always optimal).
Also useful for parsing pita data dump type files that I’m too lazy to construct multiline awk commands for.
front: open-webui skill servers: dockling - docs parsing for llm, searxng - searchengine for llm, open-terminal - linux terminal for llm (inside container) base: llama-swap Inside llama swap: qwen3-14b-q5, translator-nemo-12b, tower-plus-9b, translator-gemma4-26b, qwen3.6-35b-a3b llama stack: ROCm
Hardware: RX9070XT for llama-swap .
UPD: if you want easy way and run all in oneshot use https://github.com/Niko1221/Strata
Llama-swap running llama.cpp and strata. For the tools I use oh my pi and nvim with plugins
I’ve got an M4 Mac mini with 32 GB of unified memory, so I can run MoE models in the <40B parameter range quite comfortably. I use LM Studio as the backend with Hermes as the frontend. Works great for everyday chat, web search, and RAG.
Pi Coding Agent on the laptop, openwebui as a self hosted service.
I don’t host LLMs myself. My best hardware has 16GB of RAM, I could run a 4B model or so, but what’s the point? So I buy compute with ZDR turned on.
What’s stuck for me after trying too many things:
- Ollama (or llama.cpp server if you want control over flags) as the model server, bound to localhost only
- Open WebUI as the front-end; it also handles OpenAI-compatible APIs, so local and hosted models live in the same history
- LiteLLM in between once you have more than one backend: one endpoint, per-key budgets, and logs
- SearXNG as the search tool for anything agent-ish; keep concurrency low or upstream engines start blocking it
Biggest lesson: RAM decides what you can run, and document upload/embeddings in Open WebUI eat more memory than chat does. On small boxes, point embeddings at an API or give it 4 GB.
Openwebui pointing to llamacpp running on an old ass GPU. It’s good enough for what it is.
