Running LLMs locally in 2025-26: what to expect + how to get started

If you’re excited to run open source LLMs like GPT-OSS, Qwen3, and Gemma on your laptop, desktop, or phone as a learning experience, or just for the technerdy coolness of running actual AI locally, on your own device, you’ll have a great time as long as your expectations are aligned with how AI runs on consumer hardware.

But, if you’re hoping it will be like having your own Claude 4.5 or ChatGPT 4.2 running locally, you’ll be disappointed for three reasons:

1. The open source models that can be run on consumer hardware are not comparable to frontier models created by Google, OpenAI, Anthropic, and the other “big players.” Using Qwen3 8b with 4-bit quantization isn’t going to compare to using GPT-5.2 Thinking. Even running GPT-OSS 120b, if your device can handle it, isn’t going to be comparable. These smaller models that can run on consumer hardware, even the latest (expensive) desktop hardware, simply are not as intelligent as the trillion+ parameter frontier models.

2. What you’ll run locally will be a more raw user experience. It won’t be as refined or polished because it won’t be wrapped in the unseen backend layers that enhance the user experience of commercial chat products like ChatGPT, Gemini, and Claude. Locally, you’ll hit an LLM with a chat request and get output back. There’s no sophisticated RAG that gives the model context from your other chats, no memory solution for retaining key facts, no multi-step agentic processes for things like deep research. These often transparent enhancements simply aren’t there augmenting and improving the user experience.

3. It’s going to be a lot slower. Even on an M-series Mac utilizing MLX with ample RAM, you’re not going to get the consistent near-instant time to first token and 100+ tokens per second firehouse of streamed output. It can feel like very slow after becoming acclimated to the stellar performance of the big chatbot solutions.

So, you’re going to be using a less intelligent model, it’s not going to be as capable, and it’s going to be slower.

But don’t let that dissuade you!

I’m not trying to discourage you from starting with local LLMs, but I do want you to know what to expect going in, so you don’t get frustrated or give up and write off running models locally altogether.

And, this shouldn’t be taken as a slight against the thriving open source AI community which is doing mind-blowing things with far fewer resources than their well-funded commercial counterparts. In fact, the capabilities of the open source models are incredible and I’m a huge supporter of the open source AI community. I think it’s a critical counter-balance to a very possible tech dysotopia where the wrong people end up controlling AI and advanced robotics.

What running models locally is like vs the frontier models

But, simply for expectation setting, it’s important for people to know running Qwen3 30b locally isn’t going to be the same as chatting with GPT 5.2 or Opus 4.5. It’s going to be slower. It’s going to be less intelligent. It’s going to make more mistakes. You won’t be able to instantly flip into Deep Research mode and have it compile a 15 page report with 100 citations after reviewing 450 links.

Open source models like Qwen3 30b and GPT-OSS 20b and 120b are insanely capable for their size, marvelous feats of engineering, and it’s mind-bending that we have models as intelligent and capable as these running on our laptops, tablets, desktops, and even our phones, but these models aren’t trillions of parameters and consumer GPUs aren’t comparable to data centers, and that becomes apparent pretty fast after you get Ollama or LMstudio installed, pull your first model, and send your first few messages.

Those first few messages are incredible, don’t get me wrong. I know I felt a sense of awe, personally. This wasn’t something I thought I’d ever be able to do in my lifetime. Technology felt magical again — like logging onto the internet in 1995.

Then, my fan started roaring, my other apps started running slow, and I realized the responses I was getting weren’t what I was used to getting from the web-based paid chatbot solutions like ChatGPT, Gemini, or Claude.

When I asked for ideas, they weren’t as creative or considered, when I asked for writing, the quality was subpar. It didn’t have tools or integrations, it couldn’t make and edit files, it couldn’t search through our past discussions, so I needed to explain every bit of context it needed to do a good job. It’s context window was smaller so as the conversation continued, the quality of responses dropped off further.

It is really cool to run an AI locally on your computer, but it’s even more cool when it gives the right answer, has a great voice mode, is nearly instant, and the phone app is awesome. Also nice if I can connect to it Google Drive, Gmail, etc. That is to say, even if the hardware constraint weren’t there, the layers of invisible backend that facilitate the polished, robust chat experience we’ve come to expect, simply don’t exist for local, open source options… yet.

But just because an open source ~ 30b parameter model can’t compete head-to-head against models far, far above its weight class, that does not mean an equal or better user experience is unachievable with open source models running on consumer hardware. It just means it hasn’t been created yet and that’s what we’re doing with Ally.

How to run an LLM locally

Getting set up to run an LLM on your computer is easier than ever, but still requires some set up and command line usage. If you’d prefer to avoid using Terminal commands, you may want to skip Ollama and instead use Ally, which is the easest, simplest way to get started with local AI on a Mac.

Ollama is another excellent choice, though it does require some command line.

Run Qwen3 8B locally with Ollama (step-by-step)

1. Install Ollama • macOS: Download + install Ollama (requires macOS 14 Sonoma+).  • Windows: Download + run the installer (requires Windows 10+).  • Linux (recommended):

curl -fsSL https://ollama.com/install.sh | sh

2. Start Ollama (Linux) + sanity check

On Linux, start the server:

ollama serve

Then confirm it’s installed:

ollama -v

(On macOS/Windows, installing the app typically starts the Ollama service; the CLI should work once it’s running.) 

3. Download (pull) Qwen3 8B

ollama pull qwen3:8b

4. Run it

ollama run qwen3:8b

5. Send a message using the local HTTP API

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [{"role":"user","content":"Hello!"}],
  "stream": false
}'

How to run an LLM locally with Ally

Ally is a newer solution for running LLMs on device. It is an Apple-only solution, but if you have a Mac, there’s really no easier way to run inference on your Macbook or desktop.

1. Download Ally for macOS from the app store.
2. Ally will automatically download Qwen3 8b the first time you open it.
3. Create a new node by clicking +, hover near the bottom of the window to display the chat input field, and send a message.

That’s really it. No Docker or command line needed. You can switch to a different model in Settings > AI.

Ally also gives you the advantage of a collaborative Markdown editor with shared context, flexible, drag and drop file and chat organization, and more.