rncz.net

Run 125B AI Models Locally with Strata: Bye Bye 27B

llmstrataqwenmoelocal-aicoding-agentpiamdrx7900xtxlinuxwindows

My AMD GPU only has 24 GB of VRAM, so I thought I was stuck running small 27B models locally. Then I found Strata — and now I’m running a 125B model on the same machine by using system RAM and VRAM together. Concretely: Qwen3.8-Flash-Next on 64 GB of RAM plus a Radeon RX 7900 XTX with 24 GB of VRAM.

Strata has replaced llama.cpp for me. It is fast, it is good, and it is my daily driver right now. If you are running a Qwen3.8 27B model today, this is worth a look.

1. What is Strata (and Qwen3.8-Flash-Next)?

On the GitHub repository it is described in one line: Strata runs Qwen3.8-Flash-Next on a normal PC. It writes code, reads pictures, and works with your apps and coding agents. Nothing leaves your PC.

The trick is that Qwen3.8-Flash-Next is a mixture-of-experts model: the weights are split into thousands of small specialist “experts”, and every token only needs a handful of them. Strata keeps the experts that get used most often in VRAM, keeps all of them in system RAM, and swaps them in and out while it generates. That’s how a 125B model fits on a PC with 12 GB of VRAM or more.

So how do you position it? Think of it as something like Ollama, LM Studio or llama.cpp — but on steroids: an installer that picks the right model for your hardware, a browser app with chat and live monitoring, and an OpenAI-compatible API on localhost:8080.

What you need:

GPUNVIDIA RTX 20/30/40/50, or AMD RX 7900 XT/XTX, 7800 XT, 9070 XT, AI PRO R9700 and more — 12 GB VRAM or more
RAM32 GB minimum, 64 GB runs every model size
Disk~80 GB free, SSD strongly recommended
OSWindows 10/11 or Linux

2. Setup on Windows

Download Strata from GitHub with git clone, or just download the ZIP and extract it. Then double-click START-HERE.bat.

The installer asks you a few questions — take the recommended answer each time unless you know better:

  1. Which model — for a 32 GB RAM machine I picked option 1 (the Coder build, which is made for code and fits 32 GB).
  2. Context — I took the recommended 120K tokens.
  3. Which size — the 8-bit option in my case.
  4. Images? — yes.

Press Enter, wait for the download (the model is tens of gigabytes, so give it time), and when it’s done Strata opens in your browser.

3. Setup on Linux

On Linux it’s the same flow in a terminal — cd into the Strata folder and run:

./setup.sh

My daily setup is a bit different: I start Strata with ./setup.sh --setup, so I can re-pick the model every time without reinstalling anything. This time:

  1. Model — option 1, Qwen3.8-Flash-Next.
  2. Size — IQ3_S, the best (and slowest) size. My Linux PC has 64 GB of system RAM, so every expert fits.
  3. Context — 200K tokens.
  4. KV cache — 8-bit (int8), which buys you a lot of context for very little quality.
  5. Images — no.

Then it downloads the model and starts. Open your browser and go to localhost:8080 — Strata opens the chat for you, and you can ask whatever you want. Let’s ask it to build tic-tac-toe.

4. Chat, Monitor and About

While the model is generating, open the Monitor tab — and this is one of the really nice things about Strata: you immediately see everything that’s going on.

You can see part of the VRAM in use. Those are the active experts of the mixture-of-experts model; the other experts stay in system RAM and get swapped in as they’re needed. And I have to say it: the speed is better than when I ran Qwen3.8-27B at 4-bit quantization. A bigger model that only activates a fraction of its weights per token beats a smaller dense model on this hardware.

The About tab shows everything about the running model: the 200K context window, which GPU it found, and how much RAM is available. If something is slow, that’s where you check whether you ran out of RAM.

5. Connect Strata to the Pi coding agent

If you don’t know how to install the Pi coding agent yet, check out my earlier post on Pi + llama.cpp first.

Then go to the .pi/agent folder in your home directory and create (or edit) models.json:

{
  "providers": {
    "strata_local": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "ollama",
      "models": [
        {
          "id": "Strata (Ubuntu 26.04 LTS)",
          "name": "Qwen3.8-Flash 125B (IQ3_S by ISTA-DASLab)",
          "input": ["text"],
          "contextWindow": 200000
        }
      ]
    }
  }
}

The important part is that the base URL points to http://localhost:8080/v1. The API key doesn’t really matter — Strata accepts anything. Save the file with Ctrl+S.

Now start the agent:

pi

Run /model — and Strata is already in the list. Select it, say “hi”, and your Pi agent is connected to a 125B model running entirely on your own PC.

6. Pelican on a bicycle

Time for the usual test:

Create an SVG of a pelican on a bicycle. The pelican must be moving, and it must be made in HTML.

Press Ctrl+T to watch what it’s thinking. It tries all kinds of approaches, and then it’s already writing the HTML file.

One downside of Flash-Next compared to the 27B model: it uses noticeably more electricity. So do check your power bill.

7. The result

After a few minutes of running the agent, here’s the result — and it’s pretty impressive what this model is capable of. I use it for literally everything I build nowadays.

By the way: let me know in the comments how your experience with Strata and your agent harness has been. If you have any questions, ask away too.

Next steps

  • Try the sizes in order: Coder on 32 GB, IQ2_XS on 48–64 GB, IQ3_S on 64 GB+, and Unsloth’s 4-bit files if you have 96 GB+.
  • Turn on the 8-bit KV cache to get a much bigger context window for almost no quality loss.
  • Point every other app at it too — anything that speaks the OpenAI API can use http://localhost:8080/v1.
  • Run ./bench.sh (BENCH.bat) to measure chat, prompt reading, and an agent session on your own machine.
← All posts