Hi there!
Today I want to share how I run Qwen3.8 27B locally on my MacBook Pro M5 Pro with 48 GB of RAM. The goal was simple: make it fast enough to use every day, both as a personal assistant and as the backend for coding agents.
As I mentioned in previous posts, one of the reasons I self-host things is to stay as independent as possible from Big Tech. A local LLM keeps my prompts on my machine and works offline. The price you pay is speed: at 8 tokens per second, a 500-token answer takes about a minute. That’s fine for a demo, not for a daily tool.
I had already tried this a few months ago, when I joined the company I currently work for: I set up Qwen3.6 27B on this same laptop, but it was just too slow to be usable, so I gave up and used Deepseek v4 flash selfhosted on my company servers. With Qwen3.8 out, I decided to give it another shot, this time trying to squeeze every bit of performance out of the hardware, using the newest inference engines out there.
I started with Unsloth Studio, which gave me 6 to 10 tokens per second. With speculative decoding (MTP) enabled it got to a bit more than 20. In the end I reached about 60 tokens per second with the same model file.
Let’s see how.
The model and the hardware
Qwen3.8 27B is a dense model: all 27 billion parameters are used for every generated token. This means that for every token the GPU has to stream the entire set of weights from memory. On Apple Silicon, where CPU and GPU share the same unified memory, memory bandwidth is what sets the speed limit of a dense model.
As a rule of thumb:
max tokens/s ≈ memory bandwidth / bytes read per token (≈ model size)That’s why the choice of quantization matters so much: a smaller file means fewer bytes to read for every token. After some tests I picked the UD-Q4_K_XL quant from Unsloth. It’s a dynamic 4-bit quant, where the most sensitive layers are kept at a higher precision. The file is 17.7 GB, so it fits in 48 GB of RAM and still leaves room for a 128K context window.
It also explains why speculative decoding is the real game changer here: if every forward pass reads the whole model anyway, the only way to go faster is to get more than one token out of each pass.
Note
Unsloth updates their quants from time to time. In November they uploaded new versions of this model. If you downloaded yours a few months ago, check if a newer version is available.
Before you benchmark anything
I made a mistake at the beginning that skewed all my numbers, so let’s get it out of the way first.
Check your power mode
During my first tests the MacBook was on battery with Low Power Mode enabled. In this mode macOS throttles the GPU.
You can check the current state with:
pmset -g batt
pmset -g | grep powermodepowermode 1 means Low Power Mode, powermode 2 means High Power. Same engine, same model:
- on battery with Low Power Mode: 28 tokens per second;
- plugged in with High Power mode: 63 tokens per second.
More than 2x, just from the power settings.
Important
Run your tests with the power adapter connected. Also set the energy mode for the power adapter to High Power in System Settings → Battery.
The engines I tested
llama.cpp was the starting point, since it’s the engine behind Unsloth Studio. It runs GGUF files and supports multi-token prediction (MTP): Qwen models ship extra layers trained to predict the next few tokens. llama.cpp uses them to generate faster.
oMLX is an app built on Apple’s MLX framework. It doesn’t run GGUF files: it needs models converted to the MLX format.
Kapitsa and Splash are two open-source engines built specifically for Apple Silicon:
- Kapitsa uses its own speculative decoding method, called KFlash. It can also import GGUF files.
- Splash supports only a short list of models, but for each one it ships hand-written Metal kernels (the programs that run on the GPU) tuned for that specific architecture. It also uses a trained draft model called DFlash2.
On their GitHub pages, both Splash and Kapitsa report 70 to 77 tokens per second for Qwen3.8 27B on an M5 Pro. I wanted to verify these numbers on my own machine.
Note
Most of these engines rely on speculative decoding. A small draft model guesses the next few tokens. The big model then verifies all of them in a single forward pass. When the guesses are right, one pass of the big model produces several tokens instead of one. The big model still has the final say on every token, so the output quality doesn’t change: only the speed does.
How I tested
I kept the same conditions for every engine:
- the same file,
Qwen3.8-27B-UD-Q4_K_XL.gguf; - the same context window of 128K tokens;
- thinking disabled during the speed tests, to measure the engine and not the model’s reasoning;
- one engine running at a time, plugged in, in High Power mode.
For generation speed I used four prompts: two about code, one in Italian and one asking for a simple explanation in English. I ran each of them twice per engine through the OpenAI-compatible API. For prompt processing speed I used a prompt of about 7,500 tokens.
oMLX didn’t make it into the final comparison, since it can’t load the GGUF file. In my early tests it reached about 15 tokens per second, the slowest result of all. However, that run used different weights. On top of that, the Mac was probably still in Low Power Mode, so it wouldn’t be a fair comparison.
Results
Numbers with the same UD-Q4_K_XL file, in tokens per second:
| Engine | Generation | Prompt processing | Notes |
|---|---|---|---|
| llama.cpp with MTP | 20.3 | 331 | Used by Unsloth Studio |
| Kapitsa | 41 | 443 | The imported model grows to 21.75 GB |
| Splash | 61 | 666 | Uses the GGUF file directly |
Splash generates text 3x faster than llama.cpp and processes long prompts 2x faster. The second number is easy to overlook, but it matters a lot for coding agents, which send thousands of tokens of code with every request.
Speed also depends on the kind of text. With Splash, the code prompts ran at 75 to 78 tokens per second, close to the numbers on GitHub, while the Italian prompt ran at 42. Code is more predictable, so the draft model guesses right more often and more tokens get accepted per pass.
Kapitsa deserves a fairer mention: with its own 4-bit version of the model it generated 57 tokens per second. The problem is the GGUF import: it converts some layers to a higher precision. As a result, the model grows from 17.7 GB to 21.75 GB. That’s 4 GB more to read for every token. On a bandwidth-bound workload, that shows immediately.
Is the model still good?
A faster engine is useless if the answers get worse, so I also tested real tasks.
With the same weights, Splash gave the same answers as llama.cpp on my coding and tool-calling tests. Then I ran a bigger set of tasks on Splash alone, this time with thinking enabled, which is how I’d use the model day to day.
Personal assistant
I used tasks in both Italian and English:
- tool calls, like adding an event to the calendar;
- multiple tools in one request (send a message and check the weather);
- a monthly budget calculation;
- a question about dates;
- extracting data from an email into JSON;
- following strict formatting rules.
The model passed all 8 tasks that I could validate with a script. I also reviewed 4 more open-ended answers by hand. The 12 tasks took about 2.5 minutes in total.
In the open answers it made one factual mistake: it said that metro line A goes to Pompeii. There’s no such line: the train to Pompeii is the Circumvesuviana.
I also asked about a study that doesn’t exist. The model correctly said it had no record of it. Not a given: in an earlier test, another local model happily invented a full summary of a book that doesn’t exist.
Coding
For agentic coding, the model had to fix a small Python project, using tools to read files, write files and run the tests. It finished in 34 seconds, with all tests passing.
It did worse on the hard coding problems: a cache with expiration, a safe calculator, async downloads with retries and the validation of an Italian codice fiscale (the national tax code). It solved 4 out of 7.
Thinking length
Qwen3.8 27B thinks a lot. On a difficult coding question it can spend 10,000 to 34,000 tokens thinking before answering. In two cases it hit the 60,000-token limit without giving an answer at all. Both times, the reasoning degenerated into a loop of repeated zeros. On Kapitsa, the same calculator question reached 58,000 tokens before I stopped it, so the engine isn’t the cause: it’s the model.
Warning
If you enable thinking, use a large context window (I used 128K) and, for coding, cap the reasoning with a thinking budget. At 60 tokens per second, 60,000 tokens of thinking take more than 15 minutes.
Setting up Splash
Given these results, the choice was easy: Splash is now my engine of choice for running Qwen3.8 27B on my work laptop. It was the fastest engine in both generation and prompt processing. It uses the GGUF file as-is. On top of that, the quality of the answers matched llama.cpp.
Ok, now let’s set it up!
Important
Splash requires an Apple Silicon Mac with an M3 or newer chip, macOS 26.4 or later and at least 36 GB of RAM.
Install it with Homebrew:
brew install incoai/tap/splashOption 1: let Splash download everything
The simplest option is to let Splash pull both the model and the draft model from Hugging Face:
splash serve --model unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL --max-context 128KThe first run downloads about 22 GB: the model, a 0.9 GB file for image input and the 3.6 GB draft model. Everything ends up in ~/.cache/huggingface/hub. When you see Ready, the server is listening on 127.0.0.1:8000. Subsequent starts reuse the downloaded files and take less than a minute.
With this command, the model name exposed by the API is unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL: that’s what goes in the model field of your requests. You can double-check it with:
curl http://127.0.0.1:8000/v1/modelsOption 2: use local files
I already had the model on disk, so I used my local files. If you want to do the same, download the model and the DFlash2 draft model first:
hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-Q4_K_XL.gguf --local-dir ~/models/qwen38
hf download incoai/Qwen3.8-27B-DFlash2 --local-dir ~/models/Qwen3.8-27B-DFlash2Then start the server:
splash serve --model local/qwen38 \
--model-dir ~/models/qwen38/Qwen3.8-27B-UD-Q4_K_XL.gguf \
--draft-model ~/models/Qwen3.8-27B-DFlash2 \
--max-context 128K--modelsets the model name exposed by the API;--model-dirpoints to the GGUF file;--draft-modelpoints to the draft model used for speculative decoding.
On the first start Splash spends a few seconds profiling your hardware. When you see Ready, the server is up.
Warning
With local files, Splash does not download the draft model for you, so don’t skip the second hf download. Also, the model path must end with .gguf: the Hugging Face cache stores the actual files as blobs without an extension. Splash rejects those paths with not a GGUF file.
Test it
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "local/qwen38",
"messages": [{"role": "user", "content": "Explain speculative decoding in one sentence."}],
"reasoning_effort": "none"
}'"reasoning_effort": "none" makes the model answer without thinking. Remove it and the model will think first, since thinking is enabled by default.
Splash also comes with a small chat UI at http://127.0.0.1:8000. And from another terminal you can start a coding agent already wired to the local model:
splash opencodeNote
To cap the reasoning, Splash provides the --thinking-token-budget option. For example, with --thinking-token-budget 8000 the model stops thinking after about 2 minutes at 60 tokens per second.
Conclusion
Qwen3.8 27B now runs on my MacBook Pro M5 Pro at about 60 tokens per second, with a 128K context window. In my tests it passed all 8 assistant tasks, fixed the Python project in 34 seconds and solved 4 of the 7 hard coding problems.
Two changes made the difference:
- keeping the Mac on the power adapter in High Power mode;
- switching from llama.cpp to Splash, which was 3x faster with the very same file.
This MacBook is my work laptop, where I keep all my work files. Next, I want to test this model with a Hermes Agent running on the same machine: both the agent and the model would run locally, so no company data would ever leave my computer.
Enjoy your fast local LLM. See you!
PS: while I was getting ready to publish this article, a new model came out: Inkling 2 Mini 26B. On Reddit, people claim it’s faster than Qwen3.8 27B and just as capable as it, as benchmarks seems to proof. The key difference is that it’s a MoE (Mixture of Experts): only a fraction of its parameters is active for each token, so the GPU reads far fewer bytes per token than with a dense model. All the experts still have to fit in memory, though. It’s definitely worth testing on my hardware. Maybe in a future post!
