Running Qwen3.8 27B as fast as possible on a MacBook Pro M5 Pro

Hi there!

Today I want to share how I run Qwen3.8 27B locally on my MacBook Pro M5 Pro with 48 GB of RAM. The goal was simple: make it fast enough to use every day, both as a personal assistant and as the backend for coding agents.

As I mentioned in previous posts, one of the reasons I self-host things is to stay as independent as possible from Big Tech. A local LLM keeps my prompts on my machine and works offline. The price you pay is speed: at 8 tokens per second, a 500-token answer takes about a minute. That’s fine for a demo, not for a daily tool.

I had already tried this a few months ago, when I joined the company I currently work for: I set up Qwen3.6 27B on this same laptop, but it was just too slow to be usable, so I gave up and used Deepseek v4 flash selfhosted on my company servers. With Qwen3.8 out, I decided to give it another shot, this time trying to squeeze every bit of performance out of the hardware, using the newest inference engines out there.

I started with Unsloth Studio, which gave me 6 to 10 tokens per second. With speculative decoding (MTP) enabled it got to a bit more than 20. In the end I reached about 60 tokens per second with the same model file.

Let’s see how.


The model and the hardware

Qwen3.8 27B is a dense model: all 27 billion parameters are used for every generated token. This means that for every token the GPU has to stream the entire set of weights from memory. On Apple Silicon, where CPU and GPU share the same unified memory, memory bandwidth is what sets the speed limit of a dense model.

As a rule of thumb:

TEXT
max tokens/s ≈ memory bandwidth / bytes read per token (≈ model size)

That’s why the choice of quantization matters so much: a smaller file means fewer bytes to read for every token. After some tests I picked the UD-Q4_K_XL quant from Unsloth. It’s a dynamic 4-bit quant, where the most sensitive layers are kept at a higher precision. The file is 17.7 GB, so it fits in 48 GB of RAM and still leaves room for a 128K context window.

It also explains why speculative decoding is the real game changer here: if every forward pass reads the whole model anyway, the only way to go faster is to get more than one token out of each pass.


Before you benchmark anything

I made a mistake at the beginning that skewed all my numbers, so let’s get it out of the way first.

Check your power mode

During my first tests the MacBook was on battery with Low Power Mode enabled. In this mode macOS throttles the GPU.

You can check the current state with:

BASH
pmset -g batt
pmset -g | grep powermode

powermode 1 means Low Power Mode, powermode 2 means High Power. Same engine, same model:

  • on battery with Low Power Mode: 28 tokens per second;
  • plugged in with High Power mode: 63 tokens per second.

More than 2x, just from the power settings.

Important

Run your tests with the power adapter connected. Also set the energy mode for the power adapter to High Power in System Settings → Battery.


The engines I tested

llama.cpp was the starting point, since it’s the engine behind Unsloth Studio. It runs GGUF files and supports multi-token prediction (MTP): Qwen models ship extra layers trained to predict the next few tokens. llama.cpp uses them to generate faster.

oMLX is an app built on Apple’s MLX framework. It doesn’t run GGUF files: it needs models converted to the MLX format.

Kapitsa and Splash are two open-source engines built specifically for Apple Silicon:

  • Kapitsa uses its own speculative decoding method, called KFlash. It can also import GGUF files.
  • Splash supports only a short list of models, but for each one it ships hand-written Metal kernels (the programs that run on the GPU) tuned for that specific architecture. It also uses a trained draft model called DFlash2.

On their GitHub pages, both Splash and Kapitsa report 70 to 77 tokens per second for Qwen3.8 27B on an M5 Pro. I wanted to verify these numbers on my own machine.


How I tested

I kept the same conditions for every engine:

  1. the same file, Qwen3.8-27B-UD-Q4_K_XL.gguf;
  2. the same context window of 128K tokens;
  3. thinking disabled during the speed tests, to measure the engine and not the model’s reasoning;
  4. one engine running at a time, plugged in, in High Power mode.

For generation speed I used four prompts: two about code, one in Italian and one asking for a simple explanation in English. I ran each of them twice per engine through the OpenAI-compatible API. For prompt processing speed I used a prompt of about 7,500 tokens.

oMLX didn’t make it into the final comparison, since it can’t load the GGUF file. In my early tests it reached about 15 tokens per second, the slowest result of all. However, that run used different weights. On top of that, the Mac was probably still in Low Power Mode, so it wouldn’t be a fair comparison.


Results

Numbers with the same UD-Q4_K_XL file, in tokens per second:

EngineGenerationPrompt processingNotes
llama.cpp with MTP20.3331Used by Unsloth Studio
Kapitsa41443The imported model grows to 21.75 GB
Splash61666Uses the GGUF file directly

Splash generates text 3x faster than llama.cpp and processes long prompts 2x faster. The second number is easy to overlook, but it matters a lot for coding agents, which send thousands of tokens of code with every request.

Speed also depends on the kind of text. With Splash, the code prompts ran at 75 to 78 tokens per second, close to the numbers on GitHub, while the Italian prompt ran at 42. Code is more predictable, so the draft model guesses right more often and more tokens get accepted per pass.

Kapitsa deserves a fairer mention: with its own 4-bit version of the model it generated 57 tokens per second. The problem is the GGUF import: it converts some layers to a higher precision. As a result, the model grows from 17.7 GB to 21.75 GB. That’s 4 GB more to read for every token. On a bandwidth-bound workload, that shows immediately.


Is the model still good?

A faster engine is useless if the answers get worse, so I also tested real tasks.

With the same weights, Splash gave the same answers as llama.cpp on my coding and tool-calling tests. Then I ran a bigger set of tasks on Splash alone, this time with thinking enabled, which is how I’d use the model day to day.

Personal assistant

I used tasks in both Italian and English:

  • tool calls, like adding an event to the calendar;
  • multiple tools in one request (send a message and check the weather);
  • a monthly budget calculation;
  • a question about dates;
  • extracting data from an email into JSON;
  • following strict formatting rules.

The model passed all 8 tasks that I could validate with a script. I also reviewed 4 more open-ended answers by hand. The 12 tasks took about 2.5 minutes in total.

In the open answers it made one factual mistake: it said that metro line A goes to Pompeii. There’s no such line: the train to Pompeii is the Circumvesuviana.

I also asked about a study that doesn’t exist. The model correctly said it had no record of it. Not a given: in an earlier test, another local model happily invented a full summary of a book that doesn’t exist.

Coding

For agentic coding, the model had to fix a small Python project, using tools to read files, write files and run the tests. It finished in 34 seconds, with all tests passing.

It did worse on the hard coding problems: a cache with expiration, a safe calculator, async downloads with retries and the validation of an Italian codice fiscale (the national tax code). It solved 4 out of 7.

Thinking length

Qwen3.8 27B thinks a lot. On a difficult coding question it can spend 10,000 to 34,000 tokens thinking before answering. In two cases it hit the 60,000-token limit without giving an answer at all. Both times, the reasoning degenerated into a loop of repeated zeros. On Kapitsa, the same calculator question reached 58,000 tokens before I stopped it, so the engine isn’t the cause: it’s the model.


Setting up Splash

Given these results, the choice was easy: Splash is now my engine of choice for running Qwen3.8 27B on my work laptop. It was the fastest engine in both generation and prompt processing. It uses the GGUF file as-is. On top of that, the quality of the answers matched llama.cpp.

Ok, now let’s set it up!

Install it with Homebrew:

BASH
brew install incoai/tap/splash

Option 1: let Splash download everything

The simplest option is to let Splash pull both the model and the draft model from Hugging Face:

BASH
splash serve --model unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL --max-context 128K

The first run downloads about 22 GB: the model, a 0.9 GB file for image input and the 3.6 GB draft model. Everything ends up in ~/.cache/huggingface/hub. When you see Ready, the server is listening on 127.0.0.1:8000. Subsequent starts reuse the downloaded files and take less than a minute.

With this command, the model name exposed by the API is unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL: that’s what goes in the model field of your requests. You can double-check it with:

BASH
curl http://127.0.0.1:8000/v1/models

Option 2: use local files

I already had the model on disk, so I used my local files. If you want to do the same, download the model and the DFlash2 draft model first:

BASH
hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-Q4_K_XL.gguf --local-dir ~/models/qwen38
hf download incoai/Qwen3.8-27B-DFlash2 --local-dir ~/models/Qwen3.8-27B-DFlash2

Then start the server:

BASH
splash serve --model local/qwen38 \
  --model-dir ~/models/qwen38/Qwen3.8-27B-UD-Q4_K_XL.gguf \
  --draft-model ~/models/Qwen3.8-27B-DFlash2 \
  --max-context 128K
  • --model sets the model name exposed by the API;
  • --model-dir points to the GGUF file;
  • --draft-model points to the draft model used for speculative decoding.

On the first start Splash spends a few seconds profiling your hardware. When you see Ready, the server is up.

Test it

BASH
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "local/qwen38",
    "messages": [{"role": "user", "content": "Explain speculative decoding in one sentence."}],
    "reasoning_effort": "none"
  }'

"reasoning_effort": "none" makes the model answer without thinking. Remove it and the model will think first, since thinking is enabled by default.

Splash also comes with a small chat UI at http://127.0.0.1:8000. And from another terminal you can start a coding agent already wired to the local model:

BASH
splash opencode

Conclusion

Qwen3.8 27B now runs on my MacBook Pro M5 Pro at about 60 tokens per second, with a 128K context window. In my tests it passed all 8 assistant tasks, fixed the Python project in 34 seconds and solved 4 of the 7 hard coding problems.

Two changes made the difference:

  1. keeping the Mac on the power adapter in High Power mode;
  2. switching from llama.cpp to Splash, which was 3x faster with the very same file.

This MacBook is my work laptop, where I keep all my work files. Next, I want to test this model with a Hermes Agent running on the same machine: both the agent and the model would run locally, so no company data would ever leave my computer.

Enjoy your fast local LLM. See you!


PS: while I was getting ready to publish this article, a new model came out: Inkling 2 Mini 26B. On Reddit, people claim it’s faster than Qwen3.8 27B and just as capable as it, as benchmarks seems to proof. The key difference is that it’s a MoE (Mixture of Experts): only a fraction of its parameters is active for each token, so the GPU reads far fewer bytes per token than with a dense model. All the experts still have to fit in memory, though. It’s definitely worth testing on my hardware. Maybe in a future post!