Search⌘K / Auto / Dark / Light

Mar 31, 2026 / 5 min read / 6 tags

EdgeRunner Beats llama.cpp: A Swift/Metal LLM Engine on Apple Silicon

Built EdgeRunner in pure Swift and Metal from scratch over a weekend. On Qwen3-0.6B Q8_0, it hits 212 tok/s on M3 Max, beating llama.cpp by 16%. Here’s what worked.

LLM Metal Apple Silicon Swift Performance Benchmark

The numbers

EdgeRunner is a Swift + Metal LLM inference engine for Apple Silicon. No llama.cpp. No MLX. No C++ at all. Everything from scratch.

Qwen3-0.6B Q8_0, 128-token greedy decode, M3 Max:

Engine	Decode tok/s
EdgeRunner	212
llama.cpp	~183
MLX (Q8_0)	~278

16% ahead of llama.cpp on the same model and chip. Still behind MLX by about 24%.

522 MB peak RSS. 4.1 ms time to first token.

Why build this

Almost every local LLM tool on Mac wraps llama.cpp or MLX. Both are great projects. But wrapping them means living with their architecture, their bottlenecks, their tradeoffs.

I wanted to see what happens when you start from zero. Pure Swift. Pure Metal. No C++ interop. No Python runtime. No Go wrapper. Just compute kernels talking directly to the GPU.

34 experiments in a single weekend. Most got rolled back. The ones that stuck pushed us to 212 tok/s.

The benchmark

That’s it. One command. Deterministic, repeatable, same test every time.

Configuration:

Model: Qwen3-0.6B Q8_0 (639 MB GGUF)
Decode: 128 tokens, greedy (temperature = 0)
Chip: Apple M3 Max
Build: Release
Correctness guard: greedy prefix must start with [1, 1479, 35]

The correctness guard is the part most benchmark posts skip. It’s easy to make a model fast by breaking determinism. We don’t count those runs. If the prefix changes, it’s a regression, not an optimization.

What worked

Five things got us from 0.06 tok/s to 212 tok/s. The other 29 experiments got reverted.

1. Fused GPU pipeline

All 28 transformer layers in a single Metal command buffer. 420+ dispatches, one sync point. Biggest win in the project.

Before: each layer created its own command buffer, committed it, waited for completion. 28 round-trips to the GPU per token. After: one command buffer, one commit, one wait.

Each dispatch encoding takes about 6 microseconds. Doesn’t sound like much until you’re doing it 420 times per token.

2. Fused Q8_0 dequant + GEMV

Kernels read quantized weights directly. No float32 materialization anywhere in the hot path.

Q8_0 weights are 1 byte per element. Float32 is 4 bytes. That’s a 4x bandwidth reduction. The fused kernel does dequant math inline during the matrix-vector multiply. No separate pass.

We tried Q4_0 for the LM head. Slower. The nibble extraction (AND/SHIFT per element) added so much ALU work that it ate the bandwidth savings. On Apple Silicon, GEMV stays memory-bound until the ALU catches up.

3. The mega-kernel

fused_qk_norm_rope_gqa does three things in one dispatch: per-head Q/K RMSNorm, NeoX RoPE rotation, and GQA attention. Replaced 56 dispatches with 1.

32 threads per head. One simdgroup. Each thread handles 4 head dimensions. KV threads compute norm + rotation, write to cache, and exit. Q threads keep going into the GQA phase. No threadgroup barrier between phases. The KV threads just return.

Zero barriers in the attention loop. That’s the point.

4. KV cache with single-token decode

During decode, only the new token gets processed. All previous K/V pairs are cached. Standard practice in every inference engine, but getting it right in Metal took a few tries.

The tricky part: asymmetric Q/KV shapes. Q is 1 token, KV is N tokens. The GQA kernel handles this with kvSeqLen and qOffset parameters.

5. Decode warmup

15 dummy decode passes before the timed runs. Pre-heats Metal pipeline states, command processor cache, DRAM page tables.

This one surprised me. I expected noise. Got +16% median throughput instead. The first timed decode has a cold-start penalty from GPU pipeline JIT compilation for decode-specific kernel variants. Prefill warmup doesn’t cover that.

The progression

Stage	tok/s	What changed
Baseline	0.06	First working Qwen3 inference
Weight buffer caching	3.6	Cached MTLBuffers, GPU LM head
Batched command buffers	5.1	Q/K/V in one buffer
Fused GPU pipeline	16	Entire transformer, one sync
Q8_0 fused kernels	63	Dequant inline with GEMV
Kernel fusion	120	RMSNorm+GEMV, Q/K norm+RoPE
Mega-kernel	212	Q/K norm + RoPE + GQA in one

0.06 to 212. 3,500x improvement. Most experiments rolled back. The ones that survived the correctness guard are the ones in the table.

What’s left

MLX hits ~278 tok/s on the same model and chip. We’re at 212. 66 tok/s gap.

The bottlenecks:

DRAM bandwidth: ~172 GB/s effective, 43% of M3 Max theoretical. MLX does better.
Dispatch count: still 140+ per decode. Metal Indirect Command Buffers could help.
Swift async overhead: ~0.23ms per decode call. A non-async forward pass would remove it.

Beating llama.cpp is nice. But MLX is the real target.

Run it yourself

EdgeRunner is open source. Loads GGUF models: Q8_0, Q4_K_M, Q6_K, whatever.



let model = try await LlamaLanguageModel.load(
from: URL(fileURLWithPath: "Qwen3-0.6B-Q8_0.gguf")
)

github.com/christopherkarani/EdgeRunner

If you’ve got an M1, M2, or M4 Mac:

Takes about 5 seconds. I’d love to see numbers from other chips.

Continue Reading

ls -la ./related

✍️

EdgeRunner Architecture: What We Found When We Read the Whole Damn Codebase

We fully documented EdgeRunner, a Metal LLM inference engine for Apple Silicon. Here’s what we found that was actually interesting.

↗ READ LLM Metal Apple Silicon

Mar 25, 2026 deep dive 7 min read

✍️

Wax: One File That Survives Hard Kills and Context Resets

I built Wax, a single-file crash-safe memory engine for AI agents on Apple Silicon: WAL durability, hybrid RAG, bitemporal facts, no cloud dependency.

↗ READ swift on-device-ai ai-agents

May 28, 2026 ai 11 min read

✍️

Inside Swarm’s WebSearchTool: How I Built It to Fit a 4K Context Window

I built Swarm’s Swift web tool for 4,096-token on-device LLMs. Here’s the tiered envelope, the Wax-without-embeddings bet, and fourteen decisions behind it.

↗ READ swift on-device-ai ai-agents

Apr 22, 2026 ai 20 min read

SYSTEM ONLINE