FreeToken Runs 753B GLM-5.2 on a Single Workstation GPU, Researchers Say

According to the report, FreeToken achieves interactive speed for a 35B model on an 8GB laptop GPU, runs 284B on a gaming desktop, and handles the 753B GLM-5.2 on one workstation card. The target gap is the arithmetic of Mixture-of-Experts: DeepSeek-V4-Flash activates 6 of 256 routed experts in each of 43 layers, leaving only 13B of 284B parameters participating in any single token, but at FP4 the full expert set is roughly 140GB, so inactive experts reside in host memory and enter the execution path on demand.

The researchers isolated three failure modes in existing engines such as llama.cpp, KTransformers, Ollama and MoE-Infinity. Prefill destroys sparsity because thousands of tokens per layer route to nearly the whole expert set, so a prefill pass streams the entire pool across PCIe. Static placement misses decode traffic: llama.cpp assigns MoE tensors at load time and KTransformers pins a hot subset, while routing shifts every token, leaving most expert evaluations to the CPU with the GPU and PCIe link idle. Consumer CPUs cannot carry the remainder, as dual-channel DDR5 delivers 80–90GB/s against the 1–1.8TB/s an RTX 4090 or 5090 draws from on-package memory.

FreeToken addresses these with three mechanisms. Bandwidth-adaptive execution, called the q* policy, splits each step's cache misses between GPU-resident experts and CPU-computed experts, merging the partial sums exactly without approximation or router modification. Semantic-aware caching uses full-layer double buffering during prefill, anchors recurrent-state checkpoints at special-token boundaries such as thinking blocks and tool calls, and follows the router with a shared LRU expert cache during decode. Elastic memory management rebuilds the GPU expert cache at scheduler safe points under a revised VRAM budget without restarting the engine or reloading the host pool.

On an RTX 5090, FreeToken sustained 77–83 tokens per second on Qwen3.6-35B-A3B in BF16 and 22–25 tokens per second on DeepSeek-V4-Flash in MXFP4, or 1.5–2.3 times the strongest baseline, according to the report. Decode stayed within 12% of the single-turn rate across three agentic workloads. Worst-case time-to-first-token remained below 44 seconds in every test cell, while llama.cpp reached 232 seconds, Ollama 179 seconds and KTransformers 946 seconds somewhere in the matrix.

FreeToken is Apache-2.0 licensed on GitHub, published on PyPI as freetoken v0.1.2, and available as a one-click desktop app for Windows and Linux at flashml.ai. The CLI targets Linux x86_64 with an NVIDIA GPU on driver r580+ (CUDA 13). ft serve exposes OpenAI- and Anthropic-compatible endpoints on port 1919, and ft launch claude wires up Claude Code, Codex, OpenCode or OpenClaw against a local box.

The team said FreeToken is intended for solo developers, startups and SMB engineering teams whose agent token bills already exceed the cost of a GPU they own. Enterprises should treat it as an air-gapped or regulated-workload path, not a datacenter replacement. The strongest industry fits are healthcare and legal, where data never leaves the machine, as well as defense, finance, and IP-heavy R&D. Typical applications include local coding agents, private code review, offline contract analysis, synthetic-data generation and batch evals.