VocabX Cuts Measured Inference Cost 77% by Specializing Small-Model Vocabulary

The production system is an automated grader for Portuguese essays built on Qwen3-4B. It receives the essay, prompt and supporting texts and returns a JSON assessment with a grade and comment for each competency. The model had already been trained for the task, but its tokenizer remained the original Qwen3-4B vocabulary, built for many languages and text types. In the grader, it was used for one language, Portuguese, and one response format.

That created two costs, according to the post. The text the grader writes on every request took more tokens than needed, and at every generated token the model scored a vocabulary of 151,669 entries, including many the task almost never uses. The authors rebuilt the vocabulary with VocabX while keeping the same base model, data and training recipe.

The post said specialized models fine-tuned from general open-source models such as Qwen and Gemma inherit vocabularies built for many domains and general applications. That makes sense for general-purpose models, whose tokenizers are trained on large collections of text from source code, contracts and Chinese, among other material. The tokenizer is usually treated as a fixed property of the base model, inherited and left alone, the post said, but for a model specialized in a single domain it can be one of the largest cost levers and is set before any training step.

For a model used in a single domain, the post argued, the inherited vocabulary creates fragmentation and excess. Fragmentation occurs when the tokenizer splits the domain’s most repeated text into many pieces. In the grader, every response starts with the same JSON envelope, competency names and evaluation phrases; in contracts, the repeated text is boilerplate clauses, and in code it is identifiers. Each sequence costs several tokens where one would do, and the model pays that cost on every request. Excess occurs because the vocabulary carries tens of thousands of entries the domain never uses, and the model computes a score for all of them at every token it generates. A model tuned to grade American school biology exams will almost never write in Chinese, the post said, yet it computes the probability of every Chinese-language token at every step.

Counting tokens and trying to reduce the count is a natural response, but the post said the count is not the bill. Inference has two phases. In prefill, the model processes the whole input in a single pass through the network; each weight read from GPU memory is reused for every input token, and the limit is compute. In decode, the model generates one token at a time, and each token needs a full pass. Each weight read is used only a few times, and the GPU spends most of its time waiting for data from memory, so the limit becomes memory bandwidth.

The effect is strongest for small models served with moderate batches, the post said. Batching requests splits weight reads across them and eases the problem but does not solve it. At every decode step, the model also reads the KV cache, the attention memory that holds already processed context and grows with it. In the measured grader, a generated token cost 7.4 times a token read, though the post said the ratio varies with hardware, the problem, batch size, context length and other factors and can be larger. The direction held: generating costs more than reading.

The vocabulary itself is another part of the bill. The model’s last layer, the LM head, turns the internal state into a score for every possible token and holds one row of parameters per vocabulary entry. In a model the size of Qwen3-4B, that matrix accounts for almost 10% of the parameters, a share the post said large models absorb without noticing. The post’s method specializes the vocabulary around the cost of each token rather than its frequency.

Editor's Summary

VocabX rebuilds a small model’s vocabulary for a single domain and, in a Portuguese essay grader built on Qwen3-4B, cut measured inference cost by 77% without changing the base model. The reported savings depend on reducing both fragmented domain text and unused vocabulary entries, with generated tokens costing more than read tokens in the measured setup. The results come from a paper still under review, so the figures may change before publication.