Meta FAIR and UW Researchers Distill Models at the Byte Level, Predicting a Higher Accuracy Ceiling

Standard distillation has a large teacher model pass its probability judgments over the next token to a smaller student. Llama 3-8B, the teacher used in these experiments, has a vocabulary of 128,256 tokens, so every prediction position involves more than a hundred thousand candidate probabilities, and offline distillation usually keeps only the top-k of them to limit storage. A byte has 256 possible values, and that set does not change with the tokenizer, making the full distribution easier to retain.

The obstacle is that the teacher predicts whole tokens while the student predicts single bytes. The team's two methods, Marginalize-It and End-Of-Token, both decompose the teacher's token probabilities onto the corresponding byte positions, and both need only one teacher forward pass. Taking the word "Tiramisu" split into T, iram and isu as an example, candidates such as isu, isk and is all begin with the byte i, so their probabilities are summed for the first byte; once the true prefix is fixed, only matching candidates carry forward.

Marginalize-It aggregates and renormalizes probabilities directly. When candidates of different lengths run out, with the token "is" ending while isu and isk continue, it drops the finished candidates and renormalizes the rest. That is fast but approximate, and it discards probability information from ended tokens. End-Of-Token instead attaches a special symbol to the end of each token, so ending becomes a prediction position in its own right and the stranded probability has somewhere to go.

The experiments compared token, plain byte and end-of-token byte representations, each under supervised training and under distillation, six settings in total. All students shared the same Transformer layer count of about 1.28 billion parameters; counting vocabulary-related parameters, the token student totaled about 1.81 billion and the byte student about 1.28 billion. Training scaled to roughly one trillion bytes, with eight evaluations covering multiple-choice question answering, language generation and machine translation.

Token models learned faster early but quickly approached a plateau, while byte models started slower and kept improving as compute grew. The team did not judge by bits per byte alone, because byte models surpassed token models on that metric early while still trailing on downstream tasks. Instead they fitted training compute to bits per byte to downstream score, then extrapolated. Predicted average accuracy ceilings were 48.4 percent for token distillation, 50.5 percent for Marginalize-It and 52.4 percent for End-Of-Token, the highest of the six settings. A plain byte supervised model was predicted at 51.2 percent, above Marginalize-It distillation, which the paper attributes to the probability information that method discards.

End-Of-Token was projected to match token distillation's 48.4 percent ceiling at around 6.33x10^22 FLOPs. The paper also reports storage advantages: under its predicted compute budget, End-Of-Token would handle about one-sixth the text of the token approach, and with tokens keeping only the top-600 probabilities while bytes keep full distributions, storage would be roughly one-fifth.

Less data does not mean less computation. Because End-Of-Token inserts a marker after every token, training compute for the same amount of text is about 30.94 percent higher than for a plain byte model. The authors have not completed a comparison at equal inference cost, so whether the approach pays off in deployment remains untested.

The authors include first author Kalyani Marathe, a University of Washington doctoral student advised by Luke Zettlemoyer; Artidoro Pagnoni, a co-author of Byte Latent Transformer and QLoRA; Tomasz Limisiewicz; Margaret Li; Mike Lewis, a Llama 3 pretraining lead at Meta FAIR; Luke Zettlemoyer, a UW Allen School professor and head of Meta FAIR Seattle; and Srinivasan Iyer. The paper is posted on arXiv.