Perplexity Open Sources Lily, a Rust+Metal Inference Engine for Apple Silicon

The release makes Lily available as a standalone demo in the pplx-garden repository, offering greedy text generation through a minimal OpenAI-compatible HTTP API. The engine is deliberately narrow, supporting one model, Qwen3.6-35B-A3B, on one hardware family. It is a single-process runtime in which a Rust layer loads the checkpoint and drives the generation loop, while hand-written Metal kernels execute the model. The 4-bit checkpoint is 19.4 GB, so a Mac with at least 32 GB of unified memory is a realistic deployment floor; Perplexity's Hybrid Compute product lists macOS 15+ with a 24 GB minimum and 32 GB for best results.

Why this specialization? The report says the default Mac stack is MLX plus MLX-LM, which already implements Qwen with grouped expert work, a fused recurrent Metal kernel, and GQA-aware attention. However, its operations must stay reusable across architectures. Lily gives up that generality and puts model structure, execution plans and kernel selection in one runtime. The model stores 35 billion parameters but activates roughly 3 billion per token; a router scores 256 experts and picks eight, alongside one shared expert that sees every token. It mixes 10 full-attention layers using grouped-query attention with 30 Gated DeltaNet layers.

According to MarkTechPost, on a 40-core, 128 GB M5 Max at batch 1, using identical 4-bit checkpoint bytes across ten lengths from 256 to 128K tokens, Lily averaged 4,156 prefill tokens per second versus 3,388 for MLX-LM, and 170.0 decode tokens per second versus 126.4. That translates to 1.23x faster prefill and 1.35x faster decode on average. At a 4K prompt and 4K context, Lily reached 5,749.9 and 186.6 tokens per second against 4,737.5 and 140.9, and it was faster at every recorded point: 1.12 to 1.42x prefill and 1.31 to 1.37x decode. A teacher-forced check across 192 positions put Lily's perplexity 0.04 percent higher, with the same top-ranked token 96.35 percent of the time.

Perplexity's ablation results attribute the largest prefill gains to fused dequantization inside the grouped GEMM, which raised end-to-end prefill by 77.4 percent at a 512-token prompt, and to keeping the routing histogram, prefix scan, scatter and block map inside a single GPU command buffer, which removed CPU synchronization inside each MoE layer and added 89 percent at 512 tokens. Moving from 16-row to 32-row tiles with four simdgroups added 13.2 percent at 2K tokens, and a register-resident Gated DeltaNet scan added 5.6 percent. For decode, coalesced cache reads lifted key bandwidth from 33.8 to 47.9 GB/s and value bandwidth from 42.0 to 61.8 GB/s, while GQA packing improved decode by 23.8 percent at 32K context and a fixed-block attention layout improved decode by 40.2 percent at 128K context. The concurrent Metal pass records real dependencies so independent kernels overlap, and the selected token is written directly into the next step's GPU-resident input slot, removing a per-token CPU round trip.

Lily's design shows that sacrificing portability can produce substantial speedups for a single model on Apple silicon. The open source release is available in the pplx-garden repository, with further technical details linked from MarkTechPost's article.