Liquid AI Open-Sources Pipette to Benchmark On-Device AI Models

The first verified results came from a MacBook Pro with M5 Max, an iPhone 17 Pro and a Galaxy S26 Ultra, with AMD Ryzen AI Max+ 395 and Radeon 8060S results listed as coming soon. Pipette measures five on-device performance metrics, while quality is tracked separately on IFBench, GPQA Diamond and MATH-500. Those quality scores currently come from llama.cpp evaluation runs on NVIDIA H100 80GB reference systems and are matched to on-device runs sharing the same model and quantization — meaning a quality score shown next to phone throughput was not produced on the phone.

The report highlights four published comparisons that show how far a configuration can move a decision. At Q4_K_M on Galaxy S26 Ultra, Granite-4.0-H-350M retains 78.4% of its decode throughput from 256 to 4,096 input tokens, while Granite-4.0-350M retains only 33.8% — a demonstration that identical parameter counts can diverge sharply depending on context scaling. On the same phone, LFM2.5-8B-A1B decodes 2.4 times faster than Qwen3.5-4B and 2.6 times faster than Ministral-3-3B-Instruct-2512 at 2,048 input tokens, yet its sparse activation still peaks at 5.29 GiB because all expert weights occupy memory. On iPhone 17 Pro at Q4_K_M, MiniCPM5-1B completes a 2,048-in / 256-out workload in 3.47 seconds versus 4.12 seconds for LFM2.5-1.2B-Instruct, a 15.8% reduction in elapsed time, while LFM scores 9.0 points higher on MATH-500 on the same artifacts. Near-identical system profiles can also hide task-level reversals: on M5 Max at Q4_K_M with 2,048 input tokens, Granite-4.1-8B and Ministral-3-8B-Instruct-2512 differ by only 2.4% in decode throughput and 1.2% in peak RAM, yet Granite leads IFBench by 7.3 points while Ministral leads GPQA Diamond by 14.0 points.

Pipette ships as Apache 2.0 infrastructure, including pipette-mgmt, pipette-clients and pipette-scores, along with a public results dataset, a hosted dashboard and native iOS and Android benchmark apps. Community-submitted results are still in beta. The tooling is intended for teams shipping models onto hardware they do not own; solo developers can use the dashboard and apps without infrastructure, while mid-market product teams can run clients across an internal device fleet, and large OEMs, chip vendors and enterprises can operate the entire pipeline behind their own firewall.

Performance runs follow a published methodology: fixed token shapes, greedy decoding, a discarded warm-up, five measured repetitions and readiness gating. Before each timed repetition, a platform-specific check verifies thermal and load conditions; failing runs are excluded.