Keenable AI Open-Sources NEEDLE: Live Search Benchmark Rebuilds Query Set Every Hour

NEEDLE regenerates news queries hourly from RSS feeds and Google Trends. Finance, scholar, legal, and rare-entity queries are regenerated daily from SEC XBRL, arXiv, Europe PMC, CourtListener, and public agent logs. Fifteen search APIs run against the same query text under one protocol, and every score is read against "ultimate," a pooled oracle engine that marks what the entire field managed to find. The benchmark is reproducible as an open-source evaluation harness: it is a Python CLI installed with uv sync, requires an OpenRouter key for judging and one API key per engine, and can run on a laptop or in CI.

The name stands for News, Everyday, Expert, Deep-tail, and Legal Evaluation. News projects the newest item from about 124 curated RSS feeds and Google Trends into a keyword query. Finance asks registry facts from Wikidata and GLEIF plus single-quarter 10-Q figures from SEC XBRL. Scholar turns one paper into four query styles: a degraded title, a full-text-only detail, a natural-language clue, and a hedged tip-of-the-tongue description. Deep-tail samples rare-word queries from public agent-trajectory releases including DeepResearchGym, OpenResearcher and LRAT. Legal pulls recent CourtListener opinions across 14 federal courts and eCFR sections.

Scoring follows the same structure. News and deep-tail have no single correct result, so an LLM judge rates each result from 0 to 4 and the harness reports nDCG@5 with a duplicate-URL penalty. Finance reports answer-recall@5, measuring whether the fact reaches the agent inside a top-five snippet. Scholar and legal are known-item tasks scored by identifier match.

Every engine receives the same query text, and the runner issues one call at a time, making latency percentiles comparable. Judging happens on the engine's own ranking, titles and snippets; pages are never fetched and results are never re-ranked. Evidence is clipped to 2,000 characters for everyone, and the judge does not see the engine name. For each query, NEEDLE pools results from every engine into a synthetic oracle, then orders that combined set by relevance. The gap between delivered and achievable quality acts as an upper bound on current agentic search quality: a large gap means better results existed but every engine failed to surface or rank them well, while a weak ultimate score indicates a retrieval problem shared by the whole market.

Published seven-day means for the window ending Aug. 28 show finance close to solved: Exa scored 0.910, Keenable 0.872, Perplexity 0.871 and Google 0.847 against an ultimate of 0.965. Scholar spread from Keenable's 0.774 to Tavily's 0.310 against a 0.869 ceiling, because title queries are answerable from metadata and body queries are not. Deep-tail was the hardest and closest to real agent traffic: Exa led at 0.557 of ultimate, Keenable followed at 0.470, and Bing sat at 0.199. Latency, important because agents call search dozens of times per task, also varied: Keenable-realtime had a 193 ms p50 and 284 ms p95, Exa 1,876 ms and 2,955 ms, and Bing 2,767 ms and 9,381 ms.

The code is MIT-licensed, runs in public GitHub Actions, and per-run artifacts are shipped to a Hugging Face dataset.