Perplexity releases Q2D-Web benchmark for web-scale retrievers

Perplexity has released Q2D-Web, a retrieval benchmark that pairs private evaluation data with a public leaderboard. It is designed to test first-stage retrievers in agentic retrieval-augmented generation systems. Perplexity says the benchmark covers about 190 million web documents and 69,721 agent-reformulated queries in ten languages.

The benchmark examines the infrastructure beneath agentic search, rather than user-facing tools such as Perplexity’s shift from Spaces to Projects. A first-stage retriever searches the full index and selects the candidate documents that later systems rerank. That initial selection sets an agent’s evidence boundary: if a document does not enter the candidate pool, the agent cannot later read or cite it. Q2D-Web uses Recall@1000 as its primary metric to measure how much labeled relevant material reaches the pool before reranking.

One caveat comes from the benchmark’s origins. Perplexity says its embedding models may have an in-distribution advantage because Q2D-Web was derived from its production traffic, although the company says the evaluation queries and corpus were excluded from model training. It does not quantify the possible size of that advantage. Because the corpus, queries and relevance judgments are private, outside researchers cannot directly reproduce the reported scale, labeling quality, costs or scores from public materials.

According to the technical report, the query set was built from roughly 23,000 production searches collected over nine months. It includes 12,365 primary queries that restate a user’s intent and 57,356 support queries generated by agents seeking background, alternative phrasings or adjacent information. Users did not enter these benchmark queries directly. Agents generated them from user messages, conversation history and results already retrieved during a search. Perplexity says it used traffic from users who permitted data use and filtered out queries containing personally identifiable information.

For each query, Perplexity says its production retrieval system collected the top 5,000 documents. The company combined those results and removed near-duplicates to create the approximately 190-million-document corpus. This is not a random sample of the web: every document had been considered a plausible match for at least one benchmark query, concentrating semantically similar distractors in the collection.

Q2D-Web provides three relevance-judgment sets. One treats documents cited by an agent as relevant, another uses results from Perplexity’s production web-ranking system, and a combined set adds LLM judgments for previously unjudged pooled candidates. Perplexity reports an average of 99.6 positive judgments per query in the combined set, while noting that citations, production rankings and LLM judgments each introduce their own biases.

The company evaluated 13 lexical, dense and late-interaction retrievers. On the full corpus, Perplexity reports that pplx-embed-v1-4b led Recall@1000 on the Web Ranking set with 65.73 and on the combined set with 69.11. Nemotron-3-Embed-8B led the Citation set with 61.68. No model led every judgment set or metric: Nemotron recorded a combined nDCG@10 score of 47.44, compared with 45.84 for pplx-embed-v1-4b. These are Perplexity-produced benchmark measurements and have not been independently reproduced.

Perplexity reports that one full-corpus evaluation of pplx-embed-v1-4b required 4,608 H200 GPU-hours, while EmbeddingGemma-300M required nearly 200 H200 GPU-hours. The company says a reciprocal-rank-fusion subcorpus retaining 31.7% of documents preserved the tested models’ ordering and reduced the 4-billion-parameter model’s run to roughly 1,500 H200 GPU-hours. However, sampled-corpus Recall@1000 scores were three to seven points higher than full-corpus scores.

The evaluation encoded only the first 512 tokens of each document. The paper says most document bodies were therefore unseen and does not claim that the scores are unbiased estimates of full-document retrieval effectiveness. Perplexity is keeping the underlying data private to limit benchmark contamination while accepting open-weight model submissions through the public Q2D-Web leaderboard.