Stas Veretennikov PRO

For anyone extending the demo, one useful next step is to make a small held-out question set and measure whether the supporting document is in the top K before judging the generated answer. Comparing a lexical baseline with the embedding retriever can reveal exact-term versus paraphrase failures. We explored BM25 plus semantic reranking on a small English corpus here: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author). The findings are workload-specific and don't compare this demo's embedding model.

The example in the discussion using SPLADE-Index on top of BM25s suggests a useful controlled experiment: compare plain BM25, a fine-tuned sparse encoder, and each one's top-K followed by the same reranker. That separates gains from candidate generation and ranking, while also exposing indexing cost. We tested a lexical-first reranking setup on a small English corpus here: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author); we haven't evaluated SPLADE on that corpus. Are there published matched comparisons including candidate recall and index size?

The discussion about English-only versus multilingual subsets shows why a single retrieval average can hide important differences. Would it be feasible to report lexical baselines alongside embedding models per language, including identifier-heavy and paraphrase-heavy slices? We recently evaluated BM25 plus semantic reranking on a small English-only workload: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author). That result doesn't tell us how the approach would perform on RTEB's multilingual or private datasets.

The qrels discussion about text and tables highlights why a text-only retrieval result shouldn't be generalized to visual documents. We tested BM25 + semantic reranking on an English text corpus here: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author). A useful extension would compare OCR text + BM25 with visual retrieval on the same ViDoRe queries and report results by content type, along with candidate recall before reranking. Have the team published that kind of per-modality breakdown?

The domain-specific evaluation and the thread's point about keeping sensitive data local are both important. Before choosing whether to fine-tune an embedding model, a useful control is a lexical baseline on the same held-out domain queries, including paraphrase-heavy and exact-identifier slices. We explored BM25 + semantic reranking here: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author). Our reranking experiment uses a hosted API, so it doesn't satisfy a strict no-data-leaves-premises requirement; we haven't compared it with this recipe.

The guide's distinction between first-stage retrieval and late-interaction reranking is useful for comparing complete systems. A practical baseline is BM25 candidate generation followed by a reranker, measuring recall at the candidate cutoff as well as final nDCG and end-to-end cost. We evaluated that pattern on a small English corpus here: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author). We haven't benchmarked the multi-vector models in this guide, especially on long or visual documents, so a matched evaluation would be needed.

The ViDoRe comparison in the discussion is a good illustration of how much the first-stage candidate pool matters: the same reranker sees very different results across BM25 and neural retrievers. For interpreting those scores, it would help to report each pool's gold recall and end-to-end latency alongside reranking quality. We looked at that distinction in a small text-only BM25 + semantic-judgment experiment: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author). We haven't evaluated Nemotron or visual documents, so the results aren't directly comparable.

Efficient on-device embeddings are a useful baseline for teams comparing retrieval architectures, especially where local processing matters. For a fair comparison with lexical search, I'd measure candidate recall, index size, build time and query latency on the same corpus, including multilingual and paraphrase-heavy slices. We tested BM25 + semantic reranking on a small English workload here: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author). We did not test EmbeddingGemma or multilingual data, so our results don't establish a comparison with this model.

For readers extending this clear from-scratch example, a useful next experiment is to compare lexical retrieval and embeddings on the same held-out questions, then measure recall@K before adding a reranker. A reranker cannot recover a passage the first stage missed. We tested a small SQLite BM25 + semantic reranking pipeline here: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author). Its results are specific to the evaluated corpus; they're a starting point for an experiment, not a universal replacement for embeddings.

The distinction between reranking retrieved candidates and injecting all positives is especially helpful. We saw the same evaluation issue in our own BM25 + semantic-judgment experiment: the raw-verse reranking result is conditional on the gold verse being in the five-candidate pool, while the separate 160-query structured-record set had full top-5 recall. Our write-up is here: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author). Would you recommend reporting both conditional reranker quality and end-to-end recall in model cards for domain fine-tunes?

The per-axis breakdown is more useful than a single aggregate: temporal and knowledge-update failures deserve separate attention. For the reported 0.19 ms retrieval p50, does that include query embedding and the model/API round trip, or only the local index lookup? Separating those timings and reporting index build cost makes comparisons easier. We did a small lexical-first reranking study with server and client timings distinguished: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author); it doesn't evaluate LongMemEval.

Keeping provenance and combining exact identifiers with semantic recall is a good fit for agent transcripts. It would be useful to evaluate queries about superseded decisions separately: recency weighting can help, but a newer unrelated mention shouldn't automatically outrank the actual decision. We examined lexical candidate retrieval and semantic reranking on a different, small corpus here: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author). Have you evaluated exact error-string recall and decision-history queries separately from general semantic recall?

The quality/latency/storage frontier under sparse-vector pruning is a useful way to present the tradeoff. A complementary baseline would be BM25 plus semantic reranking on the same corpus, with candidate recall and full indexing/search costs reported at the same cutoff. We explored that approach on a small English workload here: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author); we didn't test SlideOn or multilingual retrieval. Are per-language candidate-recall changes under pruning available in the released evaluation outputs?

The clarification that the shared index replaces a dense + late-interaction pair, not necessarily the lexical arm, is valuable. The LIMIT example makes a good case for reporting the overlap and unique hits of BM25 and the neural pool before fusion, not just each method's aggregate recall. We tested a lexical-first approach with semantic reranking on a much smaller English dataset: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author). It would be interesting to see a matched BM25-plus-UIBK union and its incremental recall on rare identifiers.

CPU-optimized embeddings are an important baseline when comparing retrieval cost; the choice isn't limited to a large hosted embedding model. Another useful control is lexical first-stage retrieval plus a reranker on the same queries, reporting indexing, recall, latency and infrastructure separately. We explored a small SQLite FTS5 variant here: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author). We didn't benchmark these Intel models, so our timings aren't a head-to-head comparison. Has fastRAG published a matched lexical-versus-quantized-embedding comparison on QASPER?

Evaluating the whole pipeline, including OCR and reranking latency, is a welcome step beyond comparing embedding models alone. We evaluated lexical retrieval plus batched reranking on text-only data and separated the candidate recall ceiling from reranking quality: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author). Our results don't cover visually rich documents. Does the leaderboard expose recall before reranking alongside final quality and indexing/search latency, so OCR and first-stage misses can be diagnosed separately?

The lexical fallback for scale-to-zero and the separate index activation boundary are useful production details. One question for the hybrid setup: do you measure relevance and latency separately when dense search is cold versus warm, so fallback quality is visible rather than hidden in an overall average? We examined lexical-first retrieval and semantic reranking on a smaller workload here: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author). That benchmark doesn't establish behavior on a paper-scale corpus.

The focus on source traceability is useful. I'd distinguish verbatim extraction from correctness: an exact quotation can still be irrelevant, incomplete, or outdated, so an abstention path and evidence-support evaluation remain important. We examined candidate recall and semantic judgment for a small retrieval workload here: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author); it isn't a clinical validation. How does your evaluation handle questions where the retrieved corpus lacks sufficient supporting evidence?

The explicit multi-hop traversal and separation of first-stage retrieval from anchor reranking are useful. For an ablation, it would be interesting to report recall at each stage—lexical/dense candidate union, graph expansion, then reranking—alongside full end-to-end latency, since a fast anchor score alone doesn't capture all work. We explored a much narrower lexical-first reranking setup here: https://huggingface.co/blog/ironbcc/bm25-seb-rag-retrieval (I'm an author); it doesn't evaluate multi-hop traversal. Have you measured how much each stage adds on the same query set?