Haystack Eval Bake-off Runbook
The Haystack retrieval pipeline ships with a deterministic RAGAS smoke test that runs against a bundled fixture. A full bake-off harness with real client documents and side-by-side baseline comparison is deferred until Phase 2 real-document ingestion lands.
Smoke test (today)
The smoke test verifies that the RAGAS evaluation wiring (faithfulness + answer_relevancy) works end-to-end against deterministic stub LLM and embedding implementations. No external API calls, no real models — fast feedback that the harness itself is healthy.
Install
cd rag
pip install -e .[eval]
This pulls ragas, datasets, and the LangChain shims required by the harness (rag/pyproject.toml optional-dependency group eval).
Run
cd rag
python evals/ragas_smoke.py
This matches the CI invocation in .github/workflows/test-rag.yml. The script loads rag/evals/fixtures/ragas_smoke.json, runs ragas.evaluate with deterministic LLM + embeddings, and prints the per-metric scores for faithfulness and answer_relevancy.
When to run
- After any change to the retrieval pipeline that could shift response/context shape.
- Before promoting an OpenRouter model change to staging or prod.
- In CI: wired into the
test-ragworkflow (.github/workflows/test-rag.yml).
Full bake-off (deferred)
Real-document bake-off harness — RAGAS scoring across faithfulness, context_precision, context_recall, answer_relevancy against a curated query set, with baseline-delta comparison — is not yet implemented. The smoke test above is the only RAGAS surface in rag/evals/ today.
Trigger conditions for building the full harness:
- Paul ships real sanitized client documents post Phase 2.
- An embedder or reranker swap needs side-by-side validation.
When built, the harness will live in rag/evals/ next to ragas_smoke.py and is expected to read a docs corpus and a queries JSONL, then emit per-query scores plus baseline deltas. Update this runbook with the exact invocation once the script lands.