RAG Testing: Retrieval Regression, Precision & Recall

RAG testing (retrieval regression testing) verifies that the updated pipeline still finds the evidence a question needs and that document changes reach final context within the agreed freshness deadline, the maximum permitted delay after an authoritative source change. Compare baseline and candidate results against versioned queries, measure precision and recall with explicit definitions, and separately check revisions, deletions, permissions, and final context.

A relevance score alone is not a release decision. In the illustrative example below, rank-sensitive context precision improves to 1.00 while recall falls to 0.67. Another test returns the expected chunk ID but fails because its text contains an obsolete policy.

Table of Content

• 1. What Should RAG Retrieval Regression Testing Verify?
• 2. Measure Context Precision and Recall Without Mixing Definitions
• 3. Test Document Updates from Source to Retrieved Context
• 4. Automate Retrieval Regression Checks in Python
• 5. Set CI Gates That Do Not Hide Important Regressions
• 6. Diagnose Failures with a Retrieval Evidence Trail
• 7. A Release Checklist for Retrieval and Document Updates

1. What Should RAG Retrieval Regression Testing Verify?

Separate retrieval quality from answer quality

Retrieval-augmented generation supplies external knowledge to a language model. Retrieval and generation are distinct parts of that workflow; evaluate them separately before assessing the complete answer.

For your regression suite, capture these boundaries:

• Boundary: Candidate retrieval | Evidence to capture: Ordered IDs, text, scores, filters, source revisions | Test question: Was the required evidence found?
• Boundary: Reranking | Evidence to capture: Reordered candidates and reranker configuration | Test question: Did useful evidence move out of the usable results?
• Boundary: Final context | Evidence to capture: Exact text and order sent to the model, after filtering, deduplication, and truncation | Test question: Did the model actually receive the evidence?

Treat these as separate result sets. Finding a passage among 50 candidates does not establish that it survived a five-passage context limit. Conversely, a wrong answer with complete, correct context calls for downstream investigation rather than an automatic retrieval diagnosis.

Without retrieval traces, report answer-level observations, not verified retrieval precision or recall.

Use separate tests for retrieval changes and document changes

Use two complementary tracks:

• Fixed-corpus regression: Compare baseline and candidate on the same source snapshot and reviewed labels. Record embedding, chunking, query-rewriting, search, reranker, filter, and context-budget configurations. Identify the intentional change; hold unrelated settings constant.
• Document-lifecycle regression: Hold the retrieval configuration constant while adding, changing, withdrawing, or deleting controlled documents. Evaluate each transition against its intended state, not yesterday’s answer.

For prompt changes, keep a separate behavioral suite, as discussed in Codoid’s Prompt Regression Testing guide.

Extend that foundation with an explicit test contract:

Define relevant evidence, not just expected document IDs

• Dataset field: Query and identity | What to record: Query ID, exact question, tenant, role, language, and evaluation time
• Dataset field: Reference evidence | What to record: Required facts, acceptable supporting passages, and relevance judgments
• Dataset field: Source state | What to record: Logical document ID, revision, effective interval, and content digest
• Dataset field: Expected behavior | What to record: Answerability, mandatory evidence, forbidden revisions, and access restrictions
• Dataset field: Evaluation contract | What to record: Cutoff, metric definitions, duplicate handling, and acceptance rules
• Dataset field: Reproducibility | What to record: Dataset, corpus, parser, chunker, embedding, and retriever configuration versions

Keep a manifest, an inventory of expected chunk IDs, revisions, and text. Keep logical evidence separate from those physical IDs: after rechunking, “receipt required” may move from one chunk to two. Revalidate its supporting passages instead of preserving obsolete IDs.

When several documents independently support the same fact, distinguish document discovery from sufficient evidence. Document recall may require finding all labeled documents; claim coverage can credit one valid supporting alternative. Define alternatives explicitly rather than letting overlapping chunks multiply the reference denominator.

Cover ordinary, difficult, and unanswerable queries

Include single-document lookups, multi-document questions, exact product identifiers, paraphrases, near-duplicate distractors, current and historical questions, restricted records, and genuinely unsupported questions. Add previously observed failures with sanitized inputs.

For an unsupported question, specify whether the application should retrieve related material and abstain, or return no context. These are different contracts. Do not require empty retrieval for every unanswerable question.

Keep conversational-memory tests separate; Codoid’s AI Agent Memory Testing article addresses that adjacent boundary.

Review labels and version the evaluation contract

Have a domain reviewer approve required facts and relevance labels. For larger corpora, review pooled baseline and candidate results and disclose incomplete judgments: recall against an assessed pool is not exhaustive corpus recall.

In the demonstration, an unjudged result makes the evaluation invalid, rather than silently counting as relevant or irrelevant. Have reviewers label it, version the dataset, and rerun both configurations. This is a deliberate policy choice; evaluation tools expose different unlabeled-result conventions.

Use separate development and held-out release sets; dataset splits can support this separation. Tune on development cases, freeze the candidate, then run the reserved set at release checkpoints. Require domain approval for label changes. When a held-out failure becomes a tuning case, move it to development and refresh the reserved set rather than repeatedly tuning against it.

2. Measure Context Precision and Recall Without Mixing Definitions

Calculate precision@k and recall@k

For unique, judged results, precision measures how many retrieved items are relevant, while recall measures how much of the relevant reference set was retrieved. The demonstration uses:

Precision@k = relevant result positions within the first k / k

Recall@k = distinct relevant IDs found within the first k / total relevant reference IDs

Here, missing positions count against precision: returning two relevant items at k=5 gives precision = 5 = 0.40. A separate “precision among returned items” would be 1.00. State the denominator instead of comparing these as identical metrics.

Specify the unit, documents, chunks, or evidence units, and the pipeline stage. Do not compare document-level baseline recall with chunk-level candidate recall. If the relevant reference set contains more than k chunks, perfect ID recall is impossible at that cutoff; avoid unreachable acceptance thresholds.

Distinguish rank-sensitive context precision from ID precision

Ragas documents a rank-sensitive context-precision calculation:

Rank-sensitive context precision = sum of [precision at rank i x relevance at i] / number of relevant retrieved items

Its ID-based precision instead compares matching context IDs with retrieved context IDs. These measures are not interchangeable.

Consider a fully labeled, fictional corpus. A, B, and C are the three relevant chunks; X, Y, and Z are irrelevant. All results below use k=5:

• Retrieval order: Baseline (A, X, B, Y, C) | Precision@5: 0.60 | Recall@5: 1.00 | Rank-sensitive context precision: 0.75
• Retrieval order: Candidate (A, B, X, Y, Z) | Precision@5: 0.40 | Recall@5: 0.667 | Rank-sensitive context precision: 1.00
• Retrieval order: Complete and reordered (A, B, C, X, Y) | Precision@5: 0.60 | Recall@5: 1.00 | Rank-sensitive context precision: 1.00

For the baseline, the rank-sensitive calculation is (1 + 2/3 + 3/5) / 3. The candidate puts its two relevant chunks first, but loses C. Its perfect ranking score does not establish completeness or freedom from irrelevant trailing context.

Add normalized discounted cumulative gain when graded relevance matters, or mean reciprocal rank when first-hit placement matters; neither replaces a completeness check.

Measure reference-claim coverage separately

Ragas’s claim-based context recall checks how many reference-answer claims the retrieved context supports. Its ID-based recall instead checks reference IDs found among retrieved IDs.

For a reference requiring “14-day window,” “receipt required,” and “unopened items,” context supporting only the first two has claim coverage of 2/3. This is a calculation over the example’s reviewed claims, not a measured model result.

Require support from the supplied text, not the evaluator’s background knowledge. Review partial support, negation, exceptions, and conflicting revisions. Matching a document ID alone cannot establish that the needed sentence survived context assembly.

Handle duplicates, empty results, and missing references explicitly

Use declared conventions. Duplicate chunk IDs fail this suite’s gate; recall counts each reference ID once. For an answerable query, no relevant item in the evaluated prefix means rank-sensitive context precision is 0, including nonempty all-irrelevant results. Precision and recall are also 0. Empty retrieval follows the same rule; genuinely unanswerable cases remain N/A.

The companion’s all_irrelevant.json fixture returns X, Y, and Z for current_policy: three inspected results, all judged irrelevant, and three zero metric scores. Missing labels on an answerable case are invalid data, not an unanswerable case. Report scored and N/A case counts alongside averages.

3. Test Document Updates from Source to Retrieved Context

Validate additions, edits, rechunking, and deletion

Use isolated fixtures, restoring the initial state before each independent scenario. The following is a proposed lifecycle matrix, not a claim about a particular vendor’s guarantees.

• Change and precondition: New document; query initially unsupported | Action: Add a document containing the missing fact | Required assertion and evidence: New evidence becomes retrievable by the agreed deadline; retain source and query traces
• Change and precondition: Existing policy revision | Action: Change one fact; preserve a second supporting document | Required assertion and evidence: Retrieve the new fact plus unchanged support; verify text and revision
• Change and precondition: Document becomes shorter | Action: Remove a section and rechunk | Required assertion and evidence: New chunk manifest matches the intended state; obsolete chunks are absent where removal is required
• Change and precondition: Explicit document deletion | Action: Delete the source and trigger its removal workflow | Required assertion and evidence: Enumerate remaining records and check known old IDs; verify caches and retrieval after completion
• Change and precondition: Metadata or permission change | Action: Revoke access or change a filter attribute | Required assertion and evidence: Unauthorized users receive no protected context; authorized controls still work
• Change and precondition: Partial ingestion | Action: Fail one item or processing stage | Required assertion and evidence: Detect incomplete state; recovery produces the full intended manifest
• Change and precondition: Duplicate or reordered event | Action: Replay an event; deliver an older revision last | Required assertion and evidence: No duplicate records or unintended return to older content
• Change and precondition: Approved rollback | Action: Restore the authorized release state | Required assertion and evidence: Queries, manifests, caches, and active routing match the rollback contract

For Azure Storage indexes, change detection and deletion detection have different requirements. Microsoft warns that missing deletion handling can leave orphaned search documents; enabling a policy later does not automatically repair previously missed deletions.

Test effective dates, permissions, and cached results

Use a fictional support policy: v1 allows returns within 30 days; v2 changes the window to 14 days at October 1, 2026, 00:00 UTC. A separate receipt requirement stays unchanged.

Test immediately before, at, and after that boundary. Current questions must follow v2 after activation. Historical questions may use v1 only where the product permits historical retrieval. Explicit deletion is a separate contract; do not retain historical access where removal forbids it.

Warm the application cache before the update, then repeat queries through the normal user path. Check fresh sessions and restart recovery. For permission revocation, use both authorized and unauthorized identities, including equivalent cached questions. Do not assume an indexing freshness allowance authorizes temporary data exposure.

Exercise partial failures, retries, concurrent updates, and rollback

Inject a failure after one new chunk is written but before another is written or an old chunk is removed. Inspect item-level outcomes, retry according to the documented error policy, and compare the recovered records with the intended manifest.

Azure AI Search can return HTTP 207 for partially unsuccessful indexing requests. It also documents no ordering guarantee for actions inside a request body. These are Azure-specific reasons to inspect individual outcomes and avoid assuming request order determines final state.

Replay v1 after v2 to test obsolete-event handling. Require the application’s revision rule to win, not whichever event arrives last. Test readers during an update against the product’s declared consistency contract. A rollback must restore an approved state, not blindly revive a policy that is no longer valid.

Verify readiness and enforce a freshness deadline

Measure source acceptance, indexing visibility, correct retrieval, and correct final context separately.

For each update, record the authoritative change time, the deadline, attempts, first observed correct result, and timeout outcome. Use bounded polling with a monotonic timer for elapsed duration. Preserve the observed propagation lag; retries must not erase a missed deadline. Recheck after readiness to catch regression or cache repopulation.

Pinecone documents eventual consistency. For its serverless Database API, a query’s indexed log sequence number can confirm that a write has been reflected. It does not prove that later writes preserved the intended record or that ranking returned the needed evidence.

Assert that required new text is present and prohibited old text is absent. Enumerate all relevant pages of records and inspect known old IDs; a single top-k miss proves only that the item was not returned. Pinecone’s serverless listing functionality supports pagination and ID-prefix filtering, subject to the selected API.

If record enumeration is unavailable, document that limit rather than claiming complete deletion verification.

4. Automate Retrieval Regression Checks in Python

Define a portable fixture and retrieval-result format

What you can run: an offline scorer replay over synthetic fixtures, without a database, credentials, model calls, or third-party packages. Live ingestion, freshness, physical deletion, cache behavior, full authorization, and claim-based judging require separate adapters. Tested on Python 3.13.5; Python 3.11+ is the intended compatibility target, not a verified support matrix.

Companion release v1.1.0: Download the Python scorer, fixtures, tests, and captured reports.

Extract the archive, open its rag_retrieval_regression_v1_1_0 directory, and run the commands below from there. The adjacent .sha256 file identifies the archive; RELEASE.json identifies its code and fixtures. The synthetic replay has no live-service dependency.

dataset.json holds six queries, binary judgments, required IDs, forbidden revisions, thresholds, and the trusted manifest. Each observed hit includes chunk_id, doc_id, version, tenant, and actual text. Run metadata identifies the corpus, dataset, contract, stage, and retrieval configuration.

This is the complete content_integrity case in dataset.json (a case object, not a standalone dataset):

The manifest expects A to contain “Returns are accepted within 14 days of delivery.” The actual synthetic hit in candidate.json instead carries old text under the current ID and revision:

Do not manufacture returned text from the expected manifest when adapting the example to a live retriever. The manifest is the reference; the trace is the observation.

Calculate metrics and assert document-state requirements

The bundled scorer rejects incomplete runs, unknown or unjudged IDs, malformed records, contradictory references, and incomparable snapshots. It calculates metrics at the declared cutoff, checks mandatory evidence within that cutoff, and checks revision, content, tenant, and duplicate failures across every captured hit.

Entry point for the companion scorer – run_regression.py. This file imports the complete scorer in rag_regression.py and preserves its failure status:

# Save beside rag_regression.py and the supplied JSON fixtures.
import json
import sys
from pathlib import Path
from rag_regression import compare, load

root = Path(__file__).resolve().parent
try:
    report = compare(
        load(root / "dataset.json"),
        load(root / "baseline.json"),
        load(root / "candidate.json"),
    )
except (OSError, ValueError, TypeError, KeyError) as error:
    print(json.dumps({"status": "INVALID", "error": str(error)}))
    sys.exit(2)

print(json.dumps(report, indent=2, allow_nan=False))
sys.exit(0 if report["status"] == "PASS" else 1)
  

The implementation compares the exact returned text’s SHA-256 digest with the trusted fixture text. This detects old content presented under an expected ID or revision. It is an integrity check, not semantic relevance judgment, proof of correct embeddings, or a substitute for real authorization tests.

For controlled rechunking or text normalization, rebuild and review the manifest. For embedding updates, retain the embedding-model identifier and input-content digest, then rerun retrieval cases; matching text metadata alone does not verify the vector was regenerated.

Run self-tests and distinguish replay from live integration

For the expected-failure demonstration, run the verifier rather than pasting failing commands into one fail-fast shell sequence:

python verify_demo.py
  

The verifier checks checksums, nonzero test discovery, each expected exit, and report contents; it writes a new verification/ directory. Its exit 0 means the demonstration behaved as specified, including the intentional failures, not that its defective candidate is safe to release.

• Check: Self-tests | Expected exit: 0 | Expected result: 49 test methods pass
• Check: Defective candidate | Expected exit: 1 | Expected result: FAIL; five newly failing queries
• Check: Recovered candidate | Expected exit: 0 | Expected result: PASS
• Check: Incomplete input | Expected exit: 2 | Expected result: INVALID
• Check: All-irrelevant fixture | Expected exit: 1 | Expected result: FAIL; current-policy metrics are 0
• Check: run_regression.py | Expected exit: 1 | Expected result: Same intentional candidate failure

Individual commands and their exact expected outcomes are in the README. The verifier deliberately handles exits 1 and 2; never replace those checks with a blanket failure suppression such as || true.

Release gating is different. CI means continuous integration. Supply real captured traces and run the separate gate as a dedicated failing-on-error CI step:

python ci_gate.py dataset.json baseline.json captured_candidate.json \
  --report artifacts/retrieval.json
  

captured_candidate.json is your adapter’s output, not a supplied fixture. The gate saves its report and returns 0 for PASS, 1 for FAIL, or 2 for INVALID. Missing input or report-write failure also fails. Configure CI to preserve reports on failed steps and propagate every nonzero exit; the package does not configure your CI service.

A clean extraction of release v1.1.0 was executed locally on Python 3.13.5: 49 tests passed, including the original 40 and nine review-driven tests. Candidate, recovered, invalid, and all-irrelevant runs produced the expected exits. The candidate’s five failures cover missing evidence, stale text, a forbidden revision, a duplicate chunk, and unauthorized tenant content.

For content_integrity, this is the complete candidate score object from the captured evidence/candidate.stdout.json report (queries entry with that query_id):

The correct chunk ID gives recall 1.0, but content_mismatch:A fails the query. This is executed synthetic replay evidence, not a live freshness measurement. The README defines the remaining adapter work.

5. Set CI Gates That Do Not Hide Important Regressions

Separate hard failures from quality tolerances

Use deterministic rules for observable contracts; LangSmith similarly distinguishes code evaluators from model-based judgments. Suggested release policy:

Hard failures: unauthorized final context, prohibited revisions after their applicable deadline, missing mandatory evidence, invalid evaluation input, incomplete ingestion, or a failed recovery contract.

Quality tolerances: agreed precision/recall floors and permitted changes relative to a qualified baseline. Choose these by task and risk, not an invented universal target. The sample’s per-query thresholds and zero-drop tolerance are fixture settings, not production recommendations.

Define operational measures before a live run. Use a fixed observation window and a prescribed query schedule; do not let ad hoc retries change the denominators.

Stale-context rate: among completed, inspectable scheduled current-policy probes from the update deadline through the fixed window end, count probes containing any prohibited old revision. An empty returned context is inspectable but missing evidence, not stale evidence. Exclude historical queries from this current-policy population. Report missing evidence separately, and record timeouts or absent traces as observation failures. A zero observed-stale rate cannot pass an incomplete-observation gate.

On-time update completion: count each initiated logical transition once, using a unique transition ID assigned before the update. Among transitions whose deadlines have elapsed by the reporting cutoff, divide those that satisfied all required state and retrieval checks by that entire cohort. Ingestion failures and timeouts stay in the denominator; pending deadlines are reported separately. Late recovery does not rewrite an on-time failure. Report retry attempts and time to first verified correct state separately.

For the existing fictional policy edit, an unexecuted test specification could use transition T-001, source change 09:00 UTC, deadline 09:01, and scheduled probes at 09:01, 09:02, and 09:03. Its disposition is NOT RUN until actual context, timestamps, and checks are recorded. These are chosen test inputs, not measured propagation times or a recommended service-level target.

Compare paired queries and important slices

A paired comparison evaluates the same queries under both configurations. Keep identity, time, reference state, and metric definition equivalent. Report per-query changes, newly failing cases, and slices such as current policy, historical policy, permissions, and unsupported questions.

An overall average must not compensate for a prohibited disclosure or lost critical fact. The sample reports a macro-average, the unweighted mean across scored queries, with its contributing count. A passing candidate does not qualify a failing baseline.

For lifecycle tests, compare each state with its own approved expectations. Do not interpret the correct replacement of yesterday’s fact as a regression.

Control evaluator and retrieval variability

Pin evaluator model, rubric, prompt, and dataset versions. Keep human-reviewed calibration cases. Repeat unstable retrieval or model-judged cases and report variability before treating small score differences as meaningful.

For a larger noisy suite, predeclare a paired uncertainty method. One option is 10,000 bootstrap resamples of query pairs with replacement, computing the mean candidate-minus-baseline score for each resample and a 95% percentile interval; fix the seed and compare the interval with the agreed tolerance. Paired resampling preserves matching observations across the two configurations. Aggregate repetitions within queries first; group related paraphrases when queries are not independent. This is a recommended analysis, not implemented in the six-query replay. Deterministic security failures do not wait for statistical significance.

Offline evaluation supports comparing application versions on curated datasets; the release decision still depends on your acceptance contract. Changing the judge and retriever simultaneously should be an explicit experiment, not an unexplained baseline replacement.

6. Diagnose Failures with a Retrieval Evidence Trail

Locate the failing stage

Use the following as investigation hypotheses, not automatic root-cause verdicts:

• Observation: Fact missing from parsed text | Inspect next: Source export, parser, table extraction, and chunk boundaries
• Observation: Correct text absent from stored records | Inspect next: Ingestion item failures, namespace, and index routing
• Observation: Stored evidence missing from candidates | Inspect next: Query rewriting, filters, embeddings, and search configuration
• Observation: Candidate present but reranked out | Inspect next: Reranker inputs, model version, and cutoff
• Observation: Ranked evidence absent from final context | Inspect next: Deduplication, context packing, truncation, and token budget
• Observation: Correct ID but obsolete text | Inspect next: Cached payload, stale revision, or incorrectly relabeled record
• Observation: Correct context but incorrect answer | Inspect next: Generation prompt, conflict resolution, or answer evaluator

Preserve artifacts needed to reproduce the regression

Retain query and identity scope, evaluation time, ordered results at each stage, source revisions, actual context, filters, model/configuration versions, ingestion outcomes, readiness observations, metric inputs, and failure reasons.

Protect traces as potentially sensitive data. Store only authorized content, redact unrelated secrets, restrict report access, and retain enough provenance to reproduce the failure without exposing confidential documents.

7. A Release Checklist for Retrieval and Document Updates

Before release, verify these six checks:

• Corpus and labels are versioned, reviewed, and separated into development and reserved release cases.
• Metric units, cutoffs, duplicate handling, and zero/N/A conventions are explicit.
• Required evidence reaches the exact final context supplied to the model.
• Updates and removals meet their deadlines; timeouts and late recovery remain visible.
• Permissions, tenant boundaries, and relevant cache states have independent checks.
• Reports preserve enough inputs, versions, traces, and failure details to reproduce the result.

Retrieval regression testing is strongest when relevance metrics and document-state assertions reinforce each other. One explains whether the system found useful evidence; the other verifies that the evidence is current, permitted, and consistent with the intended source state.

CONCLUSION

Effective RAG testing requires more than a single relevance score. By separating retrieval quality from answer quality, measuring precision and recall with explicit definitions, and independently verifying document updates, you can build a defensible regression suite. This approach ensures that your RAG pipeline not only finds useful evidence but also that the evidence is current, permitted, and correctly assembled into the final context.

Codoid’s LLM & AI Application Testing Services include evaluation datasets and retrieval-quality testing for teams building this coverage.

Ready to Stabilize Your RAG Pipeline?

Talk to an AI QA Expert

Frequently Asked Questions