AI Test Cases: How to Evaluate Before Trusting Them

AI test cases can compile, pass in continuous integration, and still verify the wrong behavior. That gap exists because research on generated test oracles, the logic that decides whether a result is correct, has found that models can reproduce what a program currently does rather than what it is supposed to do. A passing test may therefore preserve a defect instead of detecting it. Execution success and testing value are also different outcomes. In a 2024 evaluation of Meta’s TestGen-LLM on Instagram’s Reels and Stories products, 75% of generated test cases built correctly, 57% passed reliably, and 25% increased coverage. Those are results from a particular system and evaluation, not expected success rates for every generator, but they illustrate why AI test cases need several independent checks.

Treat AI test cases as proposed changes to your verification system, not as trusted evidence merely because they look plausible.

How do you evaluate AI test cases before using them?

A practical acceptance framework should answer four questions: does the test check the intended behavior with a justified expectation (accuracy), which important behaviors does it actually exercise and verify (coverage), does it add useful verification beyond existing tests (redundancy), and can it run safely and reproducibly (executability). Our LLM testing services team applies this same four-dimension framework when qualifying AI test cases for clients.

These dimensions should remain separate. High coverage cannot compensate for an incorrect expected result. Reliable execution cannot justify a test that asserts nothing meaningful.

Key takeaways

• A passing AI-generated test is not proof of correctness; the oracle behind the assertion needs independent review.
• Build your scenario inventory from the approved contract first, not from what the generator happened to produce.
• Redundancy is a semantic question, not a text-similarity question; two differently worded tests can check the same thing, and two similar-looking tests can protect different boundaries.
• Executability includes discovery, setup, teardown, and stability under realistic conditions, not just “the test passed once.”
• Score accuracy, coverage, redundancy, and executability separately. Do not collapse them into one AI-test-quality score.

Start With an Approved Contract

Before evaluating generated output, define what the tests are allowed to treat as authoritative.

That usually means a versioned combination of acceptance criteria, API contracts, business rules, approved examples, and architectural constraints. Use implementation code to understand interfaces and execution paths, but do not automatically treat its current output as the correct answer.

Consider a shipping service with the following illustrative contract:

Assume the application accepts integer subtotals and exposes:

shipping_fee(subtotal_cents: int, *, premium: bool) -> int

This contract gives reviewers something concrete to evaluate. It also identifies what is not specified. For example, behavior for strings, floating-point inputs, or None needs clarification rather than an invented expectation.

Keep the generation context alongside the candidates: the requirement revision, source commit, existing-suite revision, model and prompt versions, and any supplied examples. Record unresolved assumptions explicitly.

For manual test cases, require concrete preconditions, test data, actions, expected results, and cleanup instructions. “Verify that shipping works correctly” is not an executable procedure.

For automated cases, require the same information, with a clear mapping to fixtures, application calls, and assertions.

1. Accuracy: Does the Test Verify the Right Behavior?

Accuracy begins with the test oracle, not the test name.

A scenario titled test_free_shipping_threshold might look relevant while asserting the wrong threshold behavior. Review the relationship between its preconditions, inputs, actions, and expected results.

Validate Expectations Independently

Under the shipping contract, this generated assertion is incorrect:

# Incorrect: exactly 5,000 cents qualifies for free shipping.
assert shipping_fee(5000, premium=False) == 499

The assertion is syntactically valid. It might even pass against a faulty implementation using subtotal > 5000. Neither fact makes the test correct.

A different candidate might contain a true but insufficient assertion:

# Too weak to verify either the threshold or the correct fee.
assert shipping_fee(5000, premium=False) >= 0

This test accepts both 0 and 499. It cannot distinguish correct threshold behavior from the defect above.

That distinction matters: an assertion can be logically true without adequately verifying the requirement it claims to cover.

For each candidate, ask: what plausible incorrect implementation would this test reject? A reviewer should be able to identify a specific answer: an off-by-one threshold, the wrong fee, an unauthorized state change, an omitted database write, or another relevant failure.

Watch for Circular Verification

A generated test can also calculate its expected result from the same system it is supposed to verify:

expected = shipping_fee(5000, premium=False)
actual = shipping_fee(5000, premium=False)
assert actual == expected

This checks repeatability for that input, not compliance with the shipping rules.

The same problem appears when generated tests copy production calculations into their expected-value logic or mock the function under test and then assert the mocked return value.

Prefer expectations justified by approved examples, independently reviewed calculations, or a trusted reference implementation. Differential testing against another implementation is useful evidence, but agreement between implementations is not proof that both satisfy the contract.

Likewise, ask a second model to identify suspicious assertions, but do not treat model agreement as independent ground truth.

Review the Whole Scenario

Oracle review alone is insufficient. A correct expected result attached to the wrong setup still makes a misleading test.

Suppose a scenario claims to verify standard-account shipping, but its fixture creates a premium account. An expected zero fee might pass for the wrong reason.

Inspect account roles, feature flags, initial state, request payloads, and mocked dependencies. For state-changing operations, check the relevant side effects as well as the immediate response. A successful API response and a correctly persisted order are different observations.

When evaluating existing behavior during a legacy-system migration, distinguish characterization tests, which record current behavior, from requirements tests, which assert intended behavior. Do not silently promote the former into proof of the latter.

Use a Review-Based Accuracy Metric

A practical metric is:

Validated-case rate = (Cases confirmed semantically valid ÷ Cases reviewed) × 100

Classify reviewed cases as valid, incorrect, or unresolved. Keep unresolved cases in the denominator rather than making uncertainty disappear.

For example, if 80 of 100 reviewed cases are valid, 12 have incorrect expectations, and eight depend on unresolved requirements, the validated-case rate is 80%. The unresolved 8% should be reported separately.

This is a measure of review outcomes, not an estimate of the application’s correctness. Report whether the review covered every candidate or a sample, and examine high-risk cases individually.

2. Coverage: What Does the Suite Actually Verify?

Coverage needs more than one view.

Code coverage describes execution. Requirements coverage describes which obligations have tests. Fault-detection evidence describes whether those tests reject relevant incorrect behavior.

Google’s testing guidance explicitly warns that covered lines and branches have been executed, but have not necessarily been tested correctly. It recommends using coverage alongside other evidence rather than treating it as a complete measure of test quality.

Build the Scenario Inventory Independently

Do not ask the generator to create tests and then judge completeness only against the scenarios it generated.

Create a reviewed inventory from the contract first. For the shipping service, useful obligations include:

The last scenario protects a different failure mode from the preceding one. An implementation could return free shipping for premium customers before validating the subtotal.

For a larger application, extend the inventory to state transitions, permissions, retries, partial failures, configuration combinations, and integration boundaries. Define nonfunctional obligations separately. A unit-test suite cannot establish a latency requirement without a suitable workload, measurement method, and environment.

Measure Coverage Against Explicit Obligations

A useful scenario-coverage measure is:

Scenario coverage = (Applicable obligations with valid, executed checks ÷ Total applicable obligations) × 100

A requirement ID in a comment is not enough. The test must actually exercise the scenario and evaluate a relevant outcome.

Also distinguish planned coverage from demonstrated coverage. A reviewed test that cannot run represents planned verification, not completed evidence.

For risk-sensitive systems, use a weighted version:

Risk-weighted coverage = (Σ wi × ci ÷ Σ wi) × 100

where wi is a pre-agreed risk weight for obligation i,
and ci is 1 when the obligation has a valid, executed check, and 0 otherwise.

Set weights before evaluating the generated suite. Otherwise, changing weights can become another way to improve the dashboard without improving testing.

Keep critical gaps visible individually. An aggregate score should not conceal an untested authorization rule.

Finally, coverage is not a pass rate: a valid test that exposes a product defect can cover an obligation while showing that the implementation violates it.

Measure Incremental Code Coverage

Evaluate candidates against the existing suite:

Incremental coverage = coverage(existing suite + candidates) − coverage(existing suite)

Keep the source revision, measurement scope, exclusions, and environment constant.

Prefer branch information where decisions matter. Coverage.py’s documentation illustrates how statement coverage can report all lines executed even when one outcome of an if condition was never taken; branch measurement exposes that missing transition.

Inspect which branches were added, not only the percentage-point increase. Newly executed error-handling logic may be more valuable than additional incidental initialization coverage.

Conversely, zero additional branch coverage does not automatically make a candidate useless. A stronger assertion may detect a defect on a path the existing suite already executes.

Test the Tests With Mutations

Mutation testing introduces controlled changes to application code and checks whether the tests detect them. A mutant is typically considered killed when the altered program causes a test to fail. PIT and Stryker document this approach as a way to assess test effectiveness beyond execution coverage.

For the shipping service, useful mutations include changing >= 5000 to > 5000, returning the wrong fee, removing negative-input validation, or allowing the premium check to bypass validation.

Measure which baseline-surviving mutants the candidates newly detect. Keep mutation operators and the evaluated code scope fixed when comparing suites.

Inspect result categories, not just the headline score. Tools can distinguish killed, surviving, uncovered, invalid, and timed-out mutants, and their scoring conventions matter. Equivalent mutants, changes that do not alter observable behavior, also limit the interpretation of a perfect-score target.

A mutant failing because the test environment crashed is not the same evidence as a targeted assertion detecting the intended behavioral change.

Add Properties Where They Strengthen the Contract

Property-based testing can complement concrete examples by generating inputs from a defined domain and searching for violations of an asserted property. Hypothesis supports this style of testing in Python.

For the illustrative contract, a reviewed property could state that the fee for a standard account never increases when a nonnegative subtotal increases.

That property is useful, but insufficient alone: an implementation returning zero for every subtotal would satisfy it. Retain concrete examples for exact fees and boundaries.

Generated properties and metamorphic relationships need oracle review just as much as generated example-based assertions.

Not Sure Your AI-Generated Tests Are Actually Testing Anything?

Talk to Our QA Team

3. Redundancy: Does Each Test Add Useful Evidence?

Redundancy is not the same as textual similarity.

Two differently named tests may perform the same setup, use inputs from the same partition, and assert the same outcome. Conversely, two almost identical tests at 4999 and 5000 protect different sides of a boundary.

Evaluate duplication both within the generated batch and against the existing suite.

Compare Semantics Before Deleting Tests

A useful comparison record includes the requirement, initial state, input partition, action, expected outcome, assertion target, and test layer.

Text similarity, embeddings, or normalized syntax trees can help identify candidates for review. Treat their output as a shortlist, not a deletion decision. Normalization that removes literal values can erase the difference between an ordinary input and a critical boundary.

Where available, add execution evidence. Coverage.py can record measurement contexts, allowing execution information to be associated with individual tests or other contexts.

However, identical execution coverage is still not proof of redundancy. One test might check a response code while another checks that no unauthorized state change occurred.

Use Controlled Removal Experiments

A defensible removal process is to temporarily remove a suspected duplicate, rerun the relevant evidence checks, and compare what was lost.

Evaluate scenario coverage, important mutant kills, known-defect detection, and diagnostic usefulness. Preserve deliberate verification at different layers when it protects distinct risks, for example, a unit test for a calculation and an integration test for the API’s mapping of that result.

Remove candidates incrementally. If two tests are interchangeable, either may be removable individually, but deleting both can create a gap.

A practical reporting metric is:

Redundancy reduction rate = (Candidates removed as unnecessary duplicates ÷ Valid, executable candidates before deduplication) × 100

Record the evidence supporting each removal. This measures redundancy under the chosen evaluation criteria, not universal equivalence under every future code change.

Parameterization is often a better outcome than deletion. It reduces repeated test code while preserving separate input scenarios and failure identities.

The objective is useful verification at reasonable maintenance and execution cost, not the smallest possible test count.

4. Executability: Can the Test Run Safely and Reproducibly?

Executability has several stages: discovery, dependency resolution, setup, application interaction, outcome evaluation, and cleanup.

A test that parses successfully has completed only an early check.

Review Before Running

Inspect generated imports, fixtures, API calls, selectors, dependency changes, filesystem operations, and external requests.

Reject or repair candidates that invent helper methods, reference unavailable fixtures, require undocumented accounts, or depend on local state that CI does not have. Do not automatically install every package a generated test requests.

Treat generated code as untrusted during evaluation. OpenAI’s HumanEval repository, for example, explicitly warns against executing model-generated code outside a robust security sandbox.

Use a security-reviewed disposable environment with synthetic data, restricted credentials, controlled network access, and resource limits. Do not expose production secrets, privileged host interfaces, or unrestricted write access.

Place test collection inside that boundary too: pytest imports test modules as part of its integration and discovery behavior, so collection should not be treated as passive text inspection.

Confirm That the Intended Tests Actually Ran

Compare expected scenario IDs with collected and executed IDs. A successful job is insufficient when the intended tests were skipped, deselected, or never discovered.

Pytest supports collection-only inspection and reports a distinct exit code when no tests are collected. Its skip and expected-failure mechanisms also require explicit review: an expected failure is not an ordinary passing assertion.

Track setup errors, assertion outcomes, teardown errors, skipped cases, expected failures, and unexpected passes separately.

A useful metric is:

Executable-case rate = (Scheduled cases that reach a verdict and complete cleanup ÷ Cases scheduled for execution) × 100

An assertion failure can still be a valid execution outcome. A missing fixture is a different category.

Investigate Failures Before Repairing Tests

When a candidate fails, determine whether the cause is an incorrect test, a product defect, an environment problem, or an unresolved contract.

Do not replace an expected value with the observed value merely to obtain a green run.

A valid test exposing a confirmed defect should enter the defect-management workflow. Preserve it as a reproducer and, after the fix, as regression protection. Any temporary expected-failure treatment should have an owner, a linked issue, and a removal condition.

This distinction prevents a generator’s “repair” loop from turning defect detection into defect acceptance.

Evaluate Stability Under Meaningful Variations

Run candidates in isolation, with the existing suite, in different orders, and under the intended parallelism. Control or record clocks, time zones, random seeds, database state, and external-service behavior.

Pytest’s documentation identifies insufficiently controlled state, ordering dependencies, cleanup problems, and overly strict assertions as potential causes of flaky tests.

For UI tests, prefer observable readiness conditions over arbitrary sleeps. Playwright supports actionability checks and retrying assertions; its guidance recommends resilient locators based on user-facing attributes and explicit contracts.

Report the number and conditions of repeated runs. “No failures observed in 20 runs” is evidence, not proof of determinism. Under independent runs with a true 1% failure probability, the probability of seeing 20 consecutive passes is:

0.99^20 ≈ 81.8%

Retries should not erase the first failure from evaluation records. Distinguish a consistently passing test from one that passes only after reruns.

For manual cases, perform an independent walkthrough using the documented environment and data. When an evaluator must invent missing steps, the procedure is not yet executable as written.

A Worked Example: From Contract to Reviewable Tests

For the illustrative application interface, the shipping scenarios can be expressed as follows:

# tests/generated/test_shipping.py
import pytest
from checkout.shipping import shipping_fee

@pytest.mark.parametrize(
    ("subtotal_cents", "premium", "expected_fee"),
    [
        pytest.param(0, False, 499, id="SHIP-2-zero"),
        pytest.param(4999, False, 499, id="SHIP-2-below-threshold"),
        pytest.param(5000, False, 0, id="SHIP-1-at-threshold"),
        pytest.param(5001, False, 0, id="SHIP-1-above-threshold"),
        pytest.param(0, True, 0, id="SHIP-3-premium-zero"),
    ],
)
def test_shipping_fee(
    subtotal_cents: int,
    premium: bool,
    expected_fee: int,
) -> None:
    actual_fee = shipping_fee(subtotal_cents, premium=premium)
    assert actual_fee == expected_fee

@pytest.mark.parametrize(
    "premium",
    [
        pytest.param(False, id="standard"),
        pytest.param(True, id="premium"),
    ],
)
def test_negative_subtotal_is_rejected(premium: bool) -> None:
    with pytest.raises(ValueError):
        shipping_fee(-1, premium=premium)

Pytest parameterization supports executing a test function with multiple argument sets and distinct case identifiers.

This module represents seven scenarios, not two merely because it contains two test functions.

Its expected values come directly from the stated contract. The threshold cases protect different boundaries, while the negative-input cases check that membership does not bypass validation.

It remains a candidate suite until it runs against the actual application and its contribution is compared with existing tests. It makes no claim to cover API serialization, checkout persistence, or other behavior outside this function’s contract.

Collect Execution Evidence Without Hiding Failures

For a repository containing the illustrative checkout package, the following shell script captures execution and coverage artifacts. Run it from the repository root inside the approved sandbox, with the application and approved versions of pytest and Coverage.py already installed.

#!/usr/bin/env bash
mkdir -p artifacts || exit "$?"

# Discovery must succeed before execution.
python -m pytest tests/generated --collect-only -q || exit "$?"

run_status=0
python -m coverage run --branch --source=checkout \
  -m pytest tests/generated \
  -q -ra --strict-markers \
  --junitxml=artifacts/generated.xml || run_status=$?

# Attempt to retain coverage evidence even when tests fail.
report_status=0
python -m coverage json \
  -o artifacts/generated-coverage.json || report_status=$?

# A successful report command must not hide a failed test run.
if [ "$run_status" -ne 0 ]; then
  exit "$run_status"
fi

exit "$report_status"

Coverage.py supports running Python modules under measurement, limiting source scope, and exporting JSON reports.

This script collects evidence; it is not a complete acceptance gate. CI must still audit expected versus executed cases, review skips and expected failures, evaluate security constraints, and enforce the agreed coverage policy.

Run baseline-only and combined-suite measurements separately, using clean coverage data and identical settings, to calculate incremental value.

Turn the Evaluation Into an Acceptance Gate

Use a staged workflow: quarantine candidates, validate their contracts and code, execute them safely, measure their contribution, review the evidence, and then promote selected tests.

The following is an example policy to adapt to the application’s risk, not an industry-standard threshold:

Do not collapse these conditions into one weighted “AI test quality” score. A serious failure in one dimension should remain visible.

Separate candidate yield from the quality of the accepted suite. Keep counts of initial candidates, rejected cases, repaired cases, duplicates removed, confirmed defects discovered, and tests promoted. Otherwise, repeated regeneration can make the final output look strong while concealing substantial review effort.

Attach evidence to retained tests: their requirement and oracle sources, relevant code and environment revisions, execution results, incremental verification value, reviewer, and maintenance owner.

Acceptance also needs lifecycle management. Re-evaluate generated tests when contracts, interfaces, fixtures, or dependencies change, just as you would other verification code.

Conclusion

AI test cases should earn a place in QA through evidence. Accuracy establishes that the test checks the intended behavior. Coverage shows which obligations it verifies. Redundancy analysis explains why it belongs alongside existing tests. Executability demonstrates that the check can run safely and produce interpretable results.

The most useful acceptance question is not, “How many tests did the model generate?” It is: “Which important failures can this suite now detect, and what evidence shows that those checks are correct, necessary, and reliable?” Talk to our LLM testing team if you want help building this evaluation gate into your own pipeline.