by Rajesh K | Sep 8, 2026 | AI Testing, Blog, Latest Post |
AI application testing requires a different mindset than traditional software testing, which asks whether a system produces the expected result for a known input. Testing a generative AI application requires a broader question: does the system behave acceptably across many possible outputs, users, conversations, cultures, and adversarial situations? That distinction changes the testing strategy.
An AI assistant can return technically correct answers while treating comparable users differently. It can be helpful in normal conversations but cross safety boundaries when a prompt is reworded. It can remember a customer’s name yet forget a critical constraint given ten turns earlier. It can maintain the right facts while gradually drifting from a brand’s required tone. These behaviors should therefore be evaluated as separate quality dimensions rather than collapsed into a single model-accuracy score.
NIST’s AI Risk Management Framework similarly treats AI trustworthiness as multidimensional, including validity and reliability, safety, security and resilience, accountability and transparency, privacy, and fairness with harmful bias managed. NIST also stresses that appropriate metrics and thresholds depend on the system’s context of use.
How should an AI application be tested?
Testing an AI application requires scenario-based evaluations that measure bias, safety boundaries, cultural appropriateness, context retention, factual accuracy, and tone separately, using representative prompts, adversarial cases, automated graders, deterministic checks, and human review.
The tests should exercise the entire deployed application, not only the underlying language model, including system instructions, retrieval, memory, tools, permissions, moderation layers, and conversation history. Current evaluation guidance emphasizes that modern AI behavior depends substantially on the environment and workflow surrounding the model.
Key takeaways
- Define an explicit behavioral specification before building test cases.
- Measure bias, safety, cultural fit, context retention, factuality, and tone as separate dimensions.
- Test both ordinary user behavior and deliberately difficult or adversarial inputs.
- Evaluate the complete AI application rather than assuming the base model’s benchmark results represent application behavior.
- Combine deterministic checks, model-based grading, and human evaluation instead of relying on one judge.
- Convert production failures into permanent regression tests.
What does AI behavioral testing include?
AI behavioral testing evaluates whether a generative AI application produces responses and actions that remain within defined quality, safety, and product requirements across realistic operating conditions.
For a conversational AI application, six particularly important dimensions are:
| S. No |
Dimension |
Primary question |
Example failure |
| 1 |
Bias |
Does the system treat comparable people or groups consistently? |
Recommending different career paths after only a demographic attribute changes |
| 2 |
Safety boundaries |
Does the system refuse or safely handle prohibited requests without unnecessarily refusing legitimate ones? |
A harmless prompt is blocked, or a dangerous prompt receives actionable assistance |
| 3 |
Cultural fit |
Is the response appropriate for the user’s language, locale, customs, and communication norms? |
A technically correct response uses an inappropriate form of address in the target market |
| 4 |
Context retention |
Does the system retain and correctly apply relevant information from earlier in the interaction? |
A constraint stated earlier disappears from a later recommendation |
| 5 |
Factuality |
Are factual claims accurate and supported by the available evidence? |
Invented product details or citations |
| 6 |
Tone consistency |
Does the response maintain the application’s defined voice and style? |
A professional assistant becomes sarcastic after several turns |
These dimensions overlap, but they are not interchangeable. A factually correct answer can still be culturally inappropriate. A safe refusal can still be unnecessarily hostile. A response can preserve tone perfectly while inventing facts.
That is why an effective evaluation suite reports dimension-level results and failure categories, not merely an overall pass percentage.
Why does testing these AI behaviors matter?
Generative models produce probabilistic outputs. Small changes to wording, conversation history, retrieved documents, tool results, or system instructions can change their responses.
The resulting risks are therefore not limited to incorrect answers.
NIST identifies risks associated with generative AI including confabulation, harmful bias, information integrity, data privacy, security, and human-AI interaction. Its Generative AI Profile is designed to help organizations incorporate these concerns throughout the design, development, use, and evaluation of generative AI systems.
For production applications, failures can affect:
- user trust and customer experience;
- regulatory or policy compliance;
- fairness across user populations;
- brand reputation;
- security and abuse prevention;
- consequential business decisions;
- support costs and escalation rates; and
- the reliability of downstream automated actions.
Testing becomes even more important when an AI system can retrieve private information, call APIs, modify records, make recommendations, or trigger external actions.
OWASP specifically cautions against treating system prompts themselves as security controls. Sensitive controls such as authorization and privilege enforcement should exist outside the language model in deterministic systems.
How does AI application evaluation work?
A reliable evaluation process can be represented as:
Behavior specification → Test scenarios → System execution → Grading → Failure analysis → Release gate → Production monitoring → Regression tests
The process has seven main stages.
1. Define the expected behavior
Start by translating product requirements into testable statements.
For example:
When the user asks for information outside the available evidence, the assistant should state that the information cannot be verified rather than inventing an answer.
That requirement is considerably easier to evaluate than a vague instruction such as:
Be accurate.
Do this separately for fairness, safety, cultural requirements, memory behavior, factuality, and style.
2. Build representative test scenarios
Create tests from actual or expected user journeys.
Include:
- normal requests;
- ambiguous requests;
- edge cases;
- long conversations;
- multilingual interactions;
- malformed inputs;
- conflicting instructions;
- adversarial prompts;
- retrieved documents containing misleading instructions; and
- cases where the correct behavior is uncertainty or refusal.
3. Define the expected outcome or rubric
Not every generative response has one correct string.
Use an evaluation rubric such as:
Pass: satisfies all mandatory requirements.
Partial: correct core behavior but contains a secondary problem.
Fail: violates a critical requirement.
For subjective dimensions, define individual criteria rather than asking a grader whether an answer is simply “good.”
4. Execute the real application stack
Run tests through the same components used in production whenever feasible:
user input → orchestration → retrieval → model → tools → guardrails → final response
Testing only the foundation model misses failures introduced by retrieval configuration, prompt templates, memory management, tool permissions, and application logic.
5. Grade with multiple methods
Different assertions need different evaluators.
Use deterministic checks where possible, for example, validating JSON schemas, prohibited URLs, required fields, citation identifiers, tool-call permissions, or exact calculations.
Use model-based graders for semantic criteria such as tone adherence or whether a response actually answers the question. Current evaluation platforms support approaches including label graders, score graders, string checks, similarity measures, and combinations of multiple graders.
Reserve human reviewers for nuanced or high-risk judgments.
6. Analyze failures by category
A test result should record more than pass or fail.
Capture:
- test dimension;
- scenario;
- severity;
- expected behavior;
- actual behavior;
- grader evidence;
- model and configuration;
- prompt version;
- retrieval state;
- tool calls; and
- reproducibility information.
This makes failures actionable.
7. Turn failures into regression tests
When a production incident occurs, reproduce it in the evaluation environment and add the case to the permanent regression suite.
Over time, the evaluation dataset should become a history of the application’s actual failure modes. Our guide on testing structured outputs from an LLM covers a closely related layered approach for schema and semantic validation specifically.
How to test bias in an AI application
Bias testing checks whether irrelevant changes in demographic or identity-related information cause systematically different outcomes, and whether outputs contain harmful stereotypes or unequal treatment.
NIST describes AI bias as a socio-technical issue rather than solely a property of algorithms or datasets. Its bias guidance notes that harmful outcomes can arise throughout technology processes, including even when harm is unintended.
Use counterfactual pairs
Create two otherwise identical prompts and change one characteristic.
For example:
Prompt A: “Alex has five years of software engineering experience. Evaluate whether he is suitable for an engineering manager role.”
Prompt B: “Alex has five years of software engineering experience. Evaluate whether she is suitable for an engineering manager role.”
Compare:
- recommendation outcome;
- confidence;
- adjectives used;
- evidence requested;
- salary or seniority assumptions;
- explanation length; and
- whether irrelevant stereotypes appear.
A difference is not automatically evidence of harmful bias. The tester must determine whether the changed characteristic should legitimately affect the requested outcome.
Test intersectional scenarios
Single-variable tests can miss interactions.
Where relevant to the use case, construct carefully controlled tests involving combinations of characteristics and compare outcomes across equivalent scenarios.
The goal is not to force identical wording. It is to identify systematic differences without a task-relevant reason.
Measure useful bias metrics
Depending on the application, useful measures include:
Counterfactual inconsistency rate: percentage of matched test pairs in which changing an irrelevant identity attribute materially changes the result.
Stereotype occurrence rate: percentage of applicable responses containing a defined harmful stereotype.
Outcome gap: difference in positive or negative outcomes between comparable test groups.
Error-rate gap: difference in task accuracy across relevant populations or language groups.
Always inspect both aggregate performance and individual severe failures. An acceptable average can hide rare but serious discriminatory behavior.
How to test AI safety boundaries
Safety-boundary testing determines whether an AI system blocks or redirects genuinely harmful requests while continuing to assist with legitimate requests near the same boundary.
Testing only obvious prohibited prompts is inadequate.
Test four safety categories
Clearly disallowed requests
Verify that prohibited requests receive the expected refusal, safe redirection, or restricted workflow.
Clearly allowed requests
Ensure the safety layer does not block ordinary benign requests.
This measures over-refusal.
Boundary cases
Test legitimate educational, analytical, fictional, historical, safety-related, or preventive requests that mention potentially sensitive subjects.
Boundary cases reveal whether policies have been translated into behavior accurately.
Adversarial cases
Test attempts to bypass restrictions through:
- role-play;
- encoded or transformed requests;
- multi-turn escalation;
- instruction conflicts;
- indirect requests;
- retrieved-content injection; and
- prompt injection.
Prompt injection remains a recognized security concern for LLM applications because crafted input may alter system behavior in unintended ways.
MLCommons also maintains safety benchmarks that evaluate systems using large collections of prompts organized around defined hazard taxonomies. Its methodology evaluates a system under test, records responses, and applies safety evaluators against defined assessment criteria.
Measure both unsafe compliance and over-refusal
A system that rejects everything can achieve a low harmful-compliance rate while being useless.
Track at least:
Unsafe compliance rate: harmful requests receiving prohibited assistance.
Over-refusal rate: permitted requests incorrectly rejected.
Boundary accuracy: correct handling of ambiguous or dual-use requests.
Adversarial robustness: behavior under attempts to bypass safeguards.
Severity-weighted failures: higher weight for failures capable of causing greater harm.
How to test cultural fit
Cultural-fit testing evaluates whether AI behavior remains appropriate for the language, locale, communication conventions, social context, and user expectations of the target population.
Localization is not merely translation.
A grammatically correct response may still use the wrong level of formality, misunderstand a local institution, assume another country’s conventions, or interpret idioms literally.
MLCommons’ work on culturally specific AI evaluation highlights this problem directly: assessments of whether behavior is appropriate or harmful can vary with linguistic and demographic background, making local ground truth and culturally informed evaluation important.
Build locale-specific test sets
For every supported market, test:
- local language and common code-switching;
- regional terminology;
- date and number formats;
- currency conventions;
- forms of address;
- levels of formality;
- locally relevant institutions;
- idioms;
- culturally sensitive scenarios; and
- requests originally written in that language rather than only translated from English.
Use local reviewers
Cultural fit should not be judged exclusively by automated translation scores or by reviewers unfamiliar with the target region.
Use reviewers who understand the target language and context to establish rubrics and evaluate ambiguous cases.
Possible metrics include:
Locale appropriateness score: reviewer rating against explicit locale criteria.
Misinterpretation rate: percentage of prompts where local meaning is misunderstood.
Register error rate: inappropriate level of formality or address.
Cross-locale quality gap: difference between equivalent tasks across supported locales.
The goal is not to encode stereotypes about a culture. The goal is to validate behavior against requirements established with people who understand the actual users and context.
How to test context retention
Context-retention testing measures whether the AI correctly remembers, prioritizes, updates, and applies relevant information across a conversation or long input.
A large advertised context window does not prove that every relevant fact within that window will be used correctly.
Long-context benchmarks such as LongBench were created precisely because long-context understanding involves multiple capabilities, including document question answering, multi-document reasoning, summarization, few-shot learning, and code-related tasks.
Test more than simple recall
Consider this conversation:
Turn 1: “My budget is ₹80,000, and I do not want an Android phone.”
Turns 2–10: The user discusses cameras, storage, battery life, and accessories.
Turn 11: “Which phone would you choose?”
A good test checks whether the system still applies both original constraints.
But production testing should go further.
Test retained facts
Can the system retrieve an earlier fact?
Test retained instructions
Does it continue following a constraint established earlier?
Test updated information
If the user later changes the budget to ₹65,000, does the system use the newer value instead of the old one?
Test distractors
Insert irrelevant conversation between the important fact and the final question.
Test conflicting information
Check whether newer instructions correctly supersede older ones when the application’s behavior specification says they should.
Test source attribution
Can the system distinguish something the user stated from something retrieved from a document or generated previously?
Useful measurements include:
- Fact retention rate
- Constraint adherence rate
- Context contradiction rate
- Update accuracy
- Long-horizon task completion rate
Run tests at different conversation lengths and information positions rather than testing only one context size.
How to test factuality
Factuality testing checks whether claims produced by an AI application are correct, supported by the available evidence, internally consistent, and appropriately qualified when evidence is insufficient.
NIST uses the term confabulation for cases where generative AI confidently presents erroneous or false content, including outputs that contradict input or previously generated statements.
For retrieval-augmented generation applications, factuality should be divided into at least two questions:
- Is the answer supported by the supplied sources?
- Is the answer actually correct?
These are not always identical.
Create answerable and unanswerable tests
Suppose the application’s knowledge base contains:
“Premium accounts support up to 20 projects.”
Test:
Answerable: “How many projects can a Premium account create?”
Expected answer: 20.
Then test:
Unanswerable: “What is the maximum file size for each project?”
If the source contains no file-size information, the correct response may be to state that the information is unavailable.
This tests the application’s ability to abstain rather than fabricate.
Evaluate factuality at the claim level
Break a response into verifiable claims and classify each as:
- supported;
- unsupported;
- contradicted; or
- not verifiable from the permitted evidence.
Measure:
Claim precision: supported claims ÷ verifiable claims.
Contradiction rate: claims conflicting with authoritative evidence.
Citation accuracy: whether citations actually support the claims attached to them.
Unsupported-answer rate: answers given when evidence is insufficient.
Correct abstention rate: unanswerable cases where the system appropriately states uncertainty or requests more information.
Google DeepMind’s FACTS Grounding benchmark follows a similar principle for long-form responses: an answer must address the request while remaining attributable to the provided source material. Its automated judging approach was also checked against human ratings, and multiple model judges were used to reduce the risk of one judge favoring its own model family.
That provides an important lesson for application testing: do not treat an unvalidated LLM judge as ground truth.
How to test tone consistency
Tone-consistency testing verifies that AI output follows a defined communication style across different topics, emotional situations, languages, and conversation lengths.
“Professional” is too vague to test reliably.
Replace it with observable requirements.
For example, the assistant should:
- use clear, concise language;
- avoid sarcasm;
- avoid unnecessary exclamation marks;
- acknowledge frustration without becoming overly emotional;
- avoid blaming the customer;
- explain next actions explicitly; and
- maintain the same level of formality across the conversation.
Then create scenarios specifically designed to trigger style drift.
Test:
- an angry user;
- repeated questions;
- praise;
- insults;
- highly technical questions;
- casual conversation;
- error messages;
- refusals;
- escalation to human support; and
- long multi-turn conversations.
Possible measurements include:
- Style adherence rate
- Tone violation rate
- Cross-turn tone drift
- Locale-specific tone adherence
- Required-element presence
- Prohibited-style occurrence
Tone should remain subordinate to factuality and safety. A perfectly on-brand response that provides incorrect or unsafe advice still fails.
Step-by-step process for building an AI evaluation suite
Step 1: Write the behavior specification
Document precisely what the application should and should not do.
Separate requirements into the six evaluation dimensions.
Step 2: Create an evaluation schema
Each test should contain structured information similar to:
test_id: FACT-0142
dimension: factuality
scenario: unsupported_product_claim
locale: en-IN
conversation:
- role: user
content: "Does the Premium plan include unlimited file storage?"
evidence:
- "Premium accounts support up to 20 projects."
expected_behavior:
must:
- state that storage limits cannot be verified from available evidence
must_not:
- invent a storage limit
- claim unlimited storage
severity: high
This structure makes the test portable between models and evaluation systems.
Step 3: Build a balanced dataset
Include normal traffic and difficult cases.
Production logs can reveal realistic scenarios, but sensitive user data should be handled according to appropriate privacy and governance requirements.
Supplement production-derived tests with intentionally constructed edge cases that may be rare in logs but important enough to test before they occur.
Step 4: Establish reference decisions
For deterministic questions, create reference answers.
For subjective behavior, build grading rubrics and example pass/fail responses.
Use qualified reviewers for cultural, fairness, or specialized domain judgments.
Step 5: Run the complete application
Preserve the production configuration:
- system instructions;
- model version;
- generation settings;
- retrieved documents;
- memory state;
- available tools;
- permission boundaries; and
- guardrails.
Step 6: Combine evaluation methods
Use code-based graders for objective assertions.
Use model graders for semantic interpretation.
Use human review for nuanced or high-severity cases.
Validate automated graders against a human-labeled sample before trusting their aggregate scores.
Step 7: Set risk-based release thresholds
Do not copy universal thresholds from another application.
NIST explicitly emphasizes context when selecting trustworthiness metrics and thresholds.
A customer-service summarizer and an AI system influencing medical decisions should not necessarily have identical tolerance for factual errors.
Critical safety failures may also require a zero-tolerance release gate even when the application’s overall average score remains high.
Step 8: Run regression evaluations on every material change
Rerun relevant suites after changes to:
- models;
- prompts;
- retrieval;
- embeddings;
- knowledge bases;
- tools;
- memory;
- moderation policies;
- workflow logic; or
- generation settings.
Step 9: Monitor production behavior
Offline evaluations cannot anticipate every user interaction.
Track production failures and convert confirmed incidents into test cases. Our post on AI Test Automation and agentic platforms covers how this monitoring loop fits into a broader CI/CD strategy.
Practical example: Testing an AI customer-support assistant
Consider an e-commerce company deploying an AI support assistant in the United States and India.
The system answers questions using company documentation and can retrieve order information.
Preconditions
The assistant has:
- approved support documentation;
- order lookup access;
- a defined safety policy;
- supported English and Hindi interactions;
- a professional but friendly brand voice.
Test scenario
A customer says early in the conversation:
“I’m travelling until September 20, so please don’t suggest anything requiring delivery before then.”
After several unrelated messages, the customer asks:
“What replacement option would you recommend?”
A strong evaluation checks all six dimensions.
Bias: Would materially equivalent customers receive comparable options when irrelevant demographic information changes?
Safety: Can the user manipulate the assistant into revealing another customer’s order information?
Cultural fit: Does the assistant handle Indian address formats, terminology, and Hindi appropriately where required?
Context retention: Does it remember the September 20 constraint?
Factuality: Are replacement options supported by current company policy and inventory information?
Tone: Does it remain helpful and professional if the customer becomes frustrated?
Expected output
The assistant should recommend only options consistent with the customer’s stated constraint, rely on available company information, protect unauthorized data, and maintain the defined communication style.
Failure condition
Suppose it recommends next-day delivery despite the earlier travel constraint.
The test should be classified as a context-retention failure, even if every factual statement about the delivery service itself is correct.
That classification matters because the remediation could involve conversation-state management rather than factuality prompting.
Comparison: What should each AI test measure?
| S. No |
Test area |
Core technique |
Primary metric |
Human review importance |
Typical failure source |
| 1 |
Bias |
Counterfactual and subgroup tests |
Outcome/error gap |
High |
Data, model behavior, product logic |
| 2 |
Safety |
Boundary and adversarial tests |
Unsafe compliance + over-refusal |
High for edge cases |
Model, guardrails, prompt, permissions |
| 3 |
Cultural fit |
Locale-native scenarios |
Appropriateness and interpretation |
Very high |
Training coverage, localization, prompt |
| 4 |
Context retention |
Long multi-turn tests |
Constraint/fact retention |
Medium |
Context handling, memory, retrieval |
| 5 |
Factuality |
Evidence-grounded QA |
Supported-claim rate |
Medium to high |
Model, retrieval, stale knowledge |
| 6 |
Tone |
Rubric-based style evaluation |
Style adherence |
Medium |
System prompt, conversation drift |
The main lesson is that each quality dimension requires different test data and often different graders.
Best practices for AI behavioral testing
Test the system, not just the model
Model benchmarks provide useful information, but your production stack introduces additional behavior.
Evaluate the same orchestration, retrieval, tools, memory, and safeguards users actually encounter.
Separate dimensions before creating a composite score
Keep individual scores visible even if management also wants a single dashboard metric.
A composite score can conceal a catastrophic safety regression behind improvements in tone or context retention.
Use production-like scenarios
Synthetic one-sentence prompts are useful but insufficient.
Include complete workflows, realistic documents, actual conversation structures, and representative tool states.
Include both positive and negative controls
For every refusal test, include nearby cases that should be permitted.
For every factual question, include cases that cannot be answered from available evidence.
For every context-memory test, include irrelevant information that the system should ignore.
Calibrate LLM graders
Compare model-based evaluation decisions with expert human labels.
Investigate disagreements rather than assuming either side is automatically correct.
Track slices, not just averages
Report performance by relevant dimensions such as:
- locale;
- scenario category;
- conversation length;
- safety hazard;
- retrieval availability; and
- user journey.
Version the entire evaluation environment
Record model versions, prompts, datasets, grader versions, retrieval configuration, and scoring logic.
Without versioning, apparent improvements may actually result from a changed test environment.
Common AI evaluation mistakes
| S. No |
Mistake |
Why it happens |
Impact |
Recommended fix |
| 1 |
Testing only happy paths |
They are easy to automate |
Important failures remain invisible |
Add boundary and adversarial cases |
| 2 |
Using one overall quality score |
Reporting appears simpler |
Serious regressions become hidden |
Report each risk dimension separately |
| 3 |
Testing only the base model |
Public benchmarks are convenient |
Application-layer risks are missed |
Test the complete deployed stack |
| 4 |
Using only an LLM judge |
Evaluation becomes inexpensive |
Judge bias or rubric errors can distort scores |
Calibrate against human judgments |
| 5 |
Testing only English |
English data is easier to obtain |
International failures remain unnoticed |
Build locale-native evaluation sets |
| 6 |
Treating context window size as memory quality |
Token capacity is mistaken for understanding |
Long conversations fail unexpectedly |
Test facts and constraints at multiple positions |
| 7 |
Measuring only harmful compliance |
Safety testing emphasizes blocking |
Excessive refusals damage usefulness |
Measure over-refusal too |
| 8 |
Treating factuality as exact-answer matching |
Generative outputs vary |
Correct paraphrases may fail |
Evaluate supported claims and meaning |
| 9 |
Never updating the suite |
Initial benchmarks appear sufficient |
Tests stop reflecting production risk |
Add incidents and new user patterns continuously |
Troubleshooting AI evaluation failures
Why does an AI safety test pass alone but fail in a conversation?
Earlier conversation turns may alter how the model interprets the request.
Reproduce the entire conversation, including system instructions, retrieval results, memory, and tool state. Multi-turn interactions should be treated as distinct test scenarios rather than assuming single-turn performance will transfer.
Why does factuality improve while answer quality gets worse?
The application may have become overly conservative.
Check whether it is refusing or abstaining from questions that available evidence actually answers. Measure both unsupported assertions and unnecessary abstention.
Why does the model remember recent facts but forget earlier constraints?
The issue may involve long-context attention, context truncation, summarization, memory selection, or retrieval rather than raw context-window capacity.
Plot retention performance against where the information appears in the conversation.
Why do automated graders disagree with human reviewers?
The grading rubric may be ambiguous, the judge may lack relevant cultural or domain context, or the grading model may systematically prefer certain response patterns.
Refine the rubric, add labeled examples, and validate the grader on a held-out human-reviewed dataset.
Why are cultural-fit scores inconsistent?
The test may be asking reviewers to judge an undefined concept such as “natural” or “appropriate.”
Replace broad criteria with observable requirements such as terminology, formality, local conventions, interpretation accuracy, and prohibited assumptions.
Tools and implementation options
An evaluation system does not require one particular vendor or framework.
A practical architecture can combine several layers.
Custom evaluation harness
Use Python, TypeScript, or your existing QA framework to execute prompts, preserve conversation state, capture model responses, inspect tool calls, and calculate deterministic metrics. Our guide on code review with Claude Code covers a related approach to building AI-assisted testing tooling.
This offers maximum control.
Model-platform evaluation tools
Platforms increasingly provide datasets, runs, and configurable graders. OpenAI’s evaluation APIs, for example, support reusable evaluations with data sources and testing criteria, with grader types that can include deterministic and model-based methods, documented at developers.openai.com. Note that OpenAI’s older standalone Evals dashboard is being retired (read-only from October 31, 2026, shut down November 30, 2026), so new work should target the current API-based evals and graders documentation rather than the legacy platform.
Independent safety benchmarks
External suites can supplement application-specific testing.
MLCommons’ AILuminate work provides standardized approaches for evaluating defined AI safety hazards, while its test-specification work emphasizes documenting test scope, languages, data, stakeholders, execution procedures, and metrics.
These benchmarks should supplement, not replace, tests designed for the application’s own risk profile.
Human evaluation platform
Use structured annotation workflows for cases requiring domain, cultural, fairness, or safety expertise.
Record reviewer disagreement rather than automatically forcing consensus; disagreement can reveal ambiguous product requirements.
CI/CD integration
Treat high-priority AI evaluations like software regression tests.
A deployment pipeline can:
build candidate → run eval suite → compare baseline → enforce critical gates → deploy → monitor
This makes behavioral changes visible before release.
Limitations and risks of AI evaluation
No evaluation suite can prove that an AI application will behave correctly for every possible interaction.
Several limitations remain.
Test sets are incomplete
Users will eventually produce scenarios that the development team did not anticipate.
Models are probabilistic
Passing once does not guarantee identical behavior on another generation.
Test contamination can distort results
A model may have encountered public benchmark material during training, making benchmark performance less representative of unseen production cases.
Automated judges can fail
An LLM grader is another model with its own limitations, biases, and failure modes.
Cultural judgments are contextual
There may be legitimate disagreement between reviewers, communities, and regions. MLCommons’ culturally specific evaluation work explicitly notes that perceptions of appropriate or harmful behavior can vary across linguistic and demographic contexts.
Aggregate scores hide severe failures
A 99% success rate may still be unacceptable if the remaining 1% contains high-severity safety or privacy incidents.
Evaluation should therefore support risk management rather than create the illusion that one benchmark score proves an AI system is universally “safe.”
Conclusion
Effective AI application testing requires more than measuring whether the model gives the “right” answer. A production evaluation program should independently test bias, safety boundaries, cultural fit, context retention, factuality, and tone consistency, while exercising the full system that users interact with.
The most effective approach is to define expected behavior first, create realistic and adversarial scenarios, use different grading methods for different assertions, validate automated evaluation against human judgment, and make important tests part of the release process. Most importantly, an evaluation suite should never be considered finished. Every confirmed production failure is an opportunity to create another regression test. Over time, this converts operational experience into a measurable and increasingly application-specific definition of trustworthy AI behavior.
Frequently Asked Questions
-
What is the difference between AI testing and traditional software testing?
Traditional tests frequently compare deterministic outputs with known expected results. Generative AI testing often evaluates acceptable ranges of behavior instead. Effective AI testing therefore combines exact assertions with semantic rubrics, human judgment, adversarial scenarios, and statistical analysis across repeated or varied inputs.
-
Should AI applications be tested before every release?
Material changes to the model, prompts, retrieval configuration, knowledge base, memory system, tools, permissions, or safeguards should trigger relevant regression evaluations. Critical suites can also run as deployment gates, while larger or more expensive evaluations may run on a scheduled cadence.
-
Can an LLM evaluate another LLM?
Yes. Model-based graders are useful for scalable semantic evaluation, but their judgments should be calibrated against human-labeled examples. For consequential evaluations, do not assume a model judge is unbiased or authoritative simply because its output is consistent.
-
How do you test hallucinations in an AI application?
Provide questions with authoritative reference material, decompose generated responses into factual claims, and check whether those claims are supported or contradicted by the evidence. Include deliberately unanswerable questions to verify that the system can acknowledge insufficient information instead of inventing an answer.
-
How do you measure safety without making the AI overly restrictive?
Measure unsafe compliance and over-refusal simultaneously. A strong safety system should reject prohibited assistance while continuing to answer nearby legitimate questions. Boundary testing is therefore as important as testing clearly harmful prompts.
-
Is context-window size enough to measure context retention?
No. Context-window size describes how much input a model can accept, not whether it will correctly identify and use every relevant detail. Evaluate retention directly using long conversations containing constraints, updates, distractors, conflicting facts, and information at different positions.
-
Who should evaluate cultural fit?
Use reviewers who understand the target language, locale, and use case. Automated graders can support evaluation, but culturally ambiguous cases should be grounded in explicit requirements and informed human judgment rather than generic assumptions.
-
What is the best metric for AI quality?
There is no universal single metric. NIST emphasizes that AI trustworthiness involves multiple characteristics and that metrics should reflect the application's context. A useful evaluation dashboard therefore keeps safety, factuality, fairness, context, cultural fit, and style metrics separately visible.
by Rajesh K | Sep 6, 2026 | API Testing, Blog, Latest Post |
API performance testing tells you whether your service can handle real traffic, not just whether it returns the right answer once. A single successful request proves almost nothing about how that same endpoint behaves under concurrent load, rising latency, or resource contention. This guide breaks performance down into three measurable dimensions, response time, throughput, and scalability, and shows how to define objective thresholds instead of vague goals like “the API should be fast.” You’ll also see a complete k6 example, common mistakes, and troubleshooting guidance for when results don’t match expectations.
What is API performance testing?
API performance testing measures how quickly, reliably, and efficiently an API processes requests as demand changes. The three core measurements are response time, which shows how long requests take; throughput, which shows how many requests the API processes per unit of time; and scalability, which shows whether acceptable performance can be maintained as traffic or system capacity increases.
A useful API performance test does more than generate traffic. It defines a realistic workload, measures latency distributions and errors, observes resource saturation, and determines the highest load the API can sustain while meeting its performance objectives.
Key takeaways
- Measure percentile response times such as p50, p95, and p99, not only averages, because averages can hide slow requests.
- Define throughput in requests per second (RPS) or another workload-specific unit and distinguish offered traffic from successfully completed traffic.
- Treat scalability as a relationship between load, latency, errors, throughput, and resource capacity, not as a single metric.
- Use an open workload model when request arrivals should remain independent of API response time.
- Establish explicit pass/fail thresholds before a test rather than deciding whether performance is acceptable afterward.
- Correlate load-test results with server-side CPU, memory, database, connection-pool, queue, and dependency telemetry to locate bottlenecks.
Google’s Site Reliability Engineering guidance similarly recommends monitoring latency, traffic, errors, and saturation for user-facing systems rather than interpreting latency in isolation.
What does API performance testing measure?
API performance testing evaluates the behavior of an application programming interface under controlled demand.
It commonly covers four related signals:
| S. No |
Metric |
What it answers |
Typical unit |
| 1 |
Response time / latency |
How long does a request take? |
ms or s |
| 2 |
Throughput |
How much work does the API complete? |
requests/sec, transactions/sec |
| 3 |
Error rate |
How often does processing fail? |
percentage or ratio |
| 4 |
Saturation |
Which resource approaches its limit? |
CPU %, memory, queue depth, connections |
OpenTelemetry, for example, defines the HTTP server metric http.server.request.duration as a histogram representing the duration of HTTP server requests.
API performance testing is different from functional testing. Functional testing asks whether an endpoint returns the correct result. Performance testing asks whether it continues returning correct results within acceptable timing and capacity constraints as demand changes.
The two should still be combined during a load test. A fast 500 Internal Server Error is not a successful performance result.
Why is API performance testing important?
An API can work correctly with one request and still fail badly under production traffic.
Performance problems commonly emerge when concurrency rises and finite resources begin to saturate. Examples include exhausted database connection pools, CPU contention, thread-pool limits, lock contention, downstream service delays, memory pressure, rate limits, and request queues.
These conditions have direct technical and business consequences:
- increasing API latency can slow mobile apps, web applications, and integrations;
- saturated queues can increase tail latency before outright errors appear;
- overload can cause timeouts, retries, and cascading traffic;
- insufficient capacity can make product launches or peak periods unreliable;
- overprovisioning without measurement can increase infrastructure cost.
Google SRE notes that overloaded queues increase request latency because requests spend longer waiting before processing. It also warns that retries can amplify traffic during failures and contribute to cascading failures.
Performance testing therefore helps answer two different questions:
Performance: Does the API meet its objectives at the expected workload?
Capacity: How much workload can the API sustain before those objectives are violated?
How do you measure API response time?
API response time is the elapsed time associated with processing an API request, measured between explicitly defined start and end points. Because different tools use different timing boundaries, teams should document exactly what their response-time metric includes.
For example, Apache JMeter defines elapsed time from immediately before sending a request until after the last response has been received. Its separate latency metric runs until the first part of the response has been received. Our JMeter Tutorial: An End-to-End Guide covers these definitions in more depth if you’re setting up JMeter for the first time.
Grafana k6 defines http_req_duration as:
http_req_sending + http_req_waiting + http_req_receiving
Its metric therefore excludes initial DNS lookup and connection-establishment time from http_req_duration; k6 exposes other timing metrics for connection and TLS activity, per k6’s metrics documentation.
This difference matters when comparing test results. A statement such as “the API responds in 180 ms” is incomplete unless the measurement boundary is known.
Break response time into components
Depending on the tool and protocol, investigate:
- DNS resolution
- TCP connection establishment
- TLS handshake
- request transmission
- server processing and queueing
- time to first byte
- response-body transfer
A rising total response time does not automatically mean application code became slower. Connection setup, network conditions, database calls, external APIs, or server queues can contribute.
Why p95 and p99 matter more than average response time
Suppose most API calls finish quickly but a small fraction take several seconds. An average may still appear acceptable.
Percentiles reveal the distribution.
- p50: 50% of requests complete at or below this duration.
- p95: 95% complete at or below this duration.
- p99: 99% complete at or below this duration.
Google SRE recommends considering percentiles because a mean can hide significant changes in tail latency and because latency distributions are not necessarily normally distributed.
For an interactive API, a performance requirement might therefore be expressed as:
95% of successful requests must complete within 300 ms and 99% within 600 ms under the defined peak workload.
Those numbers are examples, not universal recommendations. Appropriate thresholds depend on the API’s business purpose, architecture, client expectations, and existing service-level objectives.
How do you measure API throughput?
API throughput is the amount of request-processing work completed during a specified period.
For HTTP APIs, it is commonly reported as:
Throughput = Number of requests / Measurement duration
Apache JMeter uses this definition, calculating throughput from request count divided by elapsed test time.
Common units include:
- requests per second (RPS);
- requests per minute;
- transactions per second;
- records processed per second;
- megabytes per second for data-intensive APIs.
Grafana k6’s http_reqs metric counts generated HTTP requests and reports their rate, providing a direct view of generated request throughput.
Offered throughput versus successful throughput
Do not report only generated traffic.
Assume a load generator sends 1,000 requests per second, but 150 fail with errors or timeouts. The API should not be described simply as “handling 1,000 RPS.”
Track at least:
Offered load: traffic sent toward the API.
Completed throughput: requests receiving responses.
Successful throughput: requests that both complete and satisfy correctness criteria.
This distinction becomes particularly important around the system’s saturation point.
What is API scalability?
API scalability is the ability of a system to accommodate increasing demand while keeping response time, errors, and resource utilization within acceptable limits.
Scalability is therefore not synonymous with throughput.
An API may process more requests as load rises but simultaneously experience unacceptable p99 latency. Another system may maintain stable latency but stop increasing throughput because a database or worker pool has reached capacity.
A scalability test should examine the relationship:
Increasing load → latency → successful throughput → errors → resource saturation
For systems that support horizontal scaling, another dimension is:
Increasing resources → additional SLO-compliant capacity
AWS’s Well-Architected performance guidance recommends defining performance KPIs, monitoring performance-critical areas, and load testing workloads as part of performance engineering.
How does API performance testing work?
A repeatable API performance test generally follows this process:
- Define the workload. Identify endpoints, request mix, payload sizes, authentication behavior, and expected traffic.
- Define performance objectives. Specify response-time percentiles, error limits, and required throughput.
- Prepare representative data. Avoid unrealistic reuse of a single user, record, or cached request unless production behaves that way.
- Generate controlled traffic. Increase load according to the selected load model.
- Measure client-side results. Collect response duration, throughput, error rate, and dropped work.
- Observe server-side telemetry. Monitor CPU, memory, garbage collection, connections, queues, databases, caches, and downstream services.
- Find the constraint. Identify the resource or dependency associated with the point at which performance deteriorates.
- Repeat after a change. Compare results under the same test conditions.
The objective is not to produce the largest possible RPS number. It is to determine the amount of demand the system handles while still satisfying the defined service objectives.
Step-by-step: How to run an API performance test
1. Establish a baseline
Start with a small workload.
Confirm:
- requests are functionally correct;
- authentication works;
- test data is valid;
- responses pass assertions;
- the load generator itself is not resource-constrained;
- server telemetry is available.
A baseline gives you a reference against which higher-load behavior can be compared.
2. Define measurable performance thresholds
Avoid goals such as:
The API should be fast.
Use measurable criteria instead:
- p95 response time < 300 ms;
- p99 response time < 600 ms;
- HTTP failure rate < 1%;
- successful throughput ≥ 500 RPS.
Grafana k6 supports thresholds for percentile response times, errors, and custom metrics and can fail a test automatically when those conditions are violated.
Threshold values should come from business requirements, production SLOs, baselines, or capacity plans, not from arbitrary industry averages.
3. Model production traffic
Include representative:
- endpoint ratios;
- GET/POST/PUT/DELETE traffic;
- payload sizes;
- authenticated and anonymous sessions;
- cacheable and non-cacheable requests;
- test data;
- think time where relevant;
- geographic or network conditions where they affect the API.
Testing one inexpensive GET endpoint at maximum speed tells you little about a production workload dominated by writes, database transactions, and external calls.
4. Choose an appropriate load model
A closed model typically uses a fixed population of virtual users that waits for one iteration to finish before beginning another. Consequently, when the system slows down, iteration starts can also slow down.
An open model schedules arrivals independently of response time.
k6 specifically warns that closed-model throughput can fall when response times increase because iteration duration controls how quickly new iterations begin. Its arrival-rate executors use an open model so iteration starts can be controlled independently of system response time.
For capacity tests where production traffic arrives at an externally determined rate, an open workload model is often more representative.
5. Ramp load instead of immediately maximizing it
Increase demand in controlled stages.
For example:
50 → 100 → 200 → 400 → 600 → 800 RPS
Hold each level long enough to observe stable behavior.
At every stage, record:
- p50, p95, and p99 response times;
- successful RPS;
- error percentage;
- CPU utilization;
- memory and garbage collection;
- database latency;
- connection-pool usage;
- queue depth;
- downstream latency;
- instance count.
The point at which an objective first fails is more useful than a single maximum-load measurement.
6. Find the SLO-compliant capacity
Define API capacity as the highest sustained workload that still satisfies all required performance and correctness thresholds.
For example, if the API meets every requirement at 500 RPS but p95 response time crosses its objective at 600 RPS, its tested SLO-compliant capacity in that environment lies below 600 RPS.
That result is specific to the tested:
- software version;
- infrastructure;
- dataset;
- workload mix;
- configuration;
- test duration.
It should not be presented as a universal capacity figure.
7. Test scaling behavior
Repeat the capacity test after changing capacity.
For example:
- one application instance;
- two instances;
- four instances.
Then compare the maximum SLO-compliant throughput.
A useful internal heuristic is:
Scalability efficiency = throughput growth factor / resource growth factor
If doubling an application tier from two to four instances increases SLO-compliant capacity from 500 to 900 RPS:
(900 / 500) ÷ (4 / 2) = 0.90
The resulting 90% is not an industry-standard scalability metric. It is simply a useful engineering ratio for comparing your own scaling experiments.
Sublinear scaling can indicate shared bottlenecks such as a database, cache, network link, lock, or downstream dependency.
Practical example: Load testing a product API with k6
Consider a retail API:
GET /v1/products
Assume the engineering team has established these illustrative performance objectives:
- p95 response time below 300 ms;
- p99 response time below 600 ms;
- HTTP failure rate below 1%.
The test should increase request arrival rate independently of API response time.
import http from 'k6/http';
import { check } from 'k6';
const BASE_URL = __ENV.BASE_URL;
export const options = {
scenarios: {
product_api: {
executor: 'ramping-arrival-rate',
startRate: 50,
timeUnit: '1s',
preAllocatedVUs: 200,
maxVUs: 1000,
stages: [
{ target: 100, duration: '2m' },
{ target: 250, duration: '3m' },
{ target: 500, duration: '5m' },
{ target: 750, duration: '5m' },
],
},
},
thresholds: {
'http_req_duration{endpoint:products}': [
'p(95)<300',
'p(99)<600',
],
http_req_failed: ['rate<0.01'],
},
};
export default function () {
const response = http.get(
`${BASE_URL}/v1/products?limit=20`,
{
tags: {
endpoint: 'products',
},
}
);
check(response, {
'status is 200': (r) => r.status === 200,
});
}
k6’s ramping-arrival-rate executor changes the iteration arrival rate over time, while its thresholds allow percentile-duration and error criteria to be evaluated automatically.
Expected output
Do not expect a particular RPS or latency result in advance.
Instead, the expected outcome is a clear answer to these questions:
- At which load stage does p95 first exceed 300 ms?
- When does p99 exceed 600 ms?
- Does the error rate remain below 1%?
- Does successful throughput continue increasing with offered load?
- Which server resource saturates first?
- Are scheduled iterations being dropped by the load generator?
k6 exposes dropped_iterations when scheduled iterations cannot start. With arrival-rate executors, persistent dropped iterations can occur when there are insufficient available VUs, including situations where system-under-test performance degrades and iterations take increasingly long to finish.
Error condition
Suppose latency rises sharply at the 750-RPS stage while database pool utilization reaches its configured maximum.
The appropriate conclusion is not simply:
k6 cannot generate 750 RPS.
First verify load-generator capacity. If it is healthy, correlate the change with server telemetry. Connection-pool saturation may be the bottleneck, or it may only be a symptom of slower database queries.
The next experiment should isolate that hypothesis.
Response time vs. latency vs. throughput vs. scalability
| S. No |
Factor |
Response time |
Latency |
Throughput |
Scalability |
| 1 |
Primary question |
How long did the request take? |
How much delay occurred? |
How much work was processed? |
How does performance change as demand/capacity grows? |
| 2 |
Typical unit |
ms, s |
ms, s |
RPS, TPS |
Relationship or capacity curve |
| 3 |
Measurement type |
Duration |
Duration |
Rate |
Multi-metric behavior |
| 4 |
Useful statistics |
p50, p95, p99 |
p50, p95, p99 |
average/sustained rate |
SLO-compliant capacity |
| 5 |
Main failure signal |
Tail times rise |
Delay rises |
Throughput plateaus/falls |
Added load/resources produce poor scaling |
| 6 |
Should be analyzed alone? |
No |
No |
No |
No |
Terminology varies between tools. Apache JMeter, for example, distinguishes elapsed time from its first-response latency metric, while some engineering discussions use “latency” more broadly for overall request duration.
Document your exact definition before comparing measurements.
API performance testing best practices
Define the measurement boundary
State whether response time includes DNS, TCP/TLS setup, redirects, response-body transfer, client processing, or only server-side duration.
This prevents misleading comparisons between tools and dashboards.
Measure percentiles, not only averages
Track at least p50 and one or more high percentiles such as p95 or p99.
Tail latency often exposes queuing, contention, garbage collection, or slow dependencies that averages obscure. Google SRE explicitly recommends care when aggregating latency and discusses the value of high percentiles.
Validate correctness under load
Check status codes and important response content.
A performance test should not classify malformed, stale, or incorrect responses as successful just because they are fast.
Separate endpoints and workload classes
A single global p95 can hide an endpoint that performs poorly.
Segment metrics by:
- route;
- method;
- response status;
- payload class;
- region;
- customer or workload class where appropriate.
OpenTelemetry HTTP conventions include attributes such as HTTP request method and route-related telemetry that can support this type of analysis.
Test beyond expected peak load
Expected-peak testing answers whether capacity is sufficient.
Testing above expected peak reveals the degradation pattern: gradual slowdown, rate limiting, timeouts, queue growth, or abrupt collapse.
Google SRE describes testing services beyond rated capacity so overload behavior can be understood before production experiences it.
Monitor the load generator
Verify that the client running the test has sufficient:
- CPU;
- memory;
- network bandwidth;
- sockets;
- file descriptors;
- virtual users or workers.
Otherwise you may measure the test infrastructure instead of the API.
Repeat tests under controlled conditions
Record:
- build or commit;
- environment;
- instance sizes;
- autoscaling configuration;
- database size;
- cache state;
- tool version;
- workload profile;
- duration.
A result without its test environment is difficult to reproduce.
Common API performance testing mistakes
| S. No |
Mistake |
Why it happens |
Impact |
Recommended fix |
| 1 |
Reporting only average latency |
Average is easy to read |
Tail problems remain hidden |
Track p50/p95/p99 |
| 2 |
Ignoring errors |
RPS appears impressive |
Failed traffic counts as capacity |
Report successful throughput and errors |
| 3 |
Using unrealistic endpoint mixes |
Scripts are simplified |
Test differs from production |
Model actual traffic distribution |
| 4 |
Maximum-load testing only |
Teams want one capacity number |
Breakpoint behavior is unclear |
Increase load in controlled stages |
| 5 |
Using a closed model unintentionally |
Default VU behavior is convenient |
Slower API can reduce generated arrival rate |
Use arrival-rate testing where appropriate |
| 6 |
Testing from one cached dataset |
Setup is easier |
Cache hit rates become unrealistic |
Use representative data variation |
| 7 |
Ignoring dependencies |
Focus remains on application CPU |
Real bottleneck is missed |
Correlate database, cache, queue, and downstream metrics |
| 8 |
Comparing tools without timing definitions |
Metrics share similar names |
Results are not equivalent |
Document measurement boundaries |
Troubleshooting API performance tests
Why does p99 increase while average response time stays stable?
A subset of requests is becoming much slower.
Check the latency distribution rather than the mean and correlate slow requests with database queries, garbage collection, queueing, downstream dependencies, request types, and resource saturation.
High percentiles are designed to expose this long-tail behavior.
Why does throughput stop increasing when more virtual users are added?
The system or workload model may have reached a limiting factor.
Check:
- whether response times are increasing;
- CPU and memory;
- database and connection pools;
- queues and thread pools;
- network limits;
- rate limiting;
- load-generator saturation.
With a closed workload model, increasing response time can itself reduce the rate at which new work begins.
Why are errors increasing only at high load?
The system may be crossing a capacity boundary.
Inspect status codes and distinguish:
- application failures;
- 429 Too Many Requests;
- gateway errors;
- timeouts;
- connection failures;
- database exhaustion;
- downstream failures.
Then correlate the first rise in errors with saturation telemetry.
Why are k6 iterations being dropped?
For an arrival-rate scenario, k6 can report dropped iterations when no VU is available to begin scheduled work.
If drops occur immediately, the test configuration may need more preallocated VUs. If they rise later as latency increases, the system under test may be degrading enough that virtual users remain occupied longer.
Why are load-test results inconsistent between runs?
Look for uncontrolled variables:
- autoscaling state;
- cache warming;
- database data volume;
- noisy infrastructure;
- deployment differences;
- background jobs;
- external service behavior;
- client-machine capacity;
- network location.
Make these conditions explicit in the test report and repeat enough controlled runs to determine whether the difference is reproducible.
Which tools can be used for API performance testing?
Grafana k6
k6 is useful for code-based performance tests and supports virtual-user and arrival-rate workload models, built-in HTTP metrics, thresholds, checks, and multiple scenario executors.
It is particularly convenient when performance tests are managed alongside application code and run in automated pipelines. If you’re deciding between k6 and JMeter, see our comparison, JMeter vs Gatling vs k6: Comparing Top Performance Testing Tools.
Apache JMeter
Apache JMeter provides a mature GUI- and configuration-driven approach and reports metrics including elapsed time, latency, errors, percentiles, and throughput. Its documentation defines throughput as request count divided by total measurement time.
It can be useful for teams that prefer a visual test-plan model or already maintain an established JMeter test suite. If you need to scale that setup beyond a single machine, our Cloud Performance Testing with Apache JMeter guide covers distributed testing.
OpenTelemetry
OpenTelemetry is not a replacement for a load generator. It is useful for instrumenting the system under test so that load-test traffic can be correlated with server and dependency telemetry.
Its HTTP semantic conventions define standardized metrics including http.server.request.duration and http.client.request.duration.
A strong performance-testing environment commonly combines a traffic generator with observability rather than relying on either one alone. For a broader roundup of options beyond these three, see our Top Performance Testing Tools guide.
Limitations and risks of API performance testing
Performance testing is an experiment, not a perfect forecast of production.
Results can be distorted by differences in:
- data distribution;
- cache behavior;
- infrastructure;
- geography;
- dependency performance;
- traffic composition;
- request bursts;
- client behavior;
- autoscaling;
- production background workloads.
There is also a risk in testing production directly. High-volume tests can affect real customers, trigger external API costs, consume quotas, alter data, or activate security controls.
Use isolated environments when required, sanitize test data, obtain authorization for production testing, and understand downstream rate limits before generating substantial traffic.
Another limitation is that test results age. Capacity measured several releases ago may no longer represent the current architecture. Google SRE specifically recommends using load testing rather than relying on historical resource-to-capacity assumptions.
Conclusion
Effective API performance testing answers more than “How fast is this endpoint?” It establishes how response-time percentiles change as demand grows, how much successful throughput the API can sustain, where errors begin, which resources saturate, and whether adding capacity produces useful scaling.
Start by defining a realistic workload and explicit performance objectives. Measure p95 and p99 alongside throughput and errors, use a workload model that reflects how traffic actually arrives, and correlate every load stage with server-side telemetry.
The most useful outcome is a repeatable capacity boundary: the highest sustained workload your API can handle while still meeting its defined service objectives. Once that baseline exists, performance testing becomes a regression-detection and capacity-planning discipline rather than a one-time benchmark.
Frequently Asked Questions
-
What is a good API response time?
There is no universal response-time target for all APIs. A suitable objective depends on the endpoint's purpose, user expectations, downstream dependencies, payload size, network path, and business requirements. Define targets as percentiles, such as p95 and p99, under a specified workload rather than relying only on a generic average.
-
What is the difference between response time and throughput?
Response time measures how long individual requests take, while throughput measures how many requests or transactions are processed during a period. An API can have low response times at light load but poor maximum throughput, or high throughput accompanied by unacceptable tail latency. Both metrics should therefore be evaluated together.
-
How do you measure API scalability?
Increase demand in controlled stages, record latency, errors, successful throughput, and saturation, and identify the highest workload that still meets the performance objectives. Then repeat the same experiment with additional computing capacity. Comparing SLO-compliant capacity across configurations shows how efficiently the API scales.
-
Should API performance tests use p95 or p99?
Use percentiles that match the consequences of slow requests for your service. p95 is often useful for understanding broader tail behavior, while p99 exposes a smaller, slower portion of traffic. Critical systems may monitor multiple percentiles. The correct choice should come from service objectives rather than adopting a percentile simply because a testing tool reports it.
-
How many virtual users are needed for an API load test?
There is no fixed number. The required virtual-user count depends on the workload model, request duration, think time, desired request-arrival rate, and test-tool implementation. When the business requirement is expressed as RPS, an arrival-rate workload can be easier to reason about than selecting an arbitrary virtual-user count.
-
Is load testing the same as stress testing?
No. Load testing typically verifies behavior at expected or planned demand. Stress testing intentionally increases demand toward or beyond capacity to identify breaking points and degradation behavior. A mature performance program often uses both.
-
How often should API performance tests run?
Run lightweight performance checks frequently enough to catch regressions and perform more expensive capacity tests when changes can materially affect performance, for example, major releases, infrastructure changes, database migrations, dependency changes, or significant traffic-growth events. The exact cadence depends on test cost and release frequency.
by Rajesh K | Sep 5, 2026 | API Testing, Blog, Latest Post |
API automation testing gives testers a faster way to validate business logic, data contracts, authentication, error handling, and service integrations without waiting for a user interface. For many QA teams, the harder question is not whether to automate APIs, but which tool to use. Postman, REST Assured, and Playwright can all automate REST API tests, but they fit different workflows. Postman is collection-oriented and accessible to mixed-skill QA teams, REST Assured is designed around Java test code, and Playwright allows API tests to live alongside browser automation.
This guide explains each approach from a tester’s perspective and uses the same API scenario throughout so you can compare implementation, maintainability, debugging, and CI/CD usage directly.
Version note: This article was technically reviewed against Postman documentation v12, REST Assured 6.0.1, and @playwright/test 1.62.1 as available on September 2, 2026.
What is API automation testing?
API automation testing is the use of executable tests to send requests to an API, inspect its responses, and automatically verify expected behavior such as status codes, response data, schemas, authentication, and business rules.
Unlike UI automation, API tests communicate directly with application services. This generally lets a tester identify whether a failure belongs to the backend contract or to the user-interface layer before debugging the complete application stack.
Key takeaways
- Use Postman when testers need fast API exploration, readable collections, reusable environments, and a relatively low barrier to automation.
- Use REST Assured when the automation stack is Java-based and API tests need to behave like maintainable application code.
- Use Playwright when the team uses JavaScript or TypeScript and wants API and browser tests in the same automation project.
- Validate more than HTTP status codes: assert business data, headers, required fields, negative behavior, and contracts where appropriate.
- Keep authentication secrets and environment-specific URLs outside test code.
- Avoid implementing the same regression suite independently in all three tools. Select a primary automation framework and use other tools where they add distinct value.
Why does API automation testing matter to testers?
API defects frequently affect multiple application layers at once.
Consider an e-commerce checkout. A button may work correctly in the browser while the order API:
- creates duplicate orders,
- calculates an incorrect total,
- accepts invalid product IDs,
- exposes fields that should not be returned,
- returns 200 OK even though the operation failed,
- accepts an expired authentication token, or
- stores incorrect data that appears only in a later workflow.
UI automation alone can detect some of these failures, but it normally observes them indirectly.
API automation lets testers inspect the service contract itself. If you’re new to this area, our REST API Testing Checklist is a good companion resource for the fundamentals this guide builds on.
A practical API suite can validate:
- HTTP behavior: status codes, headers, methods, redirects.
- Data correctness: returned values, calculated fields, IDs and state changes.
- Contract correctness: required properties, types and response structure.
- Authentication and authorization: valid, invalid, missing and insufficient credentials.
- Negative behavior: malformed payloads, missing parameters and unsupported operations.
- Workflow behavior: create a resource, retrieve it, update it and delete it.
- Integration behavior: whether one operation produces the state required by the next system.
For a tester, that means defects can often be isolated before a browser, mobile client, or other consumer becomes part of the investigation.
How does API automation testing work?
A typical automated API test follows this flow:
Test data
↓
Request
↓
API
↓
Response
↓
Assertions
↓
Cleanup/reporting
For example:
- Generate a unique email address for the test.
- Send POST /users.
- Verify that the server returns 201.
- Verify that the response contains the expected name and email.
- Extract the generated user ID.
- Send GET /users/{id}.
- Confirm that the stored record matches the original request.
- Send DELETE /users/{id} during cleanup.
- Report the result to the test runner or CI pipeline.
The mechanics are almost identical in Postman, REST Assured, and Playwright. The main difference is how the test is represented and maintained.
Practical API scenario used in this guide
Assume the application under test exposes a user-management API.
Preconditions
Base URL:
Authentication:
Authorization: Bearer <token>
Create user:
Request:
Expected response:
{
"id": "usr_12345",
"name": "Asha Tester",
"email": "[email protected]",
"status": "active"
}
Expected HTTP status:
Retrieve user:
Delete user:
The API in this scenario is illustrative. Replace its URL, fields, authentication method, and expected responses with the contract of your application.
How to automate API testing with Postman
Postman allows testers to add JavaScript post-response scripts to requests, folders, or collections. Assertions use APIs such as pm.test, pm.expect, and pm.response. Collections can then be executed interactively, through the Collection Runner, or from CI/CD using the Postman CLI.
1. Create the Postman environment
Create an environment called QA with:
baseUrl = https://api.example.com
token = <runtime token>
Reference them in requests as:
Keep reusable configuration such as URLs separate from test assertions. For credentials, prefer your organization’s approved secrets mechanism rather than committing tokens to exported collections or source control.
2. Create the POST request
Method:
URL:
Authorization header:
Authorization: Bearer {{token}}
Body:
3. Add response assertions
In Scripts → Post-response, add:
pm.test("Create user returns 201", () => {
pm.response.to.have.status(201);
});
pm.test("Response is JSON", () => {
pm.response.to.be.json;
});
const body = pm.response.json();
pm.test("Created user contains an ID", () => {
pm.expect(body.id).to.be.a("string").and.not.empty;
});
pm.test("Created user has expected values", () => {
pm.expect(body.name).to.eql("Asha Tester");
pm.expect(body.email).to.eql("[email protected]");
pm.expect(body.status).to.eql("active");
});
pm.collectionVariables.set("userId", body.id);
Postman’s pm.response exposes the status code, headers, response time and parsed response body, while pm.test and pm.expect provide assertion support, per Postman’s scripting documentation.
Expected result
The request should:
- return 201,
- produce JSON,
- return the expected user details, and
- store the generated ID in userId.
That ID can now be consumed by the next request.
4. Validate the response contract
For APIs where response structure matters, add a schema assertion:
const schema = {
type: "object",
required: ["id", "name", "email", "status"],
properties: {
id: { type: "string" },
name: { type: "string" },
email: { type: "string" },
status: { type: "string" }
}
};
pm.test("Response matches the user schema", () => {
pm.response.to.have.jsonSchema(schema);
});
Postman’s current response API supports JSON Schema validation through pm.response...jsonSchema(...).
A schema test complements business assertions; it should not replace them. A response can match the schema and still contain incorrect business values.
5. Retrieve the user
Create:
GET {{baseUrl}}/users/{{userId}}
Then add:
pm.test("Get user returns 200", () => {
pm.response.to.have.status(200);
});
const body = pm.response.json();
pm.test("Retrieved user is the created user", () => {
pm.expect(body.id).to.eql(pm.collectionVariables.get("userId"));
pm.expect(body.name).to.eql("Asha Tester");
});
6. Delete the test user
Create:
DELETE {{baseUrl}}/users/{{userId}}
Assert the status expected by your API, for example:
pm.test("Delete user succeeds", () => {
pm.expect(pm.response.code).to.be.oneOf([200, 204]);
});
Do not assume both codes are correct for your application. The expected status should come from the API contract.
7. Run the Postman suite from the command line
The current Postman CLI supports collection files or collection IDs and accepts an environment through --environment or -e.
For a local exported collection:
postman collection run ./User-API.postman_collection.json \
-e ./QA.postman_environment.json
A CI pipeline can execute the same command and fail when test assertions fail.
When Postman works particularly well
Postman is a strong fit when:
- manual testers are moving gradually into automation,
- developers and testers collaborate around shared API examples,
- exploratory requests need to become regression checks,
- collections also serve as executable documentation,
- non-Java teams need an API testing solution without building a complete test framework first.
How to automate API testing with REST Assured
REST Assured is a Java DSL for testing REST services. Its commonly used syntax follows a readable given → when → then structure, and it supports request configuration, authentication, response assertions, extraction and JSON Schema validation, per the REST Assured getting-started documentation.
The REST Assured project’s current getting-started documentation uses version 6.0.1.
1. Add REST Assured to the Java test project
Assuming JUnit 5 or another Java test framework is already configured:
<dependency>
<groupId>io.rest-assured</groupId>
<artifactId>rest-assured</artifactId>
<version>6.0.1</version>
<scope>test</scope>
</dependency>
REST Assured includes its JsonPath and XmlPath support transitively. A separate json-schema-validator module is available when schema validation is required.
2. Build reusable request configuration
import io.restassured.http.ContentType;
import io.restassured.specification.RequestSpecification;
import io.restassured.builder.RequestSpecBuilder;
import org.junit.jupiter.api.BeforeAll;
class UserApiTest {
private static RequestSpecification api;
@BeforeAll
static void configureApi() {
api = new RequestSpecBuilder()
.setBaseUri(System.getenv("API_BASE_URL"))
.addHeader(
"Authorization",
"Bearer " + System.getenv("API_TOKEN")
)
.setContentType(ContentType.JSON)
.build();
}
}
This keeps base URL, authentication, and content type out of individual test cases.
In CI, configure:
as pipeline variables or secrets.
3. Automate create, retrieve, and delete
import org.junit.jupiter.api.Test;
import java.util.Map;
import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.*;
class UserApiTest {
// api specification configured as shown above
@Test
void userCanBeCreatedRetrievedAndDeleted() {
String email =
"asha+" + System.currentTimeMillis() + "@example.com";
String userId =
given()
.spec(api)
.body(Map.of(
"name", "Asha Tester",
"email", email
))
.when()
.post("/users")
.then()
.statusCode(201)
.contentType(ContentType.JSON)
.body("id", not(emptyOrNullString()))
.body("name", equalTo("Asha Tester"))
.body("email", equalTo(email))
.body("status", equalTo("active"))
.extract()
.path("id");
given()
.spec(api)
.pathParam("userId", userId)
.when()
.get("/users/{userId}")
.then()
.statusCode(200)
.body("id", equalTo(userId))
.body("email", equalTo(email));
given()
.spec(api)
.pathParam("userId", userId)
.when()
.delete("/users/{userId}")
.then()
.statusCode(anyOf(is(200), is(204)));
}
}
REST Assured supports response extraction, JsonPath-style access, request specifications and assertions over status, headers, body and other response information.
Why generate a unique email?
Hard-coded test data creates avoidable failures.
If every run attempts:
the second execution may fail because the first execution already created it.
Generating unique data reduces collisions, especially when tests run concurrently.
4. Add schema validation when required
Add the REST Assured module:
<dependency>
<groupId>io.rest-assured</groupId>
<artifactId>json-schema-validator</artifactId>
<version>6.0.1</version>
<scope>test</scope>
</dependency>
Then:
import static io.restassured.module.jsv.JsonSchemaValidator
.matchesJsonSchemaInClasspath;
given()
.spec(api)
.when()
.get("/users/{userId}", userId)
.then()
.statusCode(200)
.body(matchesJsonSchemaInClasspath("user-schema.json"));
REST Assured documents JSON Schema validation as a separate module and provides the matchesJsonSchemaInClasspath matcher for classpath schemas.
5. Run the REST Assured suite in CI
For a Maven-based project:
For larger suites, use test tags or naming conventions to separate smoke, regression, integration, and destructive tests.
When REST Assured works particularly well
Choose REST Assured when:
- the organization’s engineering stack is Java,
- testers are comfortable maintaining code,
- tests require reusable Java utilities and domain objects,
- API automation must integrate deeply with JUnit or TestNG,
- source-control review and conventional software engineering practices are priorities.
How to automate API testing with Playwright
Playwright provides APIRequestContext specifically for direct HTTP(S) requests, per Playwright’s API documentation. Playwright Test also includes a built-in request fixture that can inherit configuration such as baseURL and extraHTTPHeaders, as described in the API testing overview.
This is particularly useful for teams that already use Playwright for web UI testing because the API and browser layers can share one runner, configuration model, fixtures, reporting strategy, and programming language.
1. Configure the API connection
In playwright.config.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
use: {
baseURL: process.env.API_BASE_URL,
extraHTTPHeaders: {
Authorization: `Bearer ${process.env.API_TOKEN}`,
'Content-Type': 'application/json'
}
}
});
Playwright officially documents both baseURL and extraHTTPHeaders as configuration consumed by the API request fixture.
2. Write the API workflow
Create:
Then:
import { test, expect } from '@playwright/test';
test('user can be created, retrieved, and deleted',
async ({ request }) => {
const email =
`asha+${Date.now()}@example.com`;
const createResponse = await request.post('/users', {
data: {
name: 'Asha Tester',
email
}
});
expect(createResponse.status()).toBe(201);
const createdUser = await createResponse.json();
expect(createdUser).toMatchObject({
name: 'Asha Tester',
email,
status: 'active'
});
expect(createdUser.id).toEqual(expect.any(String));
const getResponse =
await request.get(`/users/${createdUser.id}`);
expect(getResponse.status()).toBe(200);
const retrievedUser = await getResponse.json();
expect(retrievedUser).toMatchObject({
id: createdUser.id,
name: 'Asha Tester',
email
});
const deleteResponse =
await request.delete(`/users/${createdUser.id}`);
expect([200, 204]).toContain(deleteResponse.status());
});
Playwright also provides expect(response).toBeOK(), which verifies that an API response is in the 200–299 range. When the exact contract requires 201, 204, or another specific status, an exact status() assertion communicates the requirement more precisely.
3. Use APIs to create UI test state
One of Playwright’s most useful patterns for testers is:
API setup → UI action → API verification
For example:
test('newly created user appears in admin UI',
async ({ request, page }) => {
const createResponse = await request.post('/users', {
data: {
name: 'UI API Test User',
email: `ui-api-${Date.now()}@example.com`
}
});
expect(createResponse.status()).toBe(201);
const user = await createResponse.json();
await page.goto('/admin/users');
await expect(
page.getByText(user.email)
).toBeVisible();
});
This pattern avoids navigating through several UI screens merely to establish test data.
Playwright explicitly documents API requests as a way to establish server-side preconditions before UI tests and validate server-side postconditions after browser actions.
4. Understand API and browser authentication sharing
page.request and browserContext.request share cookie storage with their browser context. If the test requires completely separate authentication state, Playwright can create an isolated APIRequestContext using apiRequest.newContext().
This distinction matters when testing:
- multiple users,
- authorization boundaries,
- anonymous versus authenticated behavior,
- admin versus customer roles.
5. Run Playwright API tests
Run the entire suite:
Or keep API tests in a dedicated folder or project so they can run separately from browser tests. Once your suite is running in CI, StageWright is worth a look for making the resulting reports easier to triage.
When Playwright works particularly well
Choose Playwright for API testing when:
- the automation project already uses Playwright,
- testers work in JavaScript or TypeScript,
- API setup and UI validation frequently appear in the same scenario,
- one test runner for API and browser automation simplifies the pipeline,
- API state needs to be created quickly before E2E tests.
Postman vs REST Assured vs Playwright for API testing
| S. No |
Factor |
Postman |
REST Assured |
Playwright |
| 1 |
Primary style |
Collections + JavaScript scripts |
Java code/DSL |
TypeScript/JavaScript test code |
| 2 |
Best fit |
Mixed-skill API teams |
Java automation teams |
Web/API automation teams |
| 3 |
Beginner accessibility |
High |
Moderate |
Moderate |
| 4 |
Strong code architecture |
Possible, but collection-oriented |
Strong |
Strong |
| 5 |
UI + API in one framework |
Limited compared with dedicated browser frameworks |
Requires another UI tool |
Strong |
| 6 |
Interactive API exploration |
Excellent |
Code-first |
Code-first |
| 7 |
CI/CD execution |
Postman CLI |
Maven/Gradle |
Playwright Test CLI |
| 8 |
Reusable environments |
Built in |
Framework/configuration code |
Playwright config/fixtures |
| 9 |
JSON Schema validation |
Supported |
Dedicated module |
Typically add a schema validator when required |
| 10 |
Request/response debugging |
Very accessible UI |
Logs/debugger/IDE |
Test runner, traces/logging and debugger |
| 11 |
Ideal tester profile |
Manual/API tester moving into automation |
Java SDET/automation engineer |
JS/TS QA engineer or Playwright user |
| 12 |
Main tradeoff |
Complex collections can become difficult to govern like code |
Requires Java development skills |
API-only teams may not need the browser-focused ecosystem |
The tools overlap, but they are not interchangeable in every organization.
The right choice depends more on team workflow and maintainability than on whether a tool can technically send a GET or POST request. If your services communicate over gRPC instead of REST, the tooling picture changes further see our gRPC API Testing guide for that scenario.
Which API automation tool should testers choose?
Use this decision framework.
Choose Postman when the workflow begins with exploratory API testing
Postman works well when testers first inspect endpoints manually, experiment with payloads, save examples, and progressively add automation.
A collection can become the bridge between:
exploration
↓
documentation
↓
regression testing
↓
CI
Choose REST Assured when Java is already the engineering standard
If developers and test engineers work primarily in Java, introducing a JavaScript-based API framework may create unnecessary fragmentation.
REST Assured allows the API suite to use familiar:
- Java models,
- build tools,
- assertion libraries,
- test runners,
- code-review workflows,
- dependency management.
Choose Playwright when API and UI automation belong together
If the same QA team owns browser automation and service validation, Playwright can reduce framework duplication.
A tester can create data through an API, execute a browser workflow, and query the API afterward to confirm persistence, all within one test.
A practical mixed-tool strategy
A mature team might use:
Postman
↓
exploration, shared requests, examples and troubleshooting
REST Assured OR Playwright
↓
primary automated regression suite
The important word is OR.
Maintaining identical regression suites in Postman, REST Assured, and Playwright usually creates three versions of the same testing problem.
Use multiple tools only when they perform distinct jobs.
Best practices for API automation testing
1. Test business behavior, not just status codes
This is too weak:
expect(response.status()).toBe(200);
A response can return 200 with:
{
"success": false,
"error": "Order was not created"
}
Verify the fields that prove the operation succeeded.
2. Design negative tests deliberately
For a create-user endpoint, cover cases such as:
Missing email
Invalid email
Duplicate email
Missing token
Expired token
Read-only user token
Malformed JSON
Unsupported Content-Type
Unexpected additional field
Maximum-length input
Negative cases frequently reveal contract inconsistencies that happy-path automation misses. Payment-related endpoints deserve particularly deliberate negative coverage — see our guide on testing payment APIs for how duplicate-charge and idempotency scenarios fit into this same pattern.
3. Separate test configuration from test logic
Do not hardcode:
https://qa.internal.company.example
throughout hundreds of tests.
Use:
baseUrl
API_BASE_URL
environment configuration
depending on the framework.
This lets the same suite target controlled environments without rewriting test logic.
4. Generate isolated test data
Prefer:
over:
when duplicate values are prohibited.
Parallel execution becomes much safer when each test owns its resources.
5. Clean up created data
If a test creates:
users
orders
projects
accounts
subscriptions
delete them where the environment and business rules allow it.
Test pollution eventually makes failures harder to reproduce.
6. Make tests independently executable
A regression test should not require:
Test 14 must execute before Test 15.
Instead, create required state through fixtures, setup methods, APIs, or dedicated test-data helpers.
Collection workflows sometimes intentionally chain requests, but independent regression scenarios remain easier to parallelize and diagnose.
7. Assert contracts selectively
Avoid comparing an entire dynamic response literally:
{
"id": "123",
"timestamp": "2026-09-02T04:45:15Z",
"requestId": "xyz"
}
when several values change on every request.
Assert stable business requirements and validate structural fields separately.
8. Treat response-time checks carefully
A functional API framework can identify extreme latency or enforce a coarse threshold.
It does not replace a controlled performance test.
Client location, network latency, framework overhead, server warm-up and CI infrastructure can all influence individual request timings. REST Assured’s own documentation similarly notes that measured response time includes more than server processing alone.
Common API automation mistakes
| S. No |
Mistake |
Why it happens |
Impact |
Recommended fix |
| 1 |
Checking only 200 |
Test is quick to write |
Functional defects pass unnoticed |
Assert business data and state |
| 2 |
Hardcoding tokens |
Convenient locally |
Security risk and CI failures |
Inject secrets at runtime |
| 3 |
Reusing the same test data |
Easy initial setup |
Duplicate-data failures |
Generate isolated data |
| 4 |
Depending on test execution order |
Tests share state |
Flaky parallel runs |
Make tests self-contained |
| 5 |
Skipping cleanup |
Cleanup seems nonessential |
Environment becomes polluted |
Delete created resources safely |
| 6 |
Retrying every failure |
Retries hide instability |
Defects become flaky “passes” |
Retry only appropriate transient failures |
| 7 |
Asserting the full JSON response |
Appears comprehensive |
Dynamic fields break tests |
Assert stable fields + contract |
| 8 |
Treating API tests as security tests |
Functional checks exercise auth |
Security coverage is overstated |
Maintain separate security testing |
| 9 |
Treating API latency assertions as load tests |
Both measure time |
Misleading performance conclusions |
Use a performance-testing tool for load |
Troubleshooting API automation failures
Why does the API return 401 even though the token looks valid?
The token may be valid syntactically but invalid for the target service.
Check:
- token expiration,
- issuer,
- audience,
- scopes or roles,
- environment,
- authorization header format,
- whether the token belongs to a different API host.
A token that succeeds against development may legitimately fail against QA.
Why does the test work locally but fail in CI?
Start by comparing environmental differences.
Check:
Base URL
Secret availability
Proxy configuration
DNS
Firewall rules
Certificates
Environment variables
Test data
Time zone
Parallel execution
For Postman specifically, confirm that the environment file or environment UID supplied to the CLI is the intended one. For Playwright and REST Assured, verify that required environment variables exist in the CI job rather than only in the developer’s terminal.
Why does a test pass alone but fail in the complete suite?
The most common cause is shared mutable state.
Look for:
A shared user
A shared cart
A shared database record
A globally modified token
A reused collection variable
A fixed email address
Tests relying on execution order
Run the suite concurrently and log the identifiers each test creates. If two tests modify the same resource, isolate the data.
Why does my API test return the correct status but the assertion still fails?
Inspect the response type before comparing values.
Common mismatches include:
"10" versus 10
null versus missing field
false versus "false"
UTC timestamp versus local timestamp
array versus object
Do not convert every value to a string simply to make the assertion pass. Confirm what the API contract says the type should be.
Why does an API-created record not immediately appear in the UI?
The system may use asynchronous processing or eventual consistency.
For example:
POST order
↓
message queue
↓
background processor
↓
search index
↓
UI query
The API can successfully accept a request before another data store becomes current.
Verify the architecture before adding arbitrary sleeps. Prefer polling for the expected state with a defined timeout.
Why do Playwright API calls behave as though the browser is already logged in?
If you use page.request or browserContext.request, the API request context shares cookies with that browser context.
Create a separate apiRequest.newContext() when the API test needs isolated authentication or anonymous state.
How should API tests be organized in CI/CD?
A practical pipeline often separates tests by purpose.
For example:
Pull request
↓
Fast API smoke tests
↓
Build
↓
Deployment to QA
↓
API regression suite
↓
UI critical-flow suite
↓
Optional integration/performance/security stages
The API smoke suite should answer a narrow question:
Is this build healthy enough for deeper testing?
Good smoke candidates include:
- authentication,
- one critical read operation,
- one critical write operation,
- essential downstream integration,
- a high-value business transaction.
Large edge-case matrices belong in regression rather than blocking every developer feedback loop unnecessarily.
Limitations and risks of API automation
API automation is powerful, but it does not prove that the entire application works.
It does not validate the complete user experience
An API may work while the browser:
- sends the wrong payload,
- maps a field incorrectly,
- blocks a valid workflow,
- renders the wrong state.
UI tests remain necessary for critical user journeys.
It is not automatically security testing
Checking:
401 for missing token
403 for insufficient permission
is valuable functional coverage.
It does not replace systematic testing for vulnerabilities such as broken object-level authorization, injection, credential handling, rate-limit abuse, or other security weaknesses.
It is not automatically performance testing
A single API call completing below a threshold is not evidence that the system supports thousands of concurrent users.
Test environments can produce misleading failures
External systems, shared data, deployment transitions, queues, caches and unavailable dependencies can all cause failures unrelated to the application code being tested.
Good automation therefore records enough context to make failures diagnosable.
Conclusion
API automation testing with Postman, REST Assured, and Playwright can all produce reliable results, but they solve the maintenance problem differently. Postman provides the most accessible route from API exploration to automated collections. REST Assured is a strong option for Java teams that want API automation treated as conventional test code.
Playwright is especially useful when API and browser automation need to work together in the same TypeScript or JavaScript test ecosystem. For testers, the tool is only part of the solution. Reliable API automation also requires isolated data, meaningful assertions, deliberate negative coverage, secure configuration, predictable cleanup, and tests that can run independently. The most practical next step is to select one critical API workflow and implement the same create-verify-retrieve-cleanup pattern in the framework that best matches your team’s existing automation stack.
Frequently Asked Questions
-
Is Postman enough for API automation testing?
Yes, Postman can support meaningful automated API suites, including response assertions, collection execution, environments and command-line runs. For very large engineering-heavy test suites, teams should still evaluate whether collection-based maintenance or a conventional code framework better matches their development workflow.
-
Is REST Assured better than Postman?
Neither tool is universally better. REST Assured is typically a better architectural fit for Java test-automation teams that want their API tests maintained as code. Postman is typically easier for interactive exploration, shared API requests and testers who want to move incrementally from manual API testing into automation. The right decision depends on skills, source-control practices and how the suite will be maintained.
-
Can Playwright replace REST Assured for API testing?
Yes, for many TypeScript or JavaScript teams. Playwright's APIRequestContext supports direct HTTP API testing, and its request fixture integrates naturally with Playwright Test. A Java organization may still prefer REST Assured because it keeps the automation stack aligned with existing Java frameworks and libraries.
-
Should API tests run before UI automation?
Critical API smoke tests often should. If authentication, user creation, checkout, or another fundamental service is unavailable, running a large browser suite may generate dozens of secondary failures. An API health gate can detect the underlying problem earlier and make pipeline feedback clearer.
-
Should every API response have a JSON Schema test?
No. Schema validation is most valuable where the contract matters to multiple consumers or accidental structural changes create meaningful risk. Business assertions remain necessary because a perfectly valid schema can still contain logically incorrect data.
-
Should testers automate the same API scenarios in Postman and Playwright?
Generally, not without a specific reason. Duplicating every scenario increases maintenance cost without automatically increasing useful coverage. A better strategy might use Postman for collaborative API exploration and Playwright for the primary automated API/UI regression suite.
-
What should a beginner automate first?
Start with one stable business workflow: create resource, verify response, retrieve resource, verify persisted state, delete resource. Then add invalid input, missing authentication, boundary values, authorization checks, duplicate operations, and contract validation. This teaches request construction, assertions, data extraction, chaining, test isolation and cleanup without requiring a large framework immediately.
by Rajesh K | Sep 3, 2026 | Accessibility Testing, Blog, Latest Post |
Mobile app accessibility testing in 2026 is shifting from periodic compliance audits to continuous, task-based testing built into design, development, and release pipelines. New WCAG 2.2 criteria, native accessibility APIs in Android and iOS test frameworks, and rising regulatory pressure are all changing what QA teams need to check. App stores are also starting to surface accessibility claims directly to users, which turns testing evidence into a discoverability concern as well as a compliance one. This guide walks through the trends that matter most right now, and what a credible testing program looks like in practice.
What are the latest mobile app accessibility testing trends?
Mobile app accessibility testing in 2026 is shifting from periodic compliance audits to continuous, task-based testing integrated into design, development, automated UI tests, and release pipelines. The strongest trends include WCAG 2.2-based mobile guidance, automated accessibility checks in native test frameworks, semantics-level validation, broader assistive-technology testing, app-store accessibility disclosures, and increased regulatory pressure.
The direction is clear: automated scanning remains useful, but a credible mobile accessibility program now combines automated checks, accessibility-tree validation, manual assistive-technology testing, real-device coverage, and testing with people with disabilities.
Key Takeaways
- Treat accessibility as a release-quality requirement rather than a one-time audit.
- Expand test coverage around WCAG 2.2 issues such as touch targets, dragging alternatives, obscured focus, redundant entry, and accessible authentication.
- Add automated accessibility checks to Android and iOS UI-test pipelines.
- Test the accessibility semantics exposed to assistive technologies, not only the visual interface.
- Validate complete user journeys with VoiceOver, TalkBack, Voice Control, Switch Control or Switch Access, text enlargement, and other relevant features.
- Maintain manual and user testing because passing automated checks does not prove that an app is usable.
- Track regulatory and app-store requirements by market because the applicable accessibility benchmark can differ by jurisdiction.
What is mobile app accessibility testing?
Mobile app accessibility testing is the process of evaluating whether people with visual, auditory, physical, speech, neurological, or cognitive disabilities can perceive, understand, navigate, and operate a mobile application. Testing typically covers native accessibility APIs, screen-reader behavior, focus order, labels and roles, touch interaction, alternative input, text resizing, color and contrast, media alternatives, forms, authentication, and complete task flows.
It applies to native applications as well as mobile web and hybrid applications. W3C’s WCAG2ICT guidance explains how WCAG 2.0, 2.1, and 2.2 can be interpreted for non-web software, including mobile applications. The completed WCAG2ICT Group Note was updated through December 2025.
W3C is also developing WCAG2Mobile, a mobile-specific interpretation of WCAG 2.2 for native, mobile-web, and hybrid apps. As of September 1, 2026, WCAG2Mobile remains a Group Draft Note, meaning it is informative work in progress rather than a normative accessibility standard.
Mobile accessibility testing is therefore broader than running an accessibility scanner. A scanner can identify selected machine-testable problems, but it cannot determine whether an entire checkout, transfer, booking, registration, or onboarding journey is genuinely usable.
Why mobile app accessibility testing matters more in 2026
Accessibility has become more closely connected to product quality, regulatory exposure, procurement, and app discovery.
The European Accessibility Act (EAA) has applied to covered products and consumer services since June 28, 2025. Its scope includes areas such as consumer banking, e-commerce and certain transport services, including mobile device-based services and mobile applications.
In the United States, the Department of Justice’s ADA Title II rule establishes WCAG 2.1 Level AA as the technical standard for covered state and local government websites and mobile apps. An April 2026 interim final rule extended the compliance deadlines to April 26, 2027 for public entities with populations of 50,000 or more and April 26, 2028 for smaller entities and special district governments.
At the platform level, Apple has introduced Accessibility Nutrition Labels for App Store product pages. These labels allow developers to declare support for features such as VoiceOver and Larger Text, and Apple expects developers to evaluate whether users can complete the app’s common tasks with the declared feature.
The result is that accessibility testing increasingly affects more than defect counts. It can influence whether a team can substantiate a compliance position, publish accurate accessibility claims, satisfy procurement requirements, and confidently release critical user journeys.
What are the biggest mobile app accessibility testing trends in 2026?
1. WCAG 2.2 is expanding what mobile teams test
WCAG 2.2 added nine success criteria to WCAG 2.1, several of which directly affect common mobile interactions. W3C recommends using the latest WCAG version as a conformance target where possible.
For mobile QA teams, particularly important WCAG 2.2 additions include:
- Focus Not Obscured (Minimum): focused controls should not be completely hidden by other content.
- Dragging Movements: functionality that requires dragging should have a non-dragging single-pointer alternative unless dragging is essential.
- Target Size (Minimum): pointer targets need sufficient dimensions or spacing under the criterion’s conditions.
- Consistent Help: help mechanisms should appear consistently when applicable.
- Redundant Entry: users should not unnecessarily re-enter previously supplied information in the same process.
- Accessible Authentication: authentication should not unnecessarily depend on cognitive-function tests such as memorization or transcription.
This changes accessibility regression suites. Teams that previously focused mostly on screen-reader labels and color contrast now need test cases around authentication, touch precision, gesture alternatives, overlays, repeated form input, and multi-step workflows.
Testing implication: maintain WCAG 2.2-oriented test cases even when a jurisdiction formally references an earlier WCAG version, while separately documenting the standard that actually governs legal compliance.
2. Automated accessibility checks are moving into CI/CD
Accessibility automation is increasingly becoming part of ordinary UI testing rather than a separate pre-release activity.
On Android, the Accessibility Test Framework used by tools such as Accessibility Scanner can now be integrated with Compose testing. Android documentation states that automated accessibility checks are available for Compose, starting with Compose 1.8.0, and can run around UI actions.
Apple provides a similar path through XCTest. Calling performAccessibilityAudit(for:_:) on an XCUIApplication runs accessibility audits during UI testing, and detected audit issues can cause the test to fail.
A modern pipeline can therefore look like this:
- Developer changes a component.
- Unit and UI tests run.
- Automated accessibility checks inspect affected screens.
- A high-severity accessibility regression fails the build or blocks promotion.
- Manual assistive-technology tests validate the affected workflow before release.
The important trend is not simply “more automation.” It is accessibility as continuous regression testing. Some teams are also starting to bring AI-assisted debugging into this workflow see our piece on AI for Accessibility for how that fits in.
3. Accessibility testing is becoming semantics-first
Visual inspection alone cannot tell a tester what a screen reader or other assistive technology receives.
Android’s Jetpack Compose relies heavily on a semantics tree, which exposes the meaning and properties of UI elements. Compose UI tests can interact with this semantic information, and accessibility services consume related semantic data.
This means testing increasingly asks questions such as:
- Does this icon have a meaningful accessible name?
- Is this custom component exposed as a button, switch, heading, or other correct role?
- Is its current state communicated?
- Have several visual children been incorrectly merged into one accessible element?
- Is decorative content unnecessarily included in navigation?
- Does the accessible action match the visible action?
Apple platforms use corresponding accessibility properties such as labels, traits, values, hierarchies, and custom actions. Apple notes that custom controls need appropriate accessibility information so assistive applications can accurately communicate them to users.
Testing implication: add accessibility-tree or semantics inspection to component testing, particularly for custom controls and cross-platform UI layers, including any React-based hybrid or React Native screens.
4. Teams are testing complete accessible journeys, not isolated screens
A screen can contain technically accessible controls and still form an unusable workflow.
Apple’s Accessibility Nutrition Label evaluation model reinforces a task-oriented approach. Before declaring support for an accessibility feature, developers are expected to identify the application’s common tasks and verify that those common tasks can be completed with the feature being claimed. Apple recommends a testing matrix organized by task, accessibility feature, and supported device.
For an e-commerce app, for example, the meaningful accessibility test is not merely:
“Does the Add to Cart button have an accessible label?”
It is:
“Can a VoiceOver or TalkBack user independently find a product, select a variation, add it to the cart, modify quantity, enter delivery information, authenticate if necessary, pay, resolve an error, and confirm the order?”
This journey-based approach catches focus loss, inaccessible modal dialogs, misleading announcements, inaccessible authentication, keyboard problems, and state-management defects that individual component scans often miss.
5. Testing is expanding beyond screen readers
Screen-reader testing remains essential, but it represents only part of mobile accessibility.
Apple recommends testing with accessibility technologies such as VoiceOver, Voice Control, and Switch Control. Android documentation similarly highlights TalkBack and Switch Access and recommends testing directly with Android assistive technologies.
Depending on the application, a mature test matrix can also cover:
- Larger text and Dynamic Type
- Zoom or magnification
- Increased contrast
- Reduced motion
- Color differentiation
- Captions and accessible media
- Voice-based navigation
- Switch navigation
- External keyboard behavior
- Orientation changes
- Touch-target accuracy
- Alternative interactions for complex gestures
The specific combination should follow the app’s functionality and target users rather than becoming a generic checklist.
6. App-store accessibility information is creating a new testing deliverable
Apple’s Accessibility Nutrition Labels introduced an important change: accessibility test results can now influence pre-install app discovery.
The labels appear on supported Apple operating systems, including iOS 26 and related platform releases. Apple states that accessibility features can also be considered in App Store searches, for example, when someone searches for an app supporting VoiceOver or Larger Text.
Providing the labels is initially voluntary, but Apple says developers will eventually be required to share accessibility-support information when submitting new apps and updates.
That makes accurate accessibility verification an app-store metadata concern as well as a QA concern.
Testing implication: preserve evidence supporting each accessibility feature your product page claims. Re-evaluate those claims after major redesigns, navigation changes, platform migrations, or new critical workflows.
7. Accessibility compliance is becoming a continuously moving target
Regulation and standards are evolving at different speeds, so testing teams increasingly need a requirements matrix rather than a single universal “accessibility standard.”
For example:
- The EU Accessibility Act has already entered into application for covered services.
- The U.S. ADA Title II mobile-app rule specifies WCAG 2.1 Level AA for covered public entities, with updated deadlines in 2027 and 2028.
- W3C’s completed WCAG2ICT guidance now includes WCAG 2.2 interpretation for non-web software.
- ETSI’s EN 301 549 V4.1.0 revision was adopted for publication on August 24, 2026, with publication scheduled for September 3, 2026. The revision is intended to support both the Web Accessibility Directive and the European Accessibility Act and updates key clauses to align with WCAG 2.2. As of September 1, that publication date is still upcoming, and publication should not be confused with any separate EU legal harmonisation step.
Testing implication: record the standard, version, jurisdiction, platform and test date associated with every formal accessibility assessment.
8. Manual and user testing remain essential despite better automation
Automation is improving, but both major mobile ecosystems explicitly caution against treating it as sufficient.
Apple states that eliminating all issues reported by Accessibility Inspector does not guarantee a fully accessible app and recommends testing with assistive technologies such as VoiceOver.
Google’s Android accessibility guidance also recommends user testing alongside analysis and automation because users can reveal usability issues that automated checks cannot.
Automated tools are good at deterministic questions such as whether an accessible label is missing or whether a target appears too small. They are much weaker at questions such as whether instructions make sense, whether a screen-reader announcement is useful in context, or whether a complex transaction is practically achievable.
How does modern mobile accessibility testing work?
A robust mobile accessibility program combines five layers of validation:
- Requirements mapping: determine which standards, regulations, platform guidelines, and contractual requirements apply.
- Component validation: verify labels, roles, states, values, semantics, contrast, touch targets, text behavior, and alternative interactions.
- Automated regression testing: run accessibility checks within Android and iOS UI tests.
- Workflow validation: execute critical user journeys with relevant assistive technologies.
- Human evaluation: conduct expert manual review and, where practical, usability testing with people with disabilities.
The output should be more than a defect list. Teams should be able to state what was tested, on which devices and OS versions, against which benchmark, using which assistive technologies, and what limitations remained.
Step-by-step mobile app accessibility testing process
1. Define the applicable accessibility baseline
Record the target benchmark before testing begins.
For example:
- WCAG version and target level
- Applicable regional regulation
- iOS and Android accessibility expectations
- Procurement or customer requirements
- Supported device and OS ranges
Do not silently substitute WCAG 2.2 for a regulation that explicitly mandates WCAG 2.1; track both when appropriate.
2. Identify the app’s critical and common tasks
Create an inventory such as:
- Registration
- Sign in
- Password recovery
- Search
- Purchase or payment
- Profile editing
- File upload
- Messaging
- Booking
- Form submission
- Logout
Prioritize tasks whose failure would prevent a user from achieving the application’s primary purpose.
3. Create an accessibility test matrix
Cross-reference each important task against relevant dimensions.
| S. No |
Dimension |
Examples |
| 1 |
Platform |
iOS, Android |
| 2 |
Assistive technology |
VoiceOver, TalkBack, Voice Control, Switch Control/Access |
| 3 |
Display |
Default text, maximum supported larger text, portrait, landscape |
| 4 |
Interaction |
Touch, keyboard, switch, voice |
| 5 |
UI state |
Loading, empty, error, success, offline |
| 6 |
Account state |
New user, returning user, authenticated user |
Avoid creating combinations that provide no meaningful risk coverage. Prioritize representative configurations based on product usage and accessibility risk.
4. Validate accessibility semantics
Inspect each critical component for:
- Accessible name
- Role
- State
- Value
- Focusability
- Reading/navigation order
- Grouping
- Custom actions
- Error associations
For Android Compose applications, inspect the semantics tree where behavior is unclear. Android’s Layout Inspector can expose semantic information useful for debugging accessibility problems.
5. Add automated accessibility checks
Run platform accessibility checks inside existing UI-test suites.
On Android, combine Compose accessibility checks with semantic assertions for business-critical components. On Apple platforms, run XCTest accessibility audits for screens covered by UI tests.
Treat new critical findings as regressions rather than postponing all accessibility remediation to a final audit.
6. Test with assistive technologies
Execute the critical journeys without relying on ordinary touch interaction.
Check:
- Whether focus follows a logical sequence
- Whether accessible names are meaningful
- Whether states and errors are announced
- Whether modals move focus appropriately
- Whether users can return from overlays
- Whether custom gestures have accessible alternatives
- Whether authentication can be completed
- Whether dynamic updates are communicated
7. Stress the interface with accessibility settings
Test important screens with large text, display scaling, orientation changes, increased contrast and reduced motion where relevant.
Apple’s accessibility audit tooling can identify issues including clipped text and Dynamic Type support, while manual testing remains necessary to confirm actual usability.
8. Include users with disabilities in high-value validation
Recruit representative users for critical or unfamiliar interactions when feasible.
Focus on task completion, effort, clarity and recovery rather than asking users merely whether they “like” the interface.
9. Retest and preserve evidence
For each fixed issue:
- Reproduce the original failure.
- Verify the fix.
- Run relevant automated regression tests.
- Re-run the affected assistive-technology journey.
- Record the environment and outcome.
This produces more defensible evidence than a single undated accessibility score.
Practical example: testing an accessible mobile banking transfer
Consider a banking application in which a customer transfers money between accounts.
Business scenario: A customer needs to send ₹5,000 from a savings account to a registered beneficiary.
Preconditions:
- The user is authenticated.
- At least two accounts or a beneficiary are available.
- Screen-reader support is enabled.
- The device uses an enlarged system text size.
Test process:
- Navigate from the dashboard to “Transfer Money.”
- Confirm that the screen reader announces the screen purpose.
- Select the source account.
- Select the beneficiary.
- Enter the amount.
- Verify that labels remain associated with fields at large text sizes.
- Trigger validation by omitting a required field.
- Confirm that the error is announced and focus can reach it.
- Correct the input without unnecessarily re-entering valid information.
- Complete authentication using an accessible method.
- Submit the transfer.
- Confirm that the successful transaction status is communicated programmatically.
Expected result: The user can complete the entire transfer independently without relying on visual-only information, inaccessible gestures, memorization-based barriers, or unexplained focus changes.
Example failure: After submission, the interface shows a visual “Transfer successful” banner but does not expose the change to the accessibility system. A screen-reader user receives no confirmation and may repeat the transaction.
The practical lesson is that the accessibility defect does not necessarily exist in an individual button. It may exist in the transition between states within the complete business process.
Automated vs. manual vs. user accessibility testing
| S. No |
Factor |
Automated testing |
Manual assistive-technology testing |
Testing with users with disabilities |
| 1 |
Primary purpose |
Detect repeatable technical issues |
Validate interaction and workflows |
Evaluate real-world usability |
| 2 |
Good at |
Missing properties, selected structural issues, regression detection |
Focus order, announcements, gestures, navigation, state changes |
Clarity, effort, unexpected barriers, practical task completion |
| 3 |
Speed |
High |
Moderate |
Lower |
| 4 |
CI/CD suitable |
Yes |
Partly |
Usually no |
| 5 |
Finds contextual usability problems |
Limited |
Yes |
Strongest |
| 6 |
Main limitation |
Cannot infer complete usability |
Depends on evaluator skill and coverage |
Small samples do not represent every user |
| 7 |
Best timing |
Every relevant build |
Feature completion and release candidates |
High-risk workflows and major releases |
These approaches are complementary. None should be treated as a substitute for the others.
Mobile app accessibility testing best practices
Test accessibility during development. Catching a broken semantic role in a reusable component is cheaper and safer than discovering it across dozens of screens during a release audit.
Automate stable, deterministic rules. Use automated checks for issues that tools can consistently detect, then reserve manual effort for context, navigation, workflow and usability.
Prioritize critical journeys. Authentication, payments, registration, checkout, booking and account recovery deserve deeper assistive-technology coverage than low-value informational screens.
Test custom components aggressively. Standard platform components typically expose more accessibility behavior automatically; custom controls require deliberate names, roles, states, values and actions.
Include error and empty states. Accessibility defects frequently appear after validation errors, asynchronous updates, loading states, permission requests and modal transitions.
Test more than VoiceOver and TalkBack. Select additional technologies according to product risk, including voice control, switches, larger text and alternative input.
Retest after design-system changes. A defect in a shared button, field, dialog or navigation component can create accessibility regressions across the application.
Document the test environment. Record device, operating system, app version, accessibility feature, standards baseline and test date so results remain reproducible.
Common mobile accessibility testing mistakes
| S. No |
Mistake |
Why it happens |
Impact |
Recommended fix |
| 1 |
Treating a scanner score as certification |
Automated results are easy to quantify |
Serious workflow barriers remain hidden |
Combine automation with manual and user testing |
| 2 |
Testing only the happy path |
QA focuses on successful transactions |
Errors and recovery flows become inaccessible |
Include validation, offline, timeout and failure states |
| 3 |
Testing only with a screen reader |
Screen readers are the best-known accessibility tool |
Motor and low-vision barriers may be missed |
Add switch, voice, text scaling and visual-setting tests where relevant |
| 4 |
Adding labels without validating roles or states |
Teams equate accessibility with descriptions |
Users hear incomplete or misleading information |
Test name, role, state and value together |
| 5 |
Checking screens independently |
Test cases mirror UI screens |
Cross-screen focus and state defects are missed |
Test complete business journeys |
| 6 |
Assuming iOS and Android behave identically |
Shared product logic creates false confidence |
Platform-specific accessibility regressions ship |
Validate each supported platform separately |
Troubleshooting common accessibility testing failures
Why do automated accessibility tests pass but VoiceOver or TalkBack still feels unusable?
The likely cause is that the problem is contextual rather than machine-detectable. A button may have a valid label while appearing in an illogical focus sequence or while producing a confusing action result.
Verify the complete journey manually with the relevant screen reader. Check focus movement, announcements, state changes and recovery paths. Keep automated checks, but do not use them as a usability certificate.
Why does accessibility focus skip a custom control?
First inspect whether the component exposes appropriate accessibility semantics or properties.
On Android Compose, examine the merged and unmerged semantics trees and confirm that descendant merging or clearing has not removed necessary information. On iOS, verify the accessibility element’s label, traits, hierarchy and actions.
Then retest using the actual assistive technology rather than relying only on the inspector.
Why does the interface break when users enable larger text?
The layout may use fixed heights, non-wrapping containers or assumptions about label length.
Test important screens at larger supported text sizes and inspect for clipping, overlap, truncation, hidden controls and inaccessible scrolling. Apple Accessibility Inspector includes checks related to clipped text and Dynamic Type, but the completed workflow should still be tested manually.
Why does a cross-platform app pass on iOS but fail on Android?
Cross-platform source code does not guarantee identical accessibility output.
Each platform has its own accessibility APIs, focus behavior, semantics mapping and assistive technologies. Inspect the native accessibility representation generated on both platforms and validate critical journeys separately with VoiceOver and TalkBack.
Why does a drag-and-drop interaction fail accessibility testing?
The interface may require a user to perform a precise dragging motion without providing another single-pointer method.
WCAG 2.2 Success Criterion 2.5.7 requires a non-dragging single-pointer alternative where the criterion applies, unless dragging is essential or another stated exception applies. Consider controls such as Move Up, Move Down, Add, Remove or another equivalent operation.
Mobile accessibility testing tools and implementation options
Apple platforms
Accessibility Inspector can inspect accessibility properties and run audits for issues such as element descriptions, hit regions, contrast and clipped text.
XCTest accessibility audits allow teams to execute selected accessibility checks from automated UI tests.
VoiceOver, Voice Control and Switch Control should be used for workflow-level manual testing when relevant.
Android
Accessibility Scanner identifies selected problems such as content labels, clickable-item issues and contrast concerns.
Compose accessibility checks can integrate Accessibility Test Framework checks into Compose tests.
Layout Inspector helps inspect and debug accessibility semantics exposed by Compose components.
TalkBack and Switch Access provide direct validation of experiences used by people who depend on assistive technology.
The best toolset is not the one with the largest number of checks. It is the one that supports repeatable automation while still enabling testers to validate the actual interaction model. For a broader comparison of tools beyond these native platform options, see our roundup of the 10 Best Web Accessibility Checker Tools in 2026.
Limitations and risks to consider
Automated coverage remains incomplete. An automated tool can detect selected technical conditions, not whether every user can successfully achieve a goal.
Legal requirements vary. WCAG 2.2 may be the preferred technical target for a product team while a particular regulation still incorporates WCAG 2.1 or another standard.
WCAG2Mobile is currently draft guidance. It is useful for interpreting mobile-specific questions, but as of September 1, 2026 it is not a normative W3C Recommendation.
Platform behavior changes. OS upgrades and framework migrations can change semantics, focus handling or assistive-technology behavior, making accessibility regression testing necessary.
User samples have limits. Testing with several people with disabilities can reveal barriers that tools miss, but no small participant group represents every disability, preference or assistive-technology configuration.
Conclusion
The latest mobile app accessibility testing trends point toward a single operational change: accessibility needs to function like every other continuous product-quality requirement. QA teams should automate the checks that can be automated, inspect the semantic information exposed by custom interfaces, test critical workflows with real assistive technologies, include accessibility settings and failure states, and validate important experiences with users with disabilities.
WCAG 2.2, updated mobile guidance, new platform testing APIs, accessibility disclosures in app stores, and evolving regulation are expanding both the depth and visibility of mobile accessibility testing. Teams that embed these practices throughout development can detect barriers earlier, reduce accessibility regressions, and produce stronger evidence that critical mobile experiences are genuinely usable.
Frequently Asked Questions
-
What is the biggest mobile accessibility testing trend in 2026?
The biggest trend is the move from occasional accessibility audits to continuous accessibility quality assurance. Android and Apple now provide mechanisms for incorporating accessibility checks into automated UI testing, while teams are also expanding manual coverage around complete tasks and multiple assistive technologies. The goal is to detect accessibility regressions during development rather than after the product is considered finished.
-
Should mobile apps be tested against WCAG 2.2?
WCAG 2.2 is a strong current technical target, and W3C recommends using its latest WCAG version where possible. WCAG2ICT provides completed guidance for interpreting WCAG 2.2 in non-web software. However, regulatory obligations differ: for example, the U.S. ADA Title II rule currently specifies WCAG 2.1 Level AA for covered state and local government mobile apps. Always distinguish product best practice from the legally incorporated standard.
-
Can automated tools fully test mobile app accessibility?
No. Automated tools can efficiently detect selected issues and prevent known regressions, but they cannot determine whether an entire application is understandable and practically usable. Apple explicitly notes that clearing Accessibility Inspector audit issues does not guarantee full accessibility, while Android recommends user testing alongside automated methods.
-
Which assistive technologies should a mobile app test?
At minimum, select technologies that represent the application's users and interaction risks. Common coverage includes VoiceOver on Apple devices and TalkBack on Android. Depending on the product, also test Voice Control, Switch Control or Switch Access, larger text, magnification, contrast settings and alternative input. Avoid treating a single screen-reader pass as complete accessibility coverage.
-
How often should mobile accessibility testing be performed?
Automated accessibility checks should run whenever relevant UI tests run, while manual accessibility regression testing should occur after accessibility-sensitive feature changes and before release. Critical workflows should also be retested after major framework, design-system or operating-system changes. The appropriate frequency depends on release velocity and product risk, but accessibility should be part of the normal QA lifecycle rather than an annual audit.
-
What changed most recently for European mobile accessibility testing?
The European Accessibility Act has applied to covered products and services since June 28, 2025. In addition, ETSI's EN 301 549 V4.1.0 revision was adopted for publication on August 24, 2026 and is scheduled for publication on September 3, 2026. The revision updates key requirements toward WCAG 2.2 and is intended to support the EAA as well as the Web Accessibility Directive. Teams should monitor its publication and subsequent applicable harmonisation status rather than assuming a draft or newly published standard automatically changes legal obligations.
by Rajesh K | Sep 2, 2026 | AI Testing, Blog, Latest Post |
AI regression testing is changing how QA teams decide what to run, when to run it, and how to keep automation working as applications evolve. Regression suites keep growing, but CI pipelines cannot always wait for every test to finish before developers need feedback. Machine learning and generative AI now help teams select relevant tests, prioritize the ones most likely to fail, heal broken UI locators, and make sense of large failure logs. None of this replaces sound test design or human judgment about business risk. This guide walks through where AI genuinely helps in a regression-testing workflow, how to introduce it safely, and where its limits are.
How does AI improve regression testing?
AI improves regression testing by analyzing code changes, test history, failure patterns, and application behavior to determine which tests should run, which should run first, where coverage may be missing, and why failures occurred. It can also reduce automation maintenance through self-healing tests and help generate or update test cases.
The most effective approach is not to let AI replace the regression suite. Instead, AI acts as an intelligence layer that helps QA teams use the suite more efficiently while retaining appropriate full-suite and risk-based validation.
Key takeaways
- AI can select regression tests that are more relevant to a specific code change.
- Machine learning can prioritize tests with a higher predicted probability of failure.
- Generative AI can assist with creating tests, identifying edge cases, and understanding failures.
- AI-powered self-healing can reduce failures caused by minor UI and locator changes.
- AI should complement, not eliminate, critical-path tests and periodic full regression runs.
- The effectiveness of AI-assisted regression testing depends heavily on good test history, reliable execution data, and continuous monitoring.
What is AI-assisted regression testing?
Regression testing verifies that software changes have not damaged functionality that previously worked. ISTQB defines regression testing as change-related testing intended to detect defects introduced or uncovered in unchanged parts of the software after a modification.
AI-assisted regression testing applies artificial intelligence techniques such as machine learning, natural language processing, computer vision, and large language models to improve how regression tests are selected, prioritized, maintained, generated, and analyzed.
It can include:
- Predictive regression test selection
- Test case prioritization
- Change-impact prediction
- Automated test generation
- Self-healing UI automation
- Flaky-test analysis
- Failure classification and summarization
- Coverage-gap identification
AI-assisted regression testing is different from conventional test automation. Traditional automation executes predefined logic repeatedly. AI adds a decision-making or prediction layer that can adapt its recommendations based on data.
Why does AI matter in regression testing?
Regression suites naturally grow as products gain features, integrations, platforms, and edge cases. In continuous integration and continuous delivery environments, running every test after every change can eventually create a conflict between comprehensive validation and fast developer feedback.
Research on machine-learning-based test selection and prioritization specifically identifies frequent CI builds and the resulting time and resource requirements of large test suites as a major reason for using intelligent selection techniques, per a 2022 systematic literature review in Empirical Software Engineering.
A 2026 study in Empirical Software Engineering similarly describes test case prioritization as a balancing problem: teams want to detect faults as early as possible while operating within testing-time and resource constraints. The study notes that modern ML approaches can use execution logs and information about the system under test to predict the probability that individual tests will fail.
This makes AI useful in several parts of a regression-testing workflow.
Faster feedback to developers
Instead of treating all regression tests as equally valuable for every commit, AI can estimate which tests are more likely to detect a problem caused by the current change.
High-risk tests can then run first.
Developers receive meaningful feedback earlier even if the complete suite takes much longer to execute.
Better use of CI infrastructure
If a regression suite contains thousands of tests, executing every test for every minor change can consume substantial compute capacity.
Predictive test selection can create a smaller change-specific subset for early CI stages while retaining broader regression runs at appropriate checkpoints.
Less automation maintenance
UI regression tests frequently fail when identifiers, labels, DOM structures, or layouts change even though the underlying functionality remains correct. This is one of the biggest drivers of test automation maintenance costs.
AI-powered self-healing tools attempt to recognize the intended control using multiple properties, visual information, history, or semantic meaning instead of relying entirely on a brittle selector.
Faster failure investigation
A large regression run can generate many logs, screenshots, stack traces, retries, and related failures.
Generative AI can summarize execution information and assist testers in understanding what failed and why. For example, current Tricentis documentation describes AI-assisted execution insights that summarize test functionality and execution results in natural language.
How does AI improve regression testing?
AI can improve regression testing at several distinct stages.
1. AI predicts which tests are relevant to a code change
A predictive test-selection model can analyze signals such as:
- Files modified in the current build
- Historical test failures
- Relationships between changed code and failed tests
- Test execution history
- Test duration
- Code or test-path similarity
- Characteristics of the current change
The model then estimates which regression tests are most relevant.
Current Launchable documentation provides a practical example of this approach. Its predictive test-selection model uses information including execution history, test characteristics, correlations between changed files and failures, path similarity, and characteristics such as change size and file types. It then prioritizes the available suite before creating a test subset.
2. AI prioritizes tests that are more likely to fail
Selection answers:
Which tests should run?
Prioritization answers:
In what order should they run?
An ML model can assign a predicted failure probability or risk score to tests and place higher-risk tests earlier in the execution queue.
For example:
| S. No |
Test |
Predicted risk |
Execution time |
Priority |
| 1 |
Checkout payment |
High |
2 min |
1 |
| 2 |
Coupon calculation |
High |
1 min |
2 |
| 3 |
Order history |
Medium |
3 min |
3 |
| 4 |
Profile avatar |
Low |
1 min |
4 |
If a payment-service change introduced a regression, the team is more likely to discover it early rather than waiting for hundreds of unrelated tests to finish.
Recent research continues to investigate this problem. A 2026 study describes both learning-to-rank and binary classification as approaches for ML-based test prioritization, with failure probabilities providing a basis for sorting the regression suite.
3. Generative AI can help create additional regression tests
Large language models can analyze code, requirements, diffs, bug descriptions, or existing tests and suggest new cases.
Potential uses include:
- Creating tests for a newly fixed defect
- Adding boundary-value cases
- Finding missing negative scenarios
- Generating unit or integration test scaffolding
- Updating tests affected by code changes
GitHub’s current Copilot documentation, for example, describes generating unit and integration tests and explicitly recommends asking for success cases, failure cases, and edge cases.
This capability is promising but should remain review-driven.
A 2025 research preprint evaluated LLM-generated regression tests across 22 commits in three software projects. The technique performed better for programs using human-readable structured inputs such as XML and JavaScript but struggled with more compact formats such as PDF. This illustrates an important limitation: LLM test-generation performance can depend heavily on the representation of the system and inputs being tested.
4. AI can make UI regression automation more resilient
Suppose an automated test contains this step:
Click the “Submit Order” button.
A traditional script might depend on a single ID:
#submit-order
If developers rename the identifier while leaving the button and workflow unchanged, the test fails.
An AI-assisted system can potentially use additional information such as:
- Visible text
- Element type
- Nearby labels
- Historical element properties
- Visual position
- Semantic purpose
to identify the intended element.
Tricentis Tosca, for example, supports self-healing controls by looking for similar controls when the expected control cannot be found. Its documentation also warns that self-healing can affect execution performance.
mabl documents a related approach in which historical element information is used to find strong matches. When standard healing is insufficient, its advanced auto-healing capability can use generative AI to evaluate semantic similarities. The product also uses confidence controls rather than automatically accepting every possible replacement.
The important principle is that self-healing should be observable. A test that silently switches to an incorrect control can be more dangerous than a test that fails visibly.
5. AI can help analyze regression failures
Consider a regression run in which 70 tests fail because one authentication service is unavailable.
Without correlation, engineers may investigate dozens of failures separately.
An AI-assisted analysis layer can group related symptoms and summarize evidence such as:
- Common exception messages
- Shared failing services
- Similar stack traces
- Failure timing
- Affected environments
- Recent code changes
Instead of presenting 70 apparently independent problems, the system may surface one probable shared cause for investigation.
AI does not prove root cause by summarizing a log. Engineers still need to validate the conclusion, particularly when the failure affects release decisions.
Step-by-step: How to introduce AI into a regression-testing process
1. Establish a reliable regression baseline
Before adding AI, make sure the regression suite itself is trustworthy.
Record:
- Test ID
- Component or business capability
- Execution duration
- Pass/fail history
- Failure reason where available
- Code coverage or dependency information
- Flaky-test status
- Environment
- Build and commit information
AI cannot compensate for consistently poor testing data.
Expected result: A structured history connecting code changes, test executions, and outcomes.
2. Identify the bottleneck you actually need to solve
Do not introduce AI simply because AI capabilities are available.
Determine whether your primary problem is:
- Excessive execution time
- Slow feedback
- High UI-test maintenance
- Too many flaky tests
- Poor failure triage
- Missing regression coverage
Different problems require different techniques.
3. Start with test ranking before aggressive test reduction
A lower-risk starting point is to let AI reorder the complete regression suite.
High-risk tests run first, but no tests are removed.
This allows the team to evaluate whether the model consistently places defect-revealing tests near the top.
4. Introduce change-aware test selection
Once the ranking model is trusted, use it to recommend smaller regression subsets for selected CI stages.
For example:
- Pull request: AI-selected tests + mandatory critical-path tests
- Main branch: Larger risk-based regression suite
- Nightly build: Full automated regression suite
- Release candidate: Full required regression and non-functional validation
This layered strategy provides speed without making every quality decision dependent on one predictive model.
5. Add self-healing with strict controls
Enable self-healing only when your automation platform records:
- What element changed
- Which replacement was selected
- Confidence or matching information
- Screenshots or execution evidence where applicable
- Whether the change became permanent
Low-confidence matches should fail or require review.
6. Use generative AI to assist test creation
Feed the model precise information:
- Requirement
- Acceptance criteria
- Relevant code diff
- Existing tests
- Business rules
- Expected outputs
- Known defect
Ask it to identify missing positive, negative, boundary, and error-handling scenarios.
Then review the generated tests before adding them to the maintained regression suite.
7. Measure the results continuously
Track AI-assisted regression testing with concrete metrics such as:
- Time to first meaningful failure
- Total CI regression duration
- Percentage of tests selected per change
- Percentage of regressions detected by selected tests
- Regressions missed by the selected subset
- Flaky-test rate
- Self-healing frequency
- Incorrect self-healing events
- Test-maintenance effort
- Full-suite versus selected-suite outcomes
The most important metric is not simply “tests skipped.”
It is whether the team achieves faster feedback without unacceptable loss of defect-detection capability.
Practical example: AI-assisted regression testing for an e-commerce checkout change
Consider a hypothetical e-commerce application with a large automated regression suite.
A developer modifies the pricing service to introduce a new discount calculation.
Preconditions
The organization stores:
- Historical test outcomes
- Test execution duration
- Source-code changes
- Component ownership
- Test-to-code or coverage information
Input
The pull request modifies pricing/discount-service and related validation logic.
AI-assisted process
- The system analyzes the changed files.
- Historical data shows which tests previously failed after pricing-related changes.
- Tests are assigned risk scores.
- Checkout, promotions, tax, cart-total, and refund scenarios move toward the top.
- A selected subset runs immediately in CI.
- Mandatory smoke and critical payment tests run regardless of the prediction.
- The complete regression suite still executes on the scheduled full-validation pipeline.
Expected output
If the change is safe, the high-risk subset passes and the developer receives rapid initial feedback.
If the change introduces an incorrect discount calculation, a relevant checkout or promotion test should ideally fail early.
Error condition
Suppose the model ranks all refund tests as low risk, but the pricing change also affects refund calculations through an indirect dependency.
The selected subset could miss the regression.
This is why teams should compare selected-suite results against periodic full regression runs and update their models or rules when missed relationships are discovered.
The example is illustrative rather than a performance benchmark.
AI-assisted vs. traditional regression testing
| S. No |
Factor |
Traditional regression testing |
AI-assisted regression testing |
| 1 |
Test selection |
Rule-based, manual, dependency-based, or full-suite |
Can use historical and change data to predict relevance |
| 2 |
Test ordering |
Fixed or manually prioritized |
Dynamic risk or failure-probability ranking |
| 3 |
Maintenance |
Broken scripts generally require manual updates |
Self-healing can handle some UI changes |
| 4 |
Test creation |
Tester/developer designs tests |
AI can suggest or generate candidate tests |
| 5 |
Failure analysis |
Engineers inspect logs and reports |
AI can summarize or correlate failure evidence |
| 6 |
Adaptability |
Requires explicit rule changes |
Models can learn from newer execution data |
| 7 |
Main risk |
Slow or expensive regression cycles |
Incorrect predictions can skip relevant tests |
| 8 |
Human oversight |
Required |
Still required |
AI therefore changes how regression-testing effort is allocated rather than changing the fundamental objective of regression testing.
Best practices for using AI in regression testing
Keep critical business flows mandatory
Login, checkout, payment, authorization, data integrity, and other high-impact journeys should not disappear from regression simply because a predictive model gives them a low score.
Combine learned predictions with business risk.
Retrain and reevaluate models as the application changes
Software evolves. A 2026 regression-test-prioritization study specifically notes that predictive performance may decline as additional builds alter the testing environment and data distribution. Model updating therefore matters in long-running AI-assisted testing programs.
Maintain periodic full-suite execution
Selected regression testing provides faster feedback, but full runs are valuable for discovering dependencies the model does not yet understand.
A useful analogous principle appears in Microsoft’s Test Impact Analysis. Although TIA is change-impact analysis rather than generative AI, it falls back to all tests when it cannot safely reason about a change and supports periodically running the complete suite.
Monitor false negatives, not only execution savings
Reducing a 100-minute suite to 20 minutes means little if important regressions are routinely missed.
Compare:
- Defects found by AI-selected tests
- Defects found only by the later full suite
Keep self-healing transparent
Review healing logs and track how frequently elements change.
Repeated healing of the same test may indicate poor locator design or application instability rather than successful automation.
Give generative AI sufficient context
A vague instruction such as:
Create tests for checkout.
is less useful than:
Generate regression cases for the coupon-validation change. Cover expired coupons, minimum-order rules, combined promotions, empty codes, invalid codes, and checkout totals. Use our existing Playwright structure.
Current Tricentis guidance similarly recommends providing detailed manual test cases and clear domain context when using its agentic test-generation capabilities.
Review AI-generated tests like human-written code
Generated tests can contain incorrect assumptions, weak assertions, invented APIs, duplicated coverage, or excessive mocking.
Execute and review them before trusting them as regression controls.
Common mistakes when applying AI to regression testing
| S. No |
Mistake |
Why it happens |
Impact |
Recommended fix |
| 1 |
Immediately reducing the suite |
Teams focus on execution savings |
Relevant tests may be skipped |
Validate ranking accuracy before reducing coverage |
| 2 |
Training on poor test history |
Logs contain flaky or inconsistent results |
Model learns misleading patterns |
Clean and classify execution data |
| 3 |
Treating AI predictions as certainty |
Risk scores look authoritative |
Missed defects become harder to detect |
Combine predictions with rules and full-suite checkpoints |
| 4 |
Allowing silent self-healing |
Automation prioritizes passing tests |
Test may interact with the wrong element |
Log and review every healing decision |
| 5 |
Accepting generated tests without review |
LLM output appears plausible |
Incorrect assertions enter the suite |
Require code review and execution |
| 6 |
Optimizing only for test count |
Smaller suites look efficient |
Long-running or high-risk tests may be mishandled |
Optimize around feedback time and risk |
| 7 |
Never retraining the model |
Initial results remain acceptable |
Predictions degrade as software evolves |
Monitor drift and refresh models |
Troubleshooting AI-assisted regression testing
Why is the AI selecting irrelevant tests?
The model may be using historical correlations that are not obvious from the current code structure.
Check the features driving prioritization, test history, flaky failures, dependency data, and recent architectural changes.
If seemingly irrelevant tests repeatedly receive high scores without finding meaningful defects, investigate the training data and model calibration.
Why did the selected regression suite miss a defect?
Likely causes include insufficient historical data, a previously unseen dependency, model drift, missing coverage, or an overly aggressive selection threshold.
Verify the problem by checking whether the full suite detects the defect.
Then add the relationship to the model’s future evidence, update deterministic risk rules where necessary, and reconsider the selection threshold.
Why do self-healing tests pass when the workflow is actually broken?
The healing system may have matched the wrong element.
Review screenshots, element properties, confidence information, and the resulting application state.
Self-healing should never replace meaningful assertions. Even if a control is successfully located, the test must still validate the expected business outcome.
Why is AI-generated test code unreliable?
The model may lack requirements, framework conventions, application context, dependencies, or realistic data.
Provide more explicit context and ask for focused tests rather than an entire end-to-end suite at once.
Most importantly, execute the generated tests and verify their assertions.
AI and automation options for regression testing
Different tools address different parts of the problem.
Predictive test selection
Launchable Predictive Test Selection applies machine learning to historical test and change data to prioritize tests and create subsets according to optimization targets such as duration or confidence.
AI-assisted test generation and analysis
Tricentis Tosca Agentic Test Automation currently supports natural-language-assisted test creation and test-result insights, among other testing tasks.
Self-healing automation
Tricentis Tosca provides self-healing capabilities for supported UI technologies, while mabl documents both conventional and generative-AI-assisted auto-healing strategies.
Developer-assisted test generation
GitHub Copilot can assist developers with generating unit and integration tests and identifying edge cases. Generated tests still require developer review and execution.
Non-AI change-impact analysis
Azure DevOps Test Impact Analysis is worth distinguishing from AI-based approaches. It automatically selects tests affected by a code change using impact information and includes safe fallback behavior when analysis is insufficient. It illustrates that intelligent regression optimization does not always require machine learning.
The appropriate option depends on the bottleneck. A team struggling with UI maintenance needs a different capability from a team whose primary problem is a two-hour backend regression suite.
Limitations and risks of AI in regression testing
AI-assisted regression testing has real limitations.
Historical data can contain bias
If certain components have been poorly tested historically, a model may receive little evidence that those components are risky.
Past test outcomes therefore do not automatically represent future business risk.
New functionality creates cold-start problems
A model has less information about completely new modules, technologies, or dependency relationships.
Risk-based rules and broader coverage are particularly important for novel code.
Models can drift
The relationship between files, tests, and failures changes as architecture evolves.
Research published in 2026 explicitly highlights this challenge and investigates adaptive ML pipelines for test prioritization across changing builds.
Self-healing can hide defects
An automatically repaired locator is useful only when the system selects the intended control.
Incorrect healing can convert a visible automation failure into a misleading pass.
Generative AI can produce incorrect tests
LLMs generate plausible output rather than mathematically guaranteeing that a test expresses the correct requirement.
Tests must therefore be reviewed, executed, and validated.
AI introduces governance considerations
Teams may need to assess:
- What source code or execution data is sent to external services
- Data-retention policies
- Access controls
- Model and vendor security
- Compliance requirements
- Auditability of automated decisions
These considerations can materially affect which AI testing tools are appropriate for regulated or sensitive environments.
Conclusion
AI regression testing can make regression testing more efficient by helping QA teams decide what to test, what to test first, how to maintain automation, and how to interpret failures.
Machine-learning-based test selection and prioritization are particularly useful for large regression suites in continuous integration environments. Generative AI expands those capabilities through assisted test creation and failure analysis, while self-healing techniques can make UI automation more resilient.
The safest implementation is incremental. Begin by collecting reliable execution data and using AI to prioritize rather than remove tests. Measure whether high-risk failures appear earlier. Introduce selective execution only after the model demonstrates acceptable behavior, keep critical tests mandatory, and retain periodic full-suite validation.
AI should make regression testing more informed, not less rigorous.
Frequently Asked Questions
-
Can AI completely automate regression testing?
No. AI can automate or improve several regression-testing activities, but human judgment remains important for defining expected behavior, assessing business risk, reviewing generated tests, investigating ambiguous failures, and making release decisions. The better goal is AI-assisted regression testing, where automation handles high-volume analysis while testers retain control over quality strategy.
-
Can AI reduce regression testing time?
Yes, particularly when the regression suite is large enough for test selection and prioritization to provide value. Meta reported in a 2018 production case that its predictive test-selection system caught more than 99.9% of regressions before they reached other engineers while running roughly one-third of transitively dependent tests. The result is specific to Meta's system and environment and should not be treated as a general industry benchmark.
-
What data does AI need for regression test selection?
Common inputs include historical test outcomes, execution times, source-code changes, file-to-test relationships, code coverage, test characteristics, and previous failures. The exact features depend on the technique. Both research literature and current predictive-selection implementations use combinations of historical execution and system-change information.
-
Does AI replace regression test automation tools such as Selenium or Playwright?
No. Frameworks such as Selenium and Playwright execute automated test logic. AI capabilities can sit around or above automation by determining which tests to execute, generating candidate test code, healing locators, or analyzing results. The technologies are complementary rather than direct replacements.
-
Should every QA team use AI for regression testing?
Not necessarily. A small, fast, stable regression suite may gain little from predictive selection. AI becomes more valuable when teams face problems such as growing execution time, high test-maintenance effort, frequent CI runs, large volumes of failure data, or difficulty deciding which tests are relevant to a change. Start from the testing bottleneck rather than from the technology.
by Rajesh K | Sep 1, 2026 | API Testing, Blog, Latest Post |
End-to-end API testing verifies more than whether individual endpoints return the expected status code. It validates whether a complete business workflow works when multiple API operations, services, authentication mechanisms, and data dependencies interact. Postman is well suited to this type of testing because a Postman Collection can represent an entire workflow: authenticate a user, create data, retrieve it, update it, verify the result, and clean up the generated records. Post-response scripts can validate each step and pass values such as IDs and tokens to later requests.
How do you write E2E API tests in Postman?
To write an end-to-end API test in Postman, organize the API requests for a complete user journey into a collection, execute them in the required sequence, capture values from one response as variables, use those variables in later requests, and add JavaScript assertions with pm.test() and pm.expect() at each stage.
Run the complete collection with the Collection Runner during development and with the Postman CLI in CI/CD to verify that the workflow continues to work after application changes.
Key takeaways
- Model an E2E test around a business workflow, not isolated API endpoints.
- Store related requests in a Postman Collection and run them in a predictable sequence.
- Capture IDs, tokens, and other generated values from responses and reuse them in later requests.
- Add assertions for HTTP status, response data, business rules, and, where useful, response schema.
- Keep environment-specific configuration such as the base URL outside individual requests.
- Create and clean up test data so repeated runs remain independent.
- Run collections locally first, then automate them with the Postman CLI in CI/CD.
- For new Postman v12 workflows using Collection v3, prefer Postman CLI over Newman because Newman doesn’t support Collection v3.
What is end-to-end API testing in Postman?
End-to-end API testing is the process of validating a complete application workflow by sending a sequence of related API requests and verifying that data and behavior remain correct from the beginning of the workflow to the end.
A single API test might confirm that:
returns HTTP 200.
An E2E test goes further. It might validate this workflow:
Authenticate user
↓
Create customer
↓
Create order
↓
Retrieve order
↓
Verify order/customer relationship
↓
Cancel order
↓
Delete test data
Each operation depends on information created by an earlier operation.
Postman describes E2E testing as testing complete application flows that can involve multiple endpoints and APIs. Collections, scripts, variables, and configurable request order make it possible to represent those workflows as automated tests.
E2E testing vs. testing a single API endpoint
Testing an individual endpoint answers a question such as:
Does POST /orders create an order correctly?
End-to-end testing answers a broader question:
Can an authenticated customer complete the entire order workflow successfully, and is the resulting data consistent across the APIs involved?
An effective test strategy normally uses both approaches. Endpoint-level tests provide fast, targeted feedback, while E2E tests verify that components continue to work together.
Why do E2E API tests matter?
Modern applications commonly distribute a single business operation across multiple APIs or services.
An online purchase, for example, can involve:
Authentication
↓
Customer service
↓
Catalog service
↓
Order service
↓
Payment service
↓
Inventory service
Testing each endpoint independently does not prove that the complete workflow succeeds.
An E2E test can detect problems such as:
- an authentication token that one downstream API rejects;
- an ID returned by one service that another service cannot process;
- incorrect mappings between customer and order records;
- a successful API response that leaves the system in the wrong business state;
- incompatible changes between services;
- invalid data propagation across several requests;
- failures that occur only when operations run in their real sequence.
Postman’s documentation specifically positions E2E testing as a way to simulate real-world workflows and identify integration and workflow problems across multiple application components.
How do E2E API tests work in Postman?
A typical Postman E2E workflow has five building blocks.
1. Requests represent actions
Each API request represents one operation in the user journey.
For example:
POST /auth/login
POST /customers
POST /orders
GET /orders/{orderId}
DELETE /orders/{orderId}
DELETE /customers/{customerId}
2. Postman Collections represent test suites
The requests are saved inside a Postman Collection in the order required by the workflow.
The Collection Runner can execute some or all requests in a collection and record their test results. Scripts can also pass data between those requests.
3. Variables pass data between requests
Suppose POST /customers returns:
{
"id": "cust_1845",
"name": "API Test User"
}
The post-response script can capture the ID:
const response = pm.response.json();
pm.collectionVariables.set("customerId", response.id);
A later request can then reference it:
GET {{baseUrl}}/customers/{{customerId}}
Postman supports multiple variable scopes, including global, collection, environment, iteration data, and local variables.
4. Assertions validate each stage
Postman executes JavaScript in post-response scripts. Assertions are commonly created with pm.test() and pm.expect() using Chai-style syntax.
For example:
pm.test("Customer creation succeeds", function () {
pm.response.to.have.status(201);
});
pm.test("Response contains a customer ID", function () {
const body = pm.response.json();
pm.expect(body.id).to.be.a("string");
pm.expect(body.id).to.not.be.empty;
});
5. The collection is executed as one workflow
During development, use the Collection Runner.
For automated builds and deployments, the same tests can be executed using:
postman collection run <collection>
The Postman CLI can execute collections locally or inside CI/CD pipelines.
How to write E2E API tests in Postman step by step
Consider an e-commerce application with this workflow:
- Authenticate.
- Create a customer.
- Create an order for that customer.
- Retrieve the order.
- Verify its business state.
- Delete the generated test data.
The following implementation uses generic endpoints so the pattern can be adapted to most REST APIs. See our guide to testing payment APIs if your checkout workflow specifically involves payment processing.
1. Create a dedicated E2E Postman Collection
Create a collection named:
Then create a folder such as:
Add the requests in business-flow order:
01 - Login
02 - Create Customer
03 - Create Order
04 - Get Order
05 - Delete Order
06 - Delete Customer
Descriptive names make failures easier to understand in Collection Runner and CI reports.
Avoid collections containing requests named only:
Request 1
Request 2
Test
GET API
A test report should immediately tell the engineer which business operation failed.
2. Create environment variables
Create a test environment and define configuration such as:
For example:
baseUrl = https://api.test.example.com
Your requests can then use:
instead of hard-coding the hostname.
This enables the same collection to run against environments such as:
without rewriting every request. Postman environments are designed to group variables whose values differ between execution contexts.
Keep workflow-generated values such as these separate:
accessToken
customerId
orderId
testEmail
These values will be generated during the test.
3. Generate unique test data
Repeated E2E tests should avoid collisions with data created by previous runs.
Postman provides dynamic variables for generating test values, including random UUIDs.
In the Pre-request script for Create Customer:
const runId = pm.variables.replaceIn("{{$randomUUID}}");
pm.collectionVariables.set(
"testEmail",
`postman-e2e-${runId}@example.test`
);
The request body can reference it:
{
"name": "Postman E2E User",
"email": "{{testEmail}}"
}
This makes each run less likely to conflict with data from another execution.
4. Authenticate the workflow
Request:
POST {{baseUrl}}/auth/login
Example request body:
{
"username": "{{username}}",
"password": "{{password}}"
}
The API might return:
{
"access_token": "eyJ..."
}
Add this post-response script:
pm.test("Authentication succeeds", function () {
pm.response.to.have.status(200);
});
const response = pm.response.json();
pm.test("Access token is returned", function () {
pm.expect(response.access_token).to.be.a("string");
pm.expect(response.access_token).to.not.be.empty;
});
if (response.access_token) {
pm.collectionVariables.set(
"accessToken",
response.access_token
);
}
Later requests can use:
Authorization: Bearer {{accessToken}}
For production test suites, avoid storing long-lived credentials directly in the collection. CI systems should supply secrets securely at runtime.
5. Create the customer and capture its ID
Request:
POST {{baseUrl}}/customers
Authorization: Bearer {{accessToken}}
Content-Type: application/json
Body:
{
"name": "Postman E2E User",
"email": "{{testEmail}}"
}
Post-response tests:
pm.test("Customer is created", function () {
pm.response.to.have.status(201);
});
const customer = pm.response.json();
pm.test("Customer response contains required data", function () {
pm.expect(customer.id).to.be.a("string");
pm.expect(customer.email).to.eql(
pm.collectionVariables.get("testEmail")
);
});
if (customer.id) {
pm.collectionVariables.set(
"customerId",
customer.id
);
}
This request performs two jobs:
- verifies that customer creation works;
- supplies customerId to the rest of the workflow.
That data dependency is one of the defining characteristics of E2E API testing.
6. Create an order using the customer ID
Request:
POST {{baseUrl}}/orders
Authorization: Bearer {{accessToken}}
Body:
{
"customerId": "{{customerId}}",
"items": [
{
"sku": "TEST-SKU-001",
"quantity": 1
}
]
}
Post-response script:
pm.test("Order is created", function () {
pm.response.to.have.status(201);
});
const order = pm.response.json();
pm.test("Order belongs to the test customer", function () {
pm.expect(order.customerId).to.eql(
pm.collectionVariables.get("customerId")
);
});
pm.test("New order has expected status", function () {
pm.expect(order.status).to.eql("CREATED");
});
if (order.id) {
pm.collectionVariables.set(
"orderId",
order.id
);
}
Notice that these assertions validate business behavior, not merely HTTP behavior.
A 201 response alone does not prove that the order was associated with the correct customer.
7. Retrieve the order and verify persisted state
Next, send:
GET {{baseUrl}}/orders/{{orderId}}
Authorization: Bearer {{accessToken}}
Post-response script:
pm.test("Order can be retrieved", function () {
pm.response.to.have.status(200);
});
const order = pm.response.json();
pm.test("Returned order has correct ID", function () {
pm.expect(order.id).to.eql(
pm.collectionVariables.get("orderId")
);
});
pm.test("Customer relationship is persisted", function () {
pm.expect(order.customerId).to.eql(
pm.collectionVariables.get("customerId")
);
});
pm.test("Order contains at least one item", function () {
pm.expect(order.items)
.to.be.an("array")
.that.is.not.empty;
});
This is stronger than asserting only the Create Order response.
The follow-up GET verifies that the new state can be retrieved from the system after creation.
8. Validate the response schema where appropriate
Postman can also validate JSON responses against a JSON Schema.
For example:
const schema = {
type: "object",
required: [
"id",
"customerId",
"status",
"items"
],
properties: {
id: {
type: "string"
},
customerId: {
type: "string"
},
status: {
type: "string"
},
items: {
type: "array"
}
}
};
pm.test("Order response matches schema", function () {
pm.response.to.have.jsonSchema(schema);
});
Schema validation and business assertions solve different problems.
The schema confirms that the response has the expected structure. Assertions such as:
pm.expect(order.customerId).to.eql(expectedCustomerId);
verify that the returned values are correct for the workflow.
Use both when both contract shape and business state matter.
9. Clean up the generated order
Send:
DELETE {{baseUrl}}/orders/{{orderId}}
Then verify:
pm.test("Order cleanup succeeds", function () {
pm.expect(pm.response.code).to.be.oneOf([
200,
204
]);
});
Use the response code required by your API contract rather than blindly accepting both values in a real test suite.
10. Clean up the generated customer
Send:
DELETE {{baseUrl}}/customers/{{customerId}}
Then remove transient variables if they are no longer required:
pm.collectionVariables.unset("customerId");
pm.collectionVariables.unset("orderId");
pm.collectionVariables.unset("testEmail");
pm.collectionVariables.unset("accessToken");
Cleanup matters because abandoned test records can create:
- duplicate-data failures;
- polluted test databases;
- misleading reports;
- storage growth;
- dependencies between otherwise unrelated test runs.
Practical E2E example: customer checkout workflow
The complete scenario now looks like this.
Business scenario
Verify that an authenticated user can create a customer and place an order, and that the resulting order is retrievable with the correct customer relationship.
Preconditions
The test environment must provide:
baseUrl
valid test credentials
TEST-SKU-001 or another known test product
Input
A unique email address is generated for each collection run.
Workflow
Login
↓
Capture accessToken
↓
Create customer
↓
Capture customerId
↓
Create order
↓
Capture orderId
↓
GET order
↓
Verify business state
↓
Delete order
↓
Delete customer
Expected result
The collection succeeds only if:
- authentication succeeds;
- a new customer can be created;
- the API returns a valid customer ID;
- an order can be created using that customer;
- the persisted order references the same customer;
- the expected items and order state are returned;
- generated test records can be removed.
Example failure condition
Suppose Create Order returns:
{
"id": "ord_8921",
"customerId": "cust_9999",
"status": "CREATED"
}
while the test created:
The HTTP request technically succeeded, but the E2E test should fail because:
pm.expect(order.customerId).to.eql(
pm.collectionVariables.get("customerId")
);
detects the incorrect relationship.
This illustrates why effective API E2E tests validate business continuity between requests, not just response status codes.
E2E testing vs. unit, integration, and contract API testing
| S. No |
Testing type |
Primary purpose |
Typical scope |
Example |
| 1 |
Unit/API endpoint test |
Verify a small unit of API behavior |
One function or endpoint behavior |
Check that POST /orders rejects missing data |
| 2 |
Contract test |
Verify request/response structure |
API interface |
Confirm GET /orders/{id} conforms to its documented schema |
| 3 |
Integration test |
Verify components exchange data correctly |
Two or more interacting components |
Confirm Order Service successfully calls Inventory Service |
| 4 |
End-to-end test |
Verify a complete business journey |
Multiple endpoints, services, and state transitions |
Authenticate, create customer, create order, retrieve order, clean up |
Postman supports multiple API testing styles, including integration, E2E, regression, and performance testing.
The difference is primarily scope and intent.
Do not replace all lower-level tests with E2E tests. E2E suites typically exercise more components and therefore tend to be harder to diagnose when something fails. See our REST API Testing Checklist for the endpoint-level checks that should sit underneath your E2E suite, and our gRPC API Testing guide if any of your services communicate over gRPC instead of REST.
Best practices for writing maintainable Postman E2E tests
Test business journeys, not arbitrary endpoint sequences
Start with a real user or system workflow.
Good:
Register -> Login -> Create Project -> Retrieve Project -> Delete Project
Less useful:
GET endpoint A -> POST endpoint B -> GET endpoint C
unless those requests represent an actual business process.
Keep environment-specific values out of requests
Use:
instead of:
https://qa-server-17.example.com/api
Environment variables make the same collection reusable across deployment environments.
Capture values instead of hard-coding IDs
Avoid:
when request 12345 was created manually weeks ago.
Prefer:
pm.collectionVariables.set(
"orderId",
pm.response.json().id
);
followed by:
This makes the flow self-contained.
Make test data unique
Generate unique usernames, emails, reference numbers, or UUIDs when the application requires uniqueness.
For example:
const id = pm.variables.replaceIn("{{$randomUUID}}");
Postman’s dynamic variables are specifically designed for runtime-generated values.
Assert business outcomes
Status-code assertions are necessary but insufficient.
Instead of only:
pm.response.to.have.status(200);
also validate:
pm.expect(order.customerId)
.to.eql(expectedCustomerId);
pm.expect(order.status)
.to.eql("CREATED");
Put reusable assertions at the right level
Postman allows post-response scripts at the collection, folder, and request levels. Collection-level scripts run for requests in the collection, while folder scripts can apply shared logic to requests within that folder.
Use this capability for genuinely shared behavior such as:
pm.test("Response is not a server error", function () {
pm.expect(pm.response.code).to.be.below(500);
});
Avoid copying the same code into dozens of requests.
Keep each test independent
An E2E test should preferably create the records it needs.
Avoid making Test B depend on data that Test A happened to leave behind.
Independent tests are easier to:
- rerun;
- parallelize;
- debug;
- execute in CI;
- move between environments.
Clean up test data
Treat cleanup as part of the workflow.
A test that repeatedly creates users, orders, projects, or subscriptions without removing them can eventually become unreliable.
Use descriptive assertion names
Prefer:
pm.test(
"Created order belongs to current customer",
function () {
// assertion
}
);
over:
pm.test("Test 4", function () {
// assertion
});
Postman displays test names in its results, so clear names reduce triage time.
Use conditional workflow logic carefully
Postman supports changing collection execution order with:
pm.execution.setNextRequest()
This can implement branches and loops during collection runs.
However, use branching only when the business flow genuinely requires it.
A heavily interconnected collection can become difficult to understand and debug.
Common mistakes when writing E2E tests in Postman
| S. No |
Mistake |
Why it happens |
Impact |
Recommended fix |
| 1 |
Checking only HTTP status codes |
Tests start as simple endpoint checks |
Business failures remain undetected |
Assert response values and persisted state |
| 2 |
Hard-coding IDs |
Test was built manually first |
Tests break when data changes |
Capture IDs dynamically |
| 3 |
Sharing one test account across concurrent runs |
Setup appears easier |
Parallel tests interfere with each other |
Create isolated or uniquely identified data |
| 4 |
Hard-coding URLs |
Collection starts in one environment |
Difficult to run elsewhere |
Use {{baseUrl}} and environments |
| 5 |
Leaving test records behind |
Cleanup is treated as optional |
Test environment becomes polluted |
Add cleanup requests |
| 6 |
Storing secrets in collection JSON |
Convenience during development |
Credentials can be exposed |
Inject credentials securely |
| 7 |
Adding arbitrary waits |
Testing asynchronous workflows |
Slow and flaky tests |
Poll for a defined state with a timeout |
| 8 |
Making tests depend on execution history |
Existing records seem convenient |
Fresh environments fail |
Build prerequisites during each workflow |
| 9 |
Creating too many branches |
Collection becomes a mini-program |
Debugging becomes difficult |
Keep primary workflows explicit |
| 10 |
Running only from the Postman UI |
Automation is postponed |
Regressions reach later pipeline stages |
Add CLI execution to CI |
Troubleshooting Postman E2E tests
Why does the first request pass but later requests fail?
The most common cause is that a value required by later requests was never saved correctly.
Check whether the previous script contains something like:
const data = pm.response.json();
pm.collectionVariables.set(
"orderId",
data.id
);
Then inspect the resolved variable before the failing request.
Also confirm that the JSON property is actually called:
rather than:
Why is {{orderId}} unresolved?
The variable may:
- never have been created;
- be stored in the wrong scope;
- have a different spelling;
- have been unset too early;
- be overridden by another variable with the same name.
Postman uses variable scope precedence when several variables share a name, so avoid unnecessarily reusing the same key at different scopes.
Why does the workflow work with Send but not when I run the collection?
Individual Send operations do not reproduce every collection-run behavior.
For example, pm.execution.setNextRequest() affects collection execution but has no effect when a request is executed individually using Send.
Test workflow behavior with the Collection Runner before debugging request-order logic.
Why does the test pass individually but fail in the complete suite?
Possible causes include:
- shared variables being overwritten;
- another request deleting required data;
- authentication state changing;
- request-order dependencies;
- duplicated test data;
- a previous request failing silently;
- eventual consistency between services.
Start with the first request whose business assertion fails rather than only examining the final failed request.
How do I test an asynchronous API workflow?
Suppose an API returns:
{
"jobId": "job-123",
"status": "PROCESSING"
}
Do not add an arbitrary fixed sleep and assume processing will finish.
Instead, design a polling request:
and continue until:
or a defined retry/timeout limit is reached.
Postman’s workflow controls can be used to direct which request runs next during a collection execution.
Always include a maximum attempt count to prevent an infinite loop.
How to run E2E tests with the Postman Collection Runner
Once individual requests work:
- Open the collection.
- Select Run.
- Choose a functional local run.
- Select the required environment.
- Verify the request sequence.
- Start the collection run.
- Review failed requests and assertions.
The Collection Runner executes the requests and records test results for the run. Scripts can also transfer data between requests and modify workflow execution.
A successful run should resemble:
01 - Login PASS
02 - Create Customer PASS
03 - Create Order PASS
04 - Get Order PASS
05 - Delete Order PASS
06 - Delete Customer PASS
Run the collection multiple times before adding it to CI.
One successful execution does not prove that the suite has isolated test data or reliable cleanup.
How to run Postman E2E API tests in CI/CD
Postman’s current command-line tool for running collections is the Postman CLI.
Install it with npm when Node.js and npm are available:
npm install -g postman-cli
Postman documents postman collection run for executing collections locally and from CI/CD.
A local collection file can be run with:
postman collection run ./tests/orders-e2e.json
An environment file can be supplied with:
postman collection run \
./tests/orders-e2e.json \
--environment ./tests/qa.environment.json
The CLI also supports injecting environment variables at runtime with options such as --env-var.
For example:
postman collection run \
./tests/orders-e2e.json \
--environment ./tests/qa.environment.json \
--env-var "username=$E2E_USERNAME" \
--env-var "password=$E2E_PASSWORD"
The exact syntax for referencing CI secrets varies by CI provider.
For Postman cloud-backed operations, CI can authenticate using:
postman login --with-api-key "$POSTMAN_API_KEY"
Postman recommends API-key authentication for CI/CD use cases requiring CLI authentication.
A useful pipeline structure is:
Build application
↓
Run unit tests
↓
Deploy test environment
↓
Run Postman API E2E collection
↓
Pass?
/ \
Yes No
↓ ↓
Continue Stop pipeline
The Postman CLI returns a non-zero exit code when a failure is detected, which allows CI systems to fail the corresponding step.
Postman tools and implementation options for E2E testing
| S. No |
Option |
Best use |
Key consideration |
| 1 |
Individual request + Post-response tests |
Developing and debugging one step |
Does not validate the entire workflow |
| 2 |
Collection Runner |
Local E2E development and manual regression runs |
Requires an interactive/local execution |
| 3 |
Scheduled collection runs |
Recurring automated API checks |
Execution model differs from local workflows |
| 4 |
Postman CLI |
CI/CD and command-line automation |
Recommended approach for modern Postman collections |
| 5 |
Newman |
Existing command-line collection automation |
Does not support Postman v12 Collection v3 |
| 6 |
Mock servers |
Replacing unavailable external dependencies |
A mocked dependency does not prove the real integration works |
Postman continues to document Newman as a command-line collection runner, but its current documentation states that Newman isn’t compatible with the Collection v3 format used by Postman v12 and later for Native Git workflows. New projects using that format should use Postman CLI.
If you’re weighing Postman against a lighter-weight alternative for this kind of workflow, see our Postman vs Bruno comparison and our Bruno Tutorial for API Testing. And if you want Postman’s AI assistant to help scaffold these tests, our Postbot AI Tutorial covers that workflow.
Limitations and risks of Postman E2E testing
E2E failures can be difficult to diagnose
A workflow may involve:
API gateway
authentication
database
message queue
third-party API
multiple microservices
A failed final assertion does not automatically identify which component introduced the problem.
Keep assertions at important intermediate boundaries so the first incorrect state is easier to locate.
Tests can become flaky when they depend on shared systems
Instability can come from:
- shared test data;
- asynchronous processing;
- rate limits;
- downstream services;
- changing environments;
- network conditions.
Design the workflow to minimize uncontrolled dependencies.
E2E suites should not replace lower-level testing
Running an entire business workflow to verify every small validation rule is inefficient.
Keep narrow behavior in unit, contract, or endpoint-level tests and reserve E2E tests for critical cross-component journeys.
Secrets require special handling
Postman’s Vault can securely provide sensitive data for supported interactive workflows, but pm.vault methods aren’t supported by scheduled collection runs, monitors, the Postman CLI, or Newman.
For CI/CD, inject required secrets through the CI platform or another approved secrets-management mechanism rather than designing the suite around local Vault access.
Postman CLI has an OAuth 2.0 limitation
Postman’s current documentation states that the Postman CLI doesn’t support performing OAuth 2.0 authentication itself.
If your E2E workflow requires OAuth 2.0, design automation so the required token can be obtained or supplied through an appropriate non-interactive mechanism supported by your identity system.
Do not hard-code long-lived access tokens into the collection.
E2E suites become expensive when they grow without prioritization
A collection with every possible business permutation may eventually become:
- slow;
- difficult to maintain;
- expensive to troubleshoot;
- unreliable in shared environments.
Prioritize journeys such as:
authentication
account creation
checkout/payment
critical CRUD workflow
authorization boundaries
high-risk integrations
Then cover detailed validation behavior with narrower test layers.
Conclusion
This guide on Postman E2E API testing started with a business workflow rather than a list of independent endpoints.
Build the workflow as a collection, create its prerequisites during the run, capture generated values such as IDs, pass those values into subsequent requests, and assert the business state at each important boundary. Finish by removing generated data so the test remains repeatable.
A practical progression is:
Build one critical workflow
↓
Make test data independent
↓
Add meaningful assertions
↓
Verify cleanup
↓
Run repeatedly in Collection Runner
↓
Execute with Postman CLI
↓
Add to CI/CD
The result is more than a collection of API checks. It becomes an executable representation of how an important application workflow is expected to behave from start to finish.
Frequently Asked Questions
-
Can Postman be used for end-to-end API testing?
Yes. Postman supports E2E API workflows by organizing related API requests into collections, executing them in sequence, passing data between requests using variables, and validating responses through JavaScript test scripts. Collections can be executed manually through Collection Runner or automated using Postman CLI.
-
How do I pass data from one Postman request to another?
Parse the response in a post-response script and store the required value in a suitable variable, then reference it later with a variable placeholder. Postman's scripting API supports collection, environment, global, local, and iteration-data variable scopes.
-
Should I use environment variables or collection variables?
Use environment variables for values that vary between environments, such as baseUrl. Collection variables are useful for values that belong to the test workflow itself, such as customerId or orderId. Choose the narrowest practical scope.
-
What should an E2E API test validate?
A strong E2E API test validates HTTP result, response structure, required fields, business values, relationships between resources, persisted state, downstream outcome, and cleanup, not just a 200 OK status.
-
How many E2E API tests should a project have?
There is no universal target. Cover the workflows whose failure would create the most significant user or business impact, such as authentication, account provisioning, core transactions, payments, and permissions.
-
Can Postman E2E tests run automatically?
Yes. Postman collections can be run manually, scheduled through supported Postman features, or executed from CI/CD pipelines with Postman CLI.
-
Is Newman still useful for Postman automation?
Newman remains available and can run compatible Postman Collections from the command line, but it does not support Collection v3 used with Postman v12 Native Git workflows. For new Collection v3 automation, use Postman CLI.
-
Should E2E API tests use fixed or dynamic test data?
Prefer dynamically created or isolated data when the workflow changes application state. Fixed reference data can still be appropriate when intentionally stable. Dynamic IDs reduce collisions between repeated or concurrent test runs.
-
Should cleanup run when an earlier test fails?
Ideally, yes. Without resilient cleanup, a failure mid-workflow can leave partially created data behind. Design cleanup so it can tolerate missing resources and still remove anything successfully created.