API Performance Testing: Response Time, Throughput, and Scalability

API performance testing tells you whether your service can handle real traffic, not just whether it returns the right answer once. A single successful request proves almost nothing about how that same endpoint behaves under concurrent load, rising latency, or resource contention. This guide breaks performance down into three measurable dimensions, response time, throughput, and scalability, and shows how to define objective thresholds instead of vague goals like “the API should be fast.” You’ll also see a complete k6 example, common mistakes, and troubleshooting guidance for when results don’t match expectations.

What is API performance testing?

API performance testing measures how quickly, reliably, and efficiently an API processes requests as demand changes. The three core measurements are response time, which shows how long requests take; throughput, which shows how many requests the API processes per unit of time; and scalability, which shows whether acceptable performance can be maintained as traffic or system capacity increases.

A useful API performance test does more than generate traffic. It defines a realistic workload, measures latency distributions and errors, observes resource saturation, and determines the highest load the API can sustain while meeting its performance objectives.

Key takeaways

• Measure percentile response times such as p50, p95, and p99, not only averages, because averages can hide slow requests.
• Define throughput in requests per second (RPS) or another workload-specific unit and distinguish offered traffic from successfully completed traffic.
• Treat scalability as a relationship between load, latency, errors, throughput, and resource capacity, not as a single metric.
• Use an open workload model when request arrivals should remain independent of API response time.
• Establish explicit pass/fail thresholds before a test rather than deciding whether performance is acceptable afterward.
• Correlate load-test results with server-side CPU, memory, database, connection-pool, queue, and dependency telemetry to locate bottlenecks.

Google’s Site Reliability Engineering guidance similarly recommends monitoring latency, traffic, errors, and saturation for user-facing systems rather than interpreting latency in isolation.

What does API performance testing measure?

API performance testing evaluates the behavior of an application programming interface under controlled demand.

It commonly covers four related signals:

OpenTelemetry, for example, defines the HTTP server metric http.server.request.duration as a histogram representing the duration of HTTP server requests.

API performance testing is different from functional testing. Functional testing asks whether an endpoint returns the correct result. Performance testing asks whether it continues returning correct results within acceptable timing and capacity constraints as demand changes.

The two should still be combined during a load test. A fast 500 Internal Server Error is not a successful performance result.

Why is API performance testing important?

An API can work correctly with one request and still fail badly under production traffic.

Performance problems commonly emerge when concurrency rises and finite resources begin to saturate. Examples include exhausted database connection pools, CPU contention, thread-pool limits, lock contention, downstream service delays, memory pressure, rate limits, and request queues.

These conditions have direct technical and business consequences:

• increasing API latency can slow mobile apps, web applications, and integrations;
• saturated queues can increase tail latency before outright errors appear;
• overload can cause timeouts, retries, and cascading traffic;
• insufficient capacity can make product launches or peak periods unreliable;
• overprovisioning without measurement can increase infrastructure cost.

Google SRE notes that overloaded queues increase request latency because requests spend longer waiting before processing. It also warns that retries can amplify traffic during failures and contribute to cascading failures.

Performance testing therefore helps answer two different questions:

Performance: Does the API meet its objectives at the expected workload?

Capacity: How much workload can the API sustain before those objectives are violated?

How do you measure API response time?

API response time is the elapsed time associated with processing an API request, measured between explicitly defined start and end points. Because different tools use different timing boundaries, teams should document exactly what their response-time metric includes.

For example, Apache JMeter defines elapsed time from immediately before sending a request until after the last response has been received. Its separate latency metric runs until the first part of the response has been received. Our JMeter Tutorial: An End-to-End Guide covers these definitions in more depth if you’re setting up JMeter for the first time.

Grafana k6 defines http_req_duration as:

http_req_sending + http_req_waiting + http_req_receiving

Its metric therefore excludes initial DNS lookup and connection-establishment time from http_req_duration; k6 exposes other timing metrics for connection and TLS activity, per k6’s metrics documentation.

This difference matters when comparing test results. A statement such as “the API responds in 180 ms” is incomplete unless the measurement boundary is known.

Break response time into components

Depending on the tool and protocol, investigate:

• DNS resolution
• TCP connection establishment
• TLS handshake
• request transmission
• server processing and queueing
• time to first byte
• response-body transfer

A rising total response time does not automatically mean application code became slower. Connection setup, network conditions, database calls, external APIs, or server queues can contribute.

Why p95 and p99 matter more than average response time

Suppose most API calls finish quickly but a small fraction take several seconds. An average may still appear acceptable.

Percentiles reveal the distribution.

• p50: 50% of requests complete at or below this duration.
• p95: 95% complete at or below this duration.
• p99: 99% complete at or below this duration.

Google SRE recommends considering percentiles because a mean can hide significant changes in tail latency and because latency distributions are not necessarily normally distributed.

For an interactive API, a performance requirement might therefore be expressed as:

95% of successful requests must complete within 300 ms and 99% within 600 ms under the defined peak workload.

Those numbers are examples, not universal recommendations. Appropriate thresholds depend on the API’s business purpose, architecture, client expectations, and existing service-level objectives.

How do you measure API throughput?

API throughput is the amount of request-processing work completed during a specified period.

For HTTP APIs, it is commonly reported as:

Throughput = Number of requests / Measurement duration

Apache JMeter uses this definition, calculating throughput from request count divided by elapsed test time.

Common units include:

• requests per second (RPS);
• requests per minute;
• transactions per second;
• records processed per second;
• megabytes per second for data-intensive APIs.

Grafana k6’s http_reqs metric counts generated HTTP requests and reports their rate, providing a direct view of generated request throughput.

Offered throughput versus successful throughput

Do not report only generated traffic.

Assume a load generator sends 1,000 requests per second, but 150 fail with errors or timeouts. The API should not be described simply as “handling 1,000 RPS.”

Track at least:

Offered load: traffic sent toward the API.

Completed throughput: requests receiving responses.

Successful throughput: requests that both complete and satisfy correctness criteria.

This distinction becomes particularly important around the system’s saturation point.

What is API scalability?

API scalability is the ability of a system to accommodate increasing demand while keeping response time, errors, and resource utilization within acceptable limits.

Scalability is therefore not synonymous with throughput.

An API may process more requests as load rises but simultaneously experience unacceptable p99 latency. Another system may maintain stable latency but stop increasing throughput because a database or worker pool has reached capacity.

A scalability test should examine the relationship:

Increasing load → latency → successful throughput → errors → resource saturation

For systems that support horizontal scaling, another dimension is:

Increasing resources → additional SLO-compliant capacity

AWS’s Well-Architected performance guidance recommends defining performance KPIs, monitoring performance-critical areas, and load testing workloads as part of performance engineering.

How does API performance testing work?

A repeatable API performance test generally follows this process:

• Define the workload. Identify endpoints, request mix, payload sizes, authentication behavior, and expected traffic.
• Define performance objectives. Specify response-time percentiles, error limits, and required throughput.
• Prepare representative data. Avoid unrealistic reuse of a single user, record, or cached request unless production behaves that way.
• Generate controlled traffic. Increase load according to the selected load model.
• Measure client-side results. Collect response duration, throughput, error rate, and dropped work.
• Observe server-side telemetry. Monitor CPU, memory, garbage collection, connections, queues, databases, caches, and downstream services.
• Find the constraint. Identify the resource or dependency associated with the point at which performance deteriorates.
• Repeat after a change. Compare results under the same test conditions.

The objective is not to produce the largest possible RPS number. It is to determine the amount of demand the system handles while still satisfying the defined service objectives.

Step-by-step: How to run an API performance test

1. Establish a baseline

Start with a small workload.

Confirm:

• requests are functionally correct;
• authentication works;
• test data is valid;
• responses pass assertions;
• the load generator itself is not resource-constrained;
• server telemetry is available.

A baseline gives you a reference against which higher-load behavior can be compared.

2. Define measurable performance thresholds

Avoid goals such as:

The API should be fast.

Use measurable criteria instead:

• p95 response time < 300 ms;
• p99 response time < 600 ms;
• HTTP failure rate < 1%;
• successful throughput ≥ 500 RPS.

Grafana k6 supports thresholds for percentile response times, errors, and custom metrics and can fail a test automatically when those conditions are violated.

Threshold values should come from business requirements, production SLOs, baselines, or capacity plans, not from arbitrary industry averages.

3. Model production traffic

Include representative:

• endpoint ratios;
• GET/POST/PUT/DELETE traffic;
• payload sizes;
• authenticated and anonymous sessions;
• cacheable and non-cacheable requests;
• test data;
• think time where relevant;
• geographic or network conditions where they affect the API.

Testing one inexpensive GET endpoint at maximum speed tells you little about a production workload dominated by writes, database transactions, and external calls.

4. Choose an appropriate load model

A closed model typically uses a fixed population of virtual users that waits for one iteration to finish before beginning another. Consequently, when the system slows down, iteration starts can also slow down.

An open model schedules arrivals independently of response time.

k6 specifically warns that closed-model throughput can fall when response times increase because iteration duration controls how quickly new iterations begin. Its arrival-rate executors use an open model so iteration starts can be controlled independently of system response time.

For capacity tests where production traffic arrives at an externally determined rate, an open workload model is often more representative.

5. Ramp load instead of immediately maximizing it

Increase demand in controlled stages.

For example:

50 → 100 → 200 → 400 → 600 → 800 RPS

Hold each level long enough to observe stable behavior.

At every stage, record:

• p50, p95, and p99 response times;
• successful RPS;
• error percentage;
• CPU utilization;
• memory and garbage collection;
• database latency;
• connection-pool usage;
• queue depth;
• downstream latency;
• instance count.

The point at which an objective first fails is more useful than a single maximum-load measurement.

6. Find the SLO-compliant capacity

Define API capacity as the highest sustained workload that still satisfies all required performance and correctness thresholds.

For example, if the API meets every requirement at 500 RPS but p95 response time crosses its objective at 600 RPS, its tested SLO-compliant capacity in that environment lies below 600 RPS.

That result is specific to the tested:

• software version;
• infrastructure;
• dataset;
• workload mix;
• configuration;
• test duration.

It should not be presented as a universal capacity figure.

7. Test scaling behavior

Repeat the capacity test after changing capacity.

For example:

• one application instance;
• two instances;
• four instances.

Then compare the maximum SLO-compliant throughput.

A useful internal heuristic is:

Scalability efficiency = throughput growth factor / resource growth factor

If doubling an application tier from two to four instances increases SLO-compliant capacity from 500 to 900 RPS:

(900 / 500) ÷ (4 / 2) = 0.90

The resulting 90% is not an industry-standard scalability metric. It is simply a useful engineering ratio for comparing your own scaling experiments.

Sublinear scaling can indicate shared bottlenecks such as a database, cache, network link, lock, or downstream dependency.

Practical example: Load testing a product API with k6

Consider a retail API:

GET /v1/products

Assume the engineering team has established these illustrative performance objectives:

• p95 response time below 300 ms;
• p99 response time below 600 ms;
• HTTP failure rate below 1%.

The test should increase request arrival rate independently of API response time.

import http from 'k6/http';
import { check } from 'k6';

const BASE_URL = __ENV.BASE_URL;

export const options = {
  scenarios: {
    product_api: {
      executor: 'ramping-arrival-rate',
      startRate: 50,
      timeUnit: '1s',
      preAllocatedVUs: 200,
      maxVUs: 1000,
      stages: [
        { target: 100, duration: '2m' },
        { target: 250, duration: '3m' },
        { target: 500, duration: '5m' },
        { target: 750, duration: '5m' },
      ],
    },
  },
  thresholds: {
    'http_req_duration{endpoint:products}': [
      'p(95)<300',
      'p(99)<600',
    ],
    http_req_failed: ['rate<0.01'],
  },
};

export default function () {
  const response = http.get(
    `${BASE_URL}/v1/products?limit=20`,
    {
      tags: {
        endpoint: 'products',
      },
    }
  );
  check(response, {
    'status is 200': (r) => r.status === 200,
  });
}

k6’s ramping-arrival-rate executor changes the iteration arrival rate over time, while its thresholds allow percentile-duration and error criteria to be evaluated automatically.

Expected output

Do not expect a particular RPS or latency result in advance.

Instead, the expected outcome is a clear answer to these questions:

• At which load stage does p95 first exceed 300 ms?
• When does p99 exceed 600 ms?
• Does the error rate remain below 1%?
• Does successful throughput continue increasing with offered load?
• Which server resource saturates first?
• Are scheduled iterations being dropped by the load generator?

k6 exposes dropped_iterations when scheduled iterations cannot start. With arrival-rate executors, persistent dropped iterations can occur when there are insufficient available VUs, including situations where system-under-test performance degrades and iterations take increasingly long to finish.

Error condition

Suppose latency rises sharply at the 750-RPS stage while database pool utilization reaches its configured maximum.

The appropriate conclusion is not simply:

k6 cannot generate 750 RPS.

First verify load-generator capacity. If it is healthy, correlate the change with server telemetry. Connection-pool saturation may be the bottleneck, or it may only be a symptom of slower database queries.

The next experiment should isolate that hypothesis.

Response time vs. latency vs. throughput vs. scalability

Terminology varies between tools. Apache JMeter, for example, distinguishes elapsed time from its first-response latency metric, while some engineering discussions use “latency” more broadly for overall request duration.

Document your exact definition before comparing measurements.

Need Help With Your API Performance Testing?

Talk to Our Performance Testing Experts

API performance testing best practices

Define the measurement boundary

State whether response time includes DNS, TCP/TLS setup, redirects, response-body transfer, client processing, or only server-side duration.

This prevents misleading comparisons between tools and dashboards.

Measure percentiles, not only averages

Track at least p50 and one or more high percentiles such as p95 or p99.

Tail latency often exposes queuing, contention, garbage collection, or slow dependencies that averages obscure. Google SRE explicitly recommends care when aggregating latency and discusses the value of high percentiles.

Validate correctness under load

Check status codes and important response content.

A performance test should not classify malformed, stale, or incorrect responses as successful just because they are fast.

Separate endpoints and workload classes

A single global p95 can hide an endpoint that performs poorly.

Segment metrics by:

• route;
• method;
• response status;
• payload class;
• region;
• customer or workload class where appropriate.

OpenTelemetry HTTP conventions include attributes such as HTTP request method and route-related telemetry that can support this type of analysis.

Test beyond expected peak load

Expected-peak testing answers whether capacity is sufficient.

Testing above expected peak reveals the degradation pattern: gradual slowdown, rate limiting, timeouts, queue growth, or abrupt collapse.

Google SRE describes testing services beyond rated capacity so overload behavior can be understood before production experiences it.

Monitor the load generator

Verify that the client running the test has sufficient:

• CPU;
• memory;
• network bandwidth;
• sockets;
• file descriptors;
• virtual users or workers.

Otherwise you may measure the test infrastructure instead of the API.

Repeat tests under controlled conditions

Record:

• build or commit;
• environment;
• instance sizes;
• autoscaling configuration;
• database size;
• cache state;
• tool version;
• workload profile;
• duration.

A result without its test environment is difficult to reproduce.

Common API performance testing mistakes

Troubleshooting API performance tests

Why does p99 increase while average response time stays stable?

A subset of requests is becoming much slower.

Check the latency distribution rather than the mean and correlate slow requests with database queries, garbage collection, queueing, downstream dependencies, request types, and resource saturation.

High percentiles are designed to expose this long-tail behavior.

Why does throughput stop increasing when more virtual users are added?

The system or workload model may have reached a limiting factor.

Check:

• whether response times are increasing;
• CPU and memory;
• database and connection pools;
• queues and thread pools;
• network limits;
• rate limiting;
• load-generator saturation.

With a closed workload model, increasing response time can itself reduce the rate at which new work begins.

Why are errors increasing only at high load?

The system may be crossing a capacity boundary.

Inspect status codes and distinguish:

• application failures;
• 429 Too Many Requests;
• gateway errors;
• timeouts;
• connection failures;
• database exhaustion;
• downstream failures.

Then correlate the first rise in errors with saturation telemetry.

Why are k6 iterations being dropped?

For an arrival-rate scenario, k6 can report dropped iterations when no VU is available to begin scheduled work.

If drops occur immediately, the test configuration may need more preallocated VUs. If they rise later as latency increases, the system under test may be degrading enough that virtual users remain occupied longer.

Why are load-test results inconsistent between runs?

Look for uncontrolled variables:

• autoscaling state;
• cache warming;
• database data volume;
• noisy infrastructure;
• deployment differences;
• background jobs;
• external service behavior;
• client-machine capacity;
• network location.

Make these conditions explicit in the test report and repeat enough controlled runs to determine whether the difference is reproducible.

Which tools can be used for API performance testing?

Grafana k6

k6 is useful for code-based performance tests and supports virtual-user and arrival-rate workload models, built-in HTTP metrics, thresholds, checks, and multiple scenario executors.

It is particularly convenient when performance tests are managed alongside application code and run in automated pipelines. If you’re deciding between k6 and JMeter, see our comparison, JMeter vs Gatling vs k6: Comparing Top Performance Testing Tools.

Apache JMeter

Apache JMeter provides a mature GUI- and configuration-driven approach and reports metrics including elapsed time, latency, errors, percentiles, and throughput. Its documentation defines throughput as request count divided by total measurement time.

It can be useful for teams that prefer a visual test-plan model or already maintain an established JMeter test suite. If you need to scale that setup beyond a single machine, our Cloud Performance Testing with Apache JMeter guide covers distributed testing.

OpenTelemetry

OpenTelemetry is not a replacement for a load generator. It is useful for instrumenting the system under test so that load-test traffic can be correlated with server and dependency telemetry.

Its HTTP semantic conventions define standardized metrics including http.server.request.duration and http.client.request.duration.

A strong performance-testing environment commonly combines a traffic generator with observability rather than relying on either one alone. For a broader roundup of options beyond these three, see our Top Performance Testing Tools guide.

Limitations and risks of API performance testing

Performance testing is an experiment, not a perfect forecast of production.

Results can be distorted by differences in:

• data distribution;
• cache behavior;
• infrastructure;
• geography;
• dependency performance;
• traffic composition;
• request bursts;
• client behavior;
• autoscaling;
• production background workloads.

There is also a risk in testing production directly. High-volume tests can affect real customers, trigger external API costs, consume quotas, alter data, or activate security controls.

Use isolated environments when required, sanitize test data, obtain authorization for production testing, and understand downstream rate limits before generating substantial traffic.

Another limitation is that test results age. Capacity measured several releases ago may no longer represent the current architecture. Google SRE specifically recommends using load testing rather than relying on historical resource-to-capacity assumptions.

Conclusion

Effective API performance testing answers more than “How fast is this endpoint?” It establishes how response-time percentiles change as demand grows, how much successful throughput the API can sustain, where errors begin, which resources saturate, and whether adding capacity produces useful scaling.

Start by defining a realistic workload and explicit performance objectives. Measure p95 and p99 alongside throughput and errors, use a workload model that reflects how traffic actually arrives, and correlate every load stage with server-side telemetry.

The most useful outcome is a repeatable capacity boundary: the highest sustained workload your API can handle while still meeting its defined service objectives. Once that baseline exists, performance testing becomes a regression-detection and capacity-planning discipline rather than a one-time benchmark.

Frequently Asked Questions