Performance Testing for SDETs: Which Test to Run and When

My application passed every single functional test in the CI pipeline, but the moment I fired up a Locust load test, it slowed to an absolute crawl. I wasn't even dealing with real users yet—just simulating concurrent traffic. While my unit tests confirmed the AI logic was perfect, the load test exposed a massive choke point: a synchronous function was secretly blocking my entire async event loop under pressure.

Functional tests tell you that a feature works for one user in a vacuum. That gap—between one user and one thousand—is where this article starts.

What Performance Testing Actually Is

Performance testing measures how your system behaves under load, not whether the functional logic is correct. Think of functional testing as checking if a car’s steering wheel turns the tires. Performance testing is putting five people in that car, filling the trunk with bricks, and seeing if it can still climb a steep hill without the engine overheating.

Why It’s Useful

The most obvious reason is catching bottlenecks before your users do. There is nothing worse than launching a highly anticipated feature only to watch the database lock up because a missing index wasn't apparent during local testing.

It also sets baseline capacity expectations. If you know your infrastructure can comfortably handle 500 requests per second, you can provision auto-scaling rules intelligently rather than guessing at CPU thresholds.

Crucially, it protects against regressions. A developer might merge a PR that refactors a query and accidentally makes an endpoint three times slower. Functional tests will still happily pass because the correct JSON is returned, but a performance gate will catch the degradation instantly.

The Types of Performance Tests, and When to Use Each

Performance testing isn't just one thing. It's a spectrum of tests designed to answer very different questions about your infrastructure.

• Smoke test: Runs a minimal load (e.g., one to five virtual users) to verify that the test script itself works and the system can handle basic traffic without immediate errors.
• Load test: Simulates expected normal and peak traffic to verify that the system meets defined performance targets, such as response times, throughput, and error-rate thresholds.
• Stress test: Pushes the system beyond its expected operating load to observe how it degrades, where bottlenecks emerge, and whether it recovers gracefully once the load drops.
• Spike test: Introduces a sudden, massive jump in traffic, simulating a viral social media post or a flash sale.
• Soak test: Sustains a representative load over an extended period to catch issues like memory leaks, gradual performance degradation, and database connection pool exhaustion.
• Breakpoint test: Gradually increases traffic until the system breaches defined performance thresholds or can no longer serve requests reliably, identifying its maximum sustainable capacity under the tested conditions.

Tools like k6 support these patterns through configurable stages, scenarios, and executors. Locust supports simpler load profiles through user counts and spawn rates, while its custom LoadTestShape class lets you script more complex patterns, such as sudden spikes, gradual ramps, and breakpoint tests.

Metrics That Actually Matter

When evaluating test results, averages alone can be misleading. If nine requests take 10ms and one takes 910ms, the average is 100ms, hiding the fact that one request was significantly slower than the others.

Always examine percentiles, particularly p95 and p99, to understand tail latency. The p95 tells you that 95% of measured requests completed within that response time, while the remaining 5% took longer. Similarly, p99 identifies the response time within which 99% of requests completed.

Throughput (Requests Per Second) and Error Rate are your other two important metrics. A high RPS means little if the error rate climbs alongside it or response times become unacceptable. Also monitor resource utilization, such as CPU, memory, database connections, and connection pool usage, to identify potential bottlenecks.

Don't treat every isolated latency spike as proof of a systemic problem, but don't automatically dismiss it either. Look for recurring patterns, investigate the underlying cause, and determine whether the spikes violate your performance targets.

Locust displays live RPS, failure statistics, and response-time percentiles in its web UI. k6 provides an aggregated summary in the terminal after execution, including averages, medians, and configured percentiles. You can explicitly configure p99 when it isn't included in the default summary.

This keeps your original structure while making the advice more technically precise.

k6 vs Locust: Picking a Tool

There is no definitive winner here; they solve the problem differently.

k6 has an engine written in Go but uses JavaScript for its test scripts. It is optimized for relatively low resource consumption, supports native pass/fail thresholds, and integrates cleanly into CI pipelines. You can define performance requirements directly in the test configuration, such as keeping p95 response time below 500ms or the error rate below 1%. If a threshold fails, k6 returns a non-zero exit code.

Locust uses scripts written in plain Python. You define simulated user behavior through Python classes and functions rather than relying on a separate configuration language. Its standard HttpUser provides an HTTP client built around requests.Session, and it also offers FastHttpUser for more efficient HTTP load generation. Locust comes with a built-in web UI, supports headless execution, and provides distributed testing through its master/worker architecture.

One difference is how they handle performance thresholds. k6 provides declarative threshold assertions out of the box, whereas Locust requires custom Python logic to enforce specific performance criteria and return appropriate exit codes. Both tools can run in CI pipelines; k6 simply makes declarative performance gating more straightforward.

As a rule of thumb: lean toward Locust when your team works heavily with Python, wants to reuse existing Python libraries, or needs highly customizable user behavior. Lean toward k6 when you want resource-conscious load generation and straightforward, declarative CI performance gates. Complex user flows alone don't automatically make Locust the better choice; k6 can also model sophisticated scenarios using JavaScript.

Learning the underlying performance-testing concepts matters far more than the syntax of either tool.

Minimal Walkthroughs

Here is a standard k6 load test. It ramps up to 50 virtual users, holds them there, and then ramps down. It also enforces that 95% of requests finish under 500ms, with an additional check for failed requests.

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '30s', target: 50 }, // Ramp up
    { duration: '1m', target: 50 },  // Hold
    { duration: '30s', target: 0 },  // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'],
    http_req_failed: ['rate<0.01'],
    checks: ['rate>0.99'],
  },
};

export default function () {
  const res = http.get('https://api.example.com/health');

  check(res, {
    'health endpoint returns 200': (r) => r.status === 200,
  });

  sleep(Math.random() * 2 + 1);
}

Here is the equivalent in Locust. We define a user class, set a wait time between tasks, and define what the user actually does.

rom locust import HttpUser, task, between

class APIUser(HttpUser):
    host = "https://api.example.com"
    wait_time = between(1, 3)

    @task
    def check_health(self):
        self.client.get("/health")

To run this Locust test with the same shape (ramping to 50 users at 5 per second for 2 minutes) without the UI, you would run: locust -f locustfile.py --headless -u 50 -r 5 -t 2m.

Setting Thresholds from Real Data, Not Guesses

Arbitrary goals like "everything must be under 500ms" are meaningless without a baseline and defined performance requirements. Use production OpenTelemetry metrics to understand your actual latency distribution, traffic patterns, and error rates, then define thresholds based on your application's SLOs.

For example, if your production p95 latency is around 300ms, a 350ms test threshold might be a reasonable initial guardrail. However, don't treat the additional 50ms as a universal allowance for test-environment differences. Validate your threshold against a representative workload and account for differences between production and test environments.

k6 handles this natively through the thresholds block in the options object. For Locust, you can use the quitting event to evaluate the collected statistics and set a non-zero process exit code when performance requirements are breached.

from locust import events

@events.quitting.add_listener
def assert_thresholds(environment, **kwargs):
    stats = environment.stats.total

    if stats.num_requests == 0:
        print("No requests were recorded")
        environment.process_exit_code = 1
        return

    failed = False

    if stats.fail_ratio > 0.01:
        print("Error rate exceeded 1%")
        failed = True

    p95 = stats.get_response_time_percentile(0.95)

    if p95 is not None and p95 > 500:
        print("p95 exceeded 500ms")
        failed = True

    if failed:
        environment.process_exit_code = 1

This allows Locust to enforce custom performance criteria in CI without abruptly terminating the process using sys.exit().

CI/CD Integration

Be practical about what runs when. Once the application is running in a test environment, a short smoke test (e.g., a few virtual users for around 10 seconds) should run on every PR to verify that the application responds and critical routes work. Heavier load tests should run nightly or as part of a release validation process.

Here is a minimal GitHub Actions workflow using k6 for PR smoke testing:

name: k6 Smoke Test

on:
  pull_request:
    branches: [main]

jobs:
  smoke-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: grafana/setup-k6-action@v1
      - name: Run smoke test
        run: k6 run scripts/smoke.js

Define performance thresholds inside smoke.js. When a threshold is breached, k6 returns a non-zero exit code, automatically failing the CI job. Keep heavier load tests in a separate workflow triggered by a nightly schedule or manual dispatch.

Locust can run headlessly in CI as well. Install it, invoke it using --headless, and use environment.process_exit_code = 1 inside its quitting event hook when custom performance thresholds are breached.

A practical consideration: don't use standard hosted CI runners as load generators for heavy stress or breakpoint tests. Their CPU and network resources are limited, and if the load generator becomes saturated, your results may reflect its limitations rather than the application. For heavier tests, use dedicated load-generation instances, a cloud load-testing service, or Locust's distributed mode with dedicated workers.

Connecting Performance Tests with Observability

A load test tells you that a system is slow, but distributed tracing helps you understand where the time is being spent.

For request-level correlation, propagate standard trace context headers such as W3C traceparent from your k6 or Locust tests. When a request takes two seconds, its trace ID can help you locate the corresponding trace and investigate whether the delay came from application code, a Redis cache miss, or a slow database query. This requires the application and downstream services to be instrumented and to propagate trace context correctly. You can also use a custom correlation header, provided your application records it in its logs or traces.

k6 can stream real-time metrics to Grafana Cloud k6 and Prometheus-compatible remote-write endpoints (its Prometheus remote-write output is currently experimental). Locust also provides optional OpenTelemetry integration, allowing it to export traces, metrics, and logs through OTLP. Correlating request latency and throughput with application CPU, memory, database connections, and other infrastructure metrics helps identify the actual bottleneck.

Test Data and Environment Realism

Testing against an empty or unrealistically small database can make queries appear much faster than they would under production conditions. Similarly, testing on an undersized staging environment and extrapolating those numbers to production can produce misleading results. Dataset size, data distribution, indexing, and hardware all influence performance.

Cache states matter too. A test running against a warm cache can produce significantly different latency percentiles from one hitting a cold cache. Ideally, test both scenarios or reproduce the cache-hit and cache-miss distribution expected in production.

User behavior must also be realistic. Hammering a single endpoint doesn't necessarily represent actual application traffic; whether it is a load or stress test depends on the traffic intensity. Locust makes it easy to model different user behaviors using task weights:

from locust import HttpUser, task, between

class APIUser(HttpUser):
    wait_time = between(1, 3)

    @task(8)  # 80% probability of task selection
    def read_data(self):
        self.client.get("/data")

    @task(2)  # 20% probability of task selection
    def write_data(self):
        self.client.post("/data", json={"payload": "test"})

These weights control how frequently tasks are selected, not an exact request-volume ratio. For this example, where each task sends one request, the resulting request mix should approximately follow the 80:20 ratio over a sufficiently long test. Use unique test data and cleanup strategies for write operations to prevent repeated requests from distorting the dataset.

Performance Testing for AI and Streaming Endpoints

If you're building multimodal applications or RAG systems, traditional request-duration metrics alone don't tell the whole story. AI endpoints often use Server-Sent Events (SSE) or other streaming mechanisms. For interactive applications, Time to First Token (TTFT) is particularly important because it measures how long users wait before receiving the first generated content.

However, TTFT isn't the only metric worth tracking. Also measure inter-token latency, token generation throughput, and total response completion time to understand the complete streaming experience.

Load testing paid LLM provider APIs without strict budget limits or rate controls can quickly become expensive and trigger provider rate limits. Consider using a mocked gateway for high-volume application testing, then run smaller, controlled tests against the actual provider. Remember that mocked tests won't measure real provider latency or generation throughput.

Locust is convenient here because its Python-based HTTP client supports streamed responses through stream=True, iter_lines(), and iter_content(). You can measure the time from request initiation to the first non-empty generated-content chunk and report it as a custom Locust metric through its request event. Just remember that a received chunk isn't necessarily a single token; some chunks contain multiple tokens or metadata.

Common Mistakes

• Running the generator on the application server: Avoid this for representative performance benchmarks because both processes compete for CPU, memory, and network resources, potentially distorting the results.
• No baseline: Running a test once and trusting the numbers without historical results or comparable test conditions.
• Staring at latency, ignoring errors: A 10ms response time is meaningless if the request returns a 500 Internal Server Error.
• Accidentally load testing production: Always double-check target URLs, environment variables, and credentials. Production load tests should only be conducted when explicitly authorized and carefully controlled.
• Under-provisioning the load generator: Running a massive test with insufficient generator resources can make the load generator itself the bottleneck instead of the application. Monitor generator CPU, memory, and network utilization, and distribute the load across dedicated workers when necessary.

Closing Checklist

Before you declare a system "performance tested," answer these five questions:

1. Are my target URLs pointing to an isolated, production-representative environment with realistic infrastructure and configuration?
2. Are my performance thresholds based on actual production telemetry, historical baselines, and defined SLOs rather than arbitrary guesses?
3. Am I evaluating p95/p99 latency, throughput, and error rates together?
4. Does my test data, cache state, and traffic mix accurately simulate realistic user journeys?
5. Do I have observability active across my application and infrastructure, and am I monitoring the load generator to ensure it isn't the bottleneck?