Performance Testing with k6 and Locust: APIs & LLM Agents

Performance problems rarely announce themselves. A release can pass every functional test and still make the product feel slow. My approach is to put response-time budgets inside the functional suites, so a regression is caught on the same commit that caused it, and to use dedicated load tests to answer capacity questions.

Response times in API tests

Every call in my backend circuit reports its status and elapsed time, so a slow endpoint is visible in the log of every run:

r = s.get(url, headers=headers)
if r.status_code == 200:
    print(str(r.request) + "  " + str(r.status_code) + " " + r.reason + " in " + str(r.elapsed))

Printing a number is observation. A budget turns it into a test. The same measurement becomes a gate with one extra line:

assert r.status_code == 200
assert r.elapsed.total_seconds() < 2.0, f"over the response-time budget: {r.elapsed}"

Latency budgets for LLM agents

In my LLM evaluation framework, latency is a first-class assertion. The agent API measures its own processing time and returns it with every answer, so the evaluators can store it per case:

started = time.perf_counter()
result = graph.invoke({"messages": incoming, "blocked": False})
latency_ms = round((time.perf_counter() - started) * 1000, 2)

# ...returned to the evaluators in the response body
"x_agent": {"variant": config.variant, "tools_called": tools_called, "latency_ms": latency_ms}

Promptfoo then enforces a budget on every test through its default assertions, so no new test can forget it:

defaultTest:
  assert:
    - type: latency
      threshold: 2000

With a real model, latency and token usage become release metrics. They are on my list of next steps for the project, together with repeated runs and tolerance bands, because real models are noisy.

Load testing with k6 and Locust

Budgets in functional tests catch regressions of a single request. They say nothing about behaviour under concurrency. For that I use k6 and Locust, and the agent’s own API from the LLM evaluation framework is the system under test. That keeps performance in the same language as quality: the same questions as the golden dataset and the same 2000 ms latency budget as the Promptfoo assertions.

export const options = {
  scenarios: {
    smoke: { executor: "shared-iterations", vus: 1, iterations: 3, maxDuration: "1m" },
  },
  thresholds: {
    http_req_failed: ["rate<0.01"],              // fewer than 1% errors
    http_req_duration: ["p(95)<2000"],          // same budget as Promptfoo
    agent_latency_ms: ["p(95)<2000"],           // server-side time from x_agent.latency_ms
    checks: ["rate==1"],                        // every correctness check must pass
  },
};

check(res, {
  "status is 200": (r) => r.status === 200,
  [`${scenario.name}: answer contains "${scenario.mustInclude}"`]: () =>
    answer.toLowerCase().includes(scenario.mustInclude.toLowerCase()),
  [`${scenario.name}: ${scenario.expectedTool} was called`]: () =>
    (agent.tools_called || []).some((t) => t.name === scenario.expectedTool),
});

The k6 script is deliberately small: one virtual user and three iterations. It is a smoke-level gate, not a stress test, so it can run on every change without putting pressure on anything. Thresholds are on percentiles instead of averages, because an average hides the slow tail users feel. When a threshold fails, k6 exits with a non-zero code and the build turns red. It also tracks the agent’s own processing time as a separate metric, so a slow network is not mistaken for a slow agent.

The same scenario is also written in Locust, where user behaviour is plain Python: a user class with weighted tasks, each response validated with catch_response, and a quit hook that turns the run into a pass or fail gate.

class SupportAgentUser(HttpUser):
    wait_time = constant(1)   # one request per second per user, nothing aggressive

    def ask(self, name, prompt, expected_tool, must_include):
        with self.client.post(
            "/v1/chat/completions",
            json={"messages": [{"role": "user", "content": prompt}]},
            name=name,                # group statistics by scenario, not by URL
            catch_response=True,
        ) as response:
            body = response.json()
            answer = body["choices"][0]["message"]["content"]
            tools = [t["name"] for t in body.get("x_agent", {}).get("tools_called", [])]
            if must_include.lower() not in answer.lower():
                response.failure(f"answer does not contain {must_include!r}")
            elif expected_tool not in tools:
                response.failure(f"{expected_tool} was not called (got {tools})")
            else:
                response.success()

    @task(3)
    def policy_question(self):
        self.ask("policy", "What is your return policy?", "search_policy", "30 days")

    @task(2)
    def order_status(self):
        self.ask("order", "Hi, where is ORD-1001? My email is alice@example.com", "lookup_order", "shipped")


@events.quitting.add_listener
def enforce_thresholds(environment, **_kwargs):
    total = environment.stats.total
    if total.fail_ratio > MAX_FAIL_RATIO or total.get_response_time_percentile(0.95) > LATENCY_BUDGET_MS:
        environment.process_exit_code = 1   # the CI job fails

Question	k6	Locust
Language	JavaScript, one fast binary	Python, so test logic can be shared with pytest suites
Where it shines	Thresholds on percentiles and checks built in, scenarios and executors, a clean CLI for CI	Complex user behaviour as code, weighted tasks, a ready-made HTML report
Gate in CI	Non-zero exit code when a threshold fails	Exit code set from a quit hook

Both scripts and a workflow that runs them every day are in the Performance folder on GitHub. For a real service, I would add ramp-up, hold and ramp-down stages, a separate stress profile on a dedicated environment, and thresholds that come from an agreed service level instead of a guess.

What I measure and which test answers which question

Test type	Question it answers
Smoke	Does the system answer at all under minimal load? Cheap enough to run on every pull request.
Load	Does it meet its targets at the expected traffic?
Stress	Where does it break, and does it fail gracefully?
Spike	Does it survive a sudden burst and recover?
Soak	Does it leak memory or degrade over hours?

The metrics that matter: latency percentiles (p50, p95, p99), error rate, throughput and resource saturation. I look at the percentiles and the errors together, because a fast endpoint that fails is not fast.

In the pipeline

Budgets run on every commit as part of the functional suites. The k6 and Locust smoke checks run in their own GitHub Actions workflow on every change to the performance scripts or the agent, and every day at 13:00 UTC, one hour after the LLM evaluation run. It writes a summary table to the run page and uploads the reports as artifacts. A heavier load profile belongs on a nightly schedule against a dedicated environment. The rest of the tests use the same CI/CD pipelines.