Coordinated omission: why a load test can wrongly say "passed"
A load test can show p95 under target, zero errors and every threshold green while the system still cannot take the load you asked for. The most common reason is coordinated omission: when the system slows down, the load generator slows down with it, and the requests that should have gone out at the worst moments are never sent, so they are never measured. This guide shows how it happens, what it looks like in a real run and how to set up a test against it.
Closed model: a slow system lowers the load by itself
In a test with a fixed number of virtual users (VUs), each VU starts a new iteration as soon as it finishes one. While the system is fast many iterations start, while it is slow few do. So the slower the target gets, the less load it receives. Real users don't behave like that: new visitors don't stop arriving because your site got slow.
A made-up example. 100 VUs, an iteration normally takes 100 ms: 1000 iterations per second. The system slows to 1 second for 10 seconds. During those 10 seconds the test sends only 100 iterations per second; the roughly 9000 iterations normal traffic would have brought are never sent. The 1000 slow requests that were measured sit against tens of thousands of fast ones from the rest of the run, and p95 may still look under target. The wait real users would have had in those 10 seconds never reaches the report.
A closed model is the right tool for "N users at the same time"; for "does it take N requests per second" it misleads. Which question each test answers: types of load tests.
Open model: iterations start on a timetable
In an open model (in Spitfire Constant arrival rate and Ramping arrival rate, constant-arrival-rate and ramping-arrival-rate) iterations start at the set rate however slow the target is. Slow answers get measured and the load does not drop. Each iteration still needs a free VU to run it, though. As the target slows down more VUs stay busy waiting; once Max VUs is reached, the iterations that cannot start are dropped (dropped_iterations). They are not queued, they are counted.
The VUs needed are about iterations per second × one iteration's length in seconds (Little's law). 200 iterations per second at 1.5 seconds an iteration need about 300 VUs, more when the target slows down. Keep Max VUs at two to three times that.
A real run: the p95 threshold passed, the system was saturated
In an open model the VU ceiling can hide latency too. The run in the sample report shows it: ramping arrival rate, at most 100 VUs, 2 requests an iteration, thresholds p95 < 800 ms and error rate < 2%.
- At 50, 100 and 150 requests per second, p95 was a few tens of milliseconds.
- At 5× the target was 125 iterations (250 requests) per second; climbing there, the service saturated at about 228 requests per second. p95 432 ms, p99 441 ms, errors 0%.
- 100 VUs / ~114 iterations per second ≈ 0.88 seconds an iteration ≈ ~440 ms a request. The latency reflects not the system's real limit but how many VUs can wait at once: with 1000 VUs the same service would have shown longer queues and a far higher p95.
Looking at p95 and error rate alone this run would have "passed": 432 ms is far under 800 ms and there were no errors. What read the run correctly was the third threshold: once more than 200 iterations could not start, the run stopped in its 8th minute and the report said why.
How to guard the test against it
- Use an open model for capacity and latency targets. A target like "500 requests per second at p95 300 ms" is tested with an arrival rate.
- Put dropped iterations in a threshold.
count<1ondropped_iterationsmakes sure the load was the load you asked for. The run page shows them as "N dropped". - Compare the target rate with the achieved rate. If requests per second stay below the target, the latency numbers tell an incomplete story.
- Be generous with Max VUs. When you see dropped iterations, raise Max VUs first; if they keep dropping while latency climbs, the system is near its limit. To put a number on that limit: breakpoint testing.
"thresholds": [
{ "metric": "req_duration", "expr": "p(95)<800" },
{ "metric": "req_failed", "expr": "rate<0.02" },
{ "metric": "dropped_iterations", "expr": "count<1" }
]Read the three thresholds together: p95 speaks for latency, the error rate for correctness, and dropped iterations confirm the latency was measured at the load you asked for. How to read percentiles: what are p95 and p99?
Spitfire installs on Docker or Kubernetes with one command; every testing feature and protocol is open in the free edition.