Have parameter to abort if test doesn't reach target iterations per second

[MattDodsonEnglish]

Feature Description

In the arrival rate executors, the test continues even if k6 can't reach the target iteration rate. If the duration property is long, the test will go on for a while, using resources to do something that that tester doesn't want to test for anyway.

This is especially true if the testing set up is automated. I can imagine a situation where a commit to the SUT causes the median iteration of the test to increase, which in term means that the preAllocated number of VUs is no longer sufficient.

Sorry if this has been discussed― I couldn't find anything about it. Or, maybe this just doesn't make sense from a testing standpoint. Anyway, at least this issue will create a chance to document the reason why it doesn't make sense.

Suggested Solution (optional)

Have an extra property called something like abortWhenTooFewVUS or abortOnInsufficientAllocation. Already k6 provides a warning, so I guess the only thing to do would be to send abort with a non-zero exit code if that property is set to true.

Already existing or connected issues / PRs (optional)

No response

[na--]

k6 already emits a dropped_iterations metric - a Counter that is incremented once for every iteration that k6 was supposed to make but couldn't. In arrival-rate executors, it is going to be emitted when there aren't any available initialized VUs, but k6 was supposed to make another iteration. That should allow a user to set an abortOnFail threshold to abort the test 🤔

Something like a scenario abortOnInsufficientAllocation boolean option will be much simpler for beginner users, but I'd argue it might be a bit too simple and inflexible. It would be difficult to gauge what actually is "insufficient allocation". In a scenario with thousands of VUs that make tens of millions of iterations, I'd argue that a few dozen dropped iterations (e.g. because of a temporary network or service slow-down) shouldn't abort it, even if k6 exhausted all of the initialized VUs for a few seconds. On the contrary, it probably provides valuable information and the test should continue running.

With a metric like dropped_iterations, the user can determine a specific abort threshold that makes sense for their use case. It will be a bit more complicated and it's also not super flexible at the moment, but that's just because the current k6 thresholds are somewhat limited.

The solution that makes the most sense to me is to improve their capabilities (#2379, #1136) and UX (#1313, #1441 or something like PromQL support), not to add extra options to arrival-rate scenarios 🤔

[MattDodsonEnglish]

You know, I knew that dropped_iterations was a metric, but I was thought of it as something that happened when the SUT starts failing and the generator queue becomes so long it starts skipping (is that how it works?). I never thought of it as something that also happens when the generator just doesn't have enough allocated VUs to send the correct load. It makes sense of course: just a blind spot from the mental model I approached the problem with. ^{1Though I just thought of a naming note:thinking:}

I see that it would be possible to set a counter to abortOnFail after some early amount of skipped iterations―probably in conjunction with delayAbortEval to make sure it's not too early. The only problem I see is that it requires some guesswork. I don't know how precise you could be. On the other hand, k6 already gives a warning if the preAllocatedVUs are all used before the target iteration per second is reached, so I was thinking it'd be very simple to implement an abortWhenInsufficient on that trigger. And, I'm guessing that a user under allocating VUs is a much more common case random network errors happening early in the run. But I see your point there.

Despite what I wrote, I'm okay with just closing this issue if it seems too finicky or simple to implement.

Footnotes

1. 🤔 ...but in the case of insufficient allocation, are the iterations really dropped (implying they were there to begin with), or were they never really reached? ↩

[na--]

something that happened when the SUT starts failing and the generator queue becomes so long it starts skipping (is that how it works?). I never thought of it as something that also happens when the generator just doesn't have enough allocated VUs to send the correct load.

Both of these things can be connected. When the remote service/website gets overloaded and starts responding more slowly, requests will take longer to execute, so k6 iterations will also take longer to run. When they take longer, even if we have a constant number of newly started iterations every second (i.e. constant-arrival-rate), more VUs would be needed to actually run the iterations until their end. So, the executor might exhaust whatever number of preAllocatedVUs were initialized.

Any time that an executor was supposed to execute an iteration but couldn't, it will emit a dropped_iterations metric sample. This can happen in a shared-iterations and per-vu-iterations scenarios that reach their maxDuration before all of their iterations were completed. And it can also happen in a constant-arrival-rate or ramping-arrival-rate scenario when there are no free VUs.

For arrival-rate executors, that will happen even if the user specified maxVUs > preAllocatedVUs! k6 will still emit a dropped_iterations metric sample for the iteration it was supposed to execute, but it will also start initializing a new VU mid-test, in the background. Also, because VU initialization can be costly in terms of CPU usage, k6 won't ever initialize more than 1 mid-test VU at a time for a given scenario, to hopefully avoid super-spikes of CPU usage.

For example, if the scenario specifies that k6 should start a new iteration every 1ms, and that VU init takes 5ms, if the scenario runs out of free VUs (and none free up from previous iterations), then k6 will emit ~5 dropped_iterations: 1 metric samples before a new VU is initialized and can be used.

Anyway, in summary, dropped_iterations tracks iterations that were supposed to be started according to the options the user specified, but for some reason (likely mis-cofniguration), k6 was unable to start them. And, btw, we also have some potential plans to add an interrupted_iterations metric to track iterations that were actually started, but were interrupted mid-way through their execution. This could happen due to script errors, test abort, VU ramp-down in ramping-vus, scenario end, etc. - see #877 (comment) for details.

The only problem I see is that it requires some guesswork. I don't know how precise you could be.

Yeah, this is a problem. Though the limitations of the thresholds are the biggest problem in my mind 🤔 If users could specify something like "abort the test if the number of dropped_iterations is more than 1% of the number of iterations over the last 60s", that would be super useful... But there is a long way to go before k6 thresholds are anywhere near that capable 😅 Still, we have been heavily investing in the groundwork for that over the last year, with significant refactoring of how the metrics and thresholds are handled, so 🤞

[MattDodsonEnglish]

For future readers who want to follow the flow of information, a lot the info from this last post made its way into the docs via: grafana/k6-docs#974