10

Just for clarity, the stress test I have written steadily increases the load on the system until it reaches a breaking point. It theoretically runs indefinitely, but as the system resources are finite, it is expected to fail after some point of time. I have an expected load for the system, but this is tested separately in a load test. The purpose of this stress test is to find out how much load I can put on the system before I need to implement scaling.

I'm in the process of writing a stress test for a system, and I'm wondering whether it makes sense to have pass/fail criteria. By nature of the test, the load steadily increases until it hits a breaking point (i.e. it fails). I obviously don't know what this breaking point is beforehand, and therefore no expectation of the load the system can handle (in theory anyway).

Now I do have other performance tests to test the system under an expected load etc., which I can easily set pass/fail criteria for, and I could use these criteria as a basis for my stress test. In other words, I could set a minimum baseline for my stress test to reach, but I'm not sure if this is the right thing to do (is this 'duplicating' my other test?).

I'm hoping someone with more experience in performance testing can help me out here. What pass/fail criteria have others used when stress testing (if any)?

Alex
  • 211

3 Answers3

10

In a stress test your job is not to define the stress the subject should be able to take. It's to measure the stress it takes before it fails.

You can use the performance criteria to define what a stress failure is. But the result of a stress test isn't pass/fail. It's "failed after 90 hours under 100% utilization with ventilation 50% compromised".

candied_orange
  • 119,268
1

It depends on the requirements, if your requirements specifies that the expected result for the app performance it's X and actually you got Y, so it's a Fail.
If you don't have requirements defined, so you might stress your system and collect boundary data and then figure out and doc these limits.

Thiago F. Peçanha
  • 11
0

You can easily update your primary stress test to also support a QA pass/fail verification, something like "able to reach/sustain X load without breaking". Ideally with X being configurable (for different release branches, for example).

The result would be fail if the system breaks before the load reaches X and pass if it doesn't break. You'd just have to stop increasing the load once it reaches the X value in a "sustain" scenario.

IMHO an automated test like this can be very useful in the context of CI/CD, especially on production branches.

Dan Cornilescu
  • 887