What is the point of repeatedly executing the same test?

Question

I have recently learned about the not-well-known and not-widely-used annotation @RepeatedTest that, as the name implies, repeats the very same test n-times. Baeldung provides a short guide to this feature, however; I have found nowhere to explain why this annotation exists in the first place.

I initially assumed the reason was to test a feature that involves some randomization. However, in that case, I would personally mock the randomness by extracting the hardcoded generation as a dependency and testing the boundaries and fixed values only to keep the tests predictable.

Of course, unless I write my own random number generator for which solely the test repeating feature was unlikely implemented at all.

I'm curious as to why the identical test should be run several times from the unit and integration testing perspective. Could you provide me with a few examples of when this feature is required?

Answer 1

This is to allow you to detect the difference between passed, failed and flaky tests in a single test run.

A flaky test is one where it intermittently fails which can be due to a number of reason sometimes outside your control. Rather than run the build again and shrug your shoulders when it passes this time, you run tests multiple times and only count them as passing if there are no failures.

For example you might have "click a button on the webpage and check the div appears" test. but due to load on the box and load order of elements on the page there is an unknown delay before the div appears. In your test you have to choose a timeout.

Or the order in which the tests runs affects the test result. You cant find the exact cause of this rare failure but need to know which tests are affected by it

Or you may simply have an intermittent bug you are ignoring but want to avoid having to rerun your builds multiple times to get a pass.

Answer 2

@Ewan posted a nice answer, but let me add this feature isn't only useful for potentially unstable or intermittently failing tests.

By using the RepetitionInfo argument, this can be used as a simple way of making parametrized tests, where the increasing loop index is the parameter which controls it. The loop index might be used directly as an input to the method under test, or indirectly by picking some input values from an array or other source, or calculating different input values. It may be possible to achieve similar functionality through JUnit's @ParameterizedTest feature, but I guess that wouldn't necessarily become simpler.

Another use case are automated tests with potential side effects (for example, a test which runs against some database, maybe a small local file-based database), where one wants to validate the idempotency of a certain function which makes modifications to the database. In the @BeforeAll method, the test database is initialized, then the test is repeated two or three times and validates that the outcome is still the same.

A third use case I can think of are performance tests or benchmarks. Though JUnit originally was most probably not intended to be used for this purpose, one can utilize it for this, since the total running time for all tests (and individual ones) is a standard result one gets from each run. Repeating the same test here several times might be used to measure the average running time of a certain process, for example.

Answer 3

I initially assumed the reason was to test a feature that involves some randomization.

Randomization or other non-determinism is indeed a very common reason to run a unit test multiple times.

However, in that case, I would personally mock the randomness by extracting the hardcoded generation as a dependency

This is indeed a good idea: if you don't, failures may be hard to reproduce. But it is often not possible. There are many circumstances where you have no control over the source of nondeterminism.

A very common case is concurrent programming, where the nondeterminism comes from the thread scheduler. If the test is multithreaded and has a race condition, running it multiple times increases the chance that the race will happen.

You may also want to run multiple instances of the same test concurrently.

It can also happen that the nondeterminism comes from a library or an external program that's out of your control.