Shell Tricks for Repeatedly Running Flaky Tests

Example Test Command

Below is a typical test command for running a single test in a Django project. The specific command doesn’t really matter but we’ll reference this in some examples below.

$ pytest --reuse-db example/tests.py::ExampleTests::test_thing

This command uses the pytest test runner. The --reuse-db option from the pytest-django plugin speeds up repeat runs.

Rerun Until Failure

Useful for tests that fail rarely. Use a while loop on your test command to repeat it until failure.

while <test>; do :; done

<test> needs replacing with your test command. : is a special command that always succeeds instantly, so the body of this loop is effectively empty.

You don’t need to replace <test> if you use this form:

while !pytest; do :; done

!pytest will trigger history expansion to include the last command you ran that included the word pytest (ignoring use of the !pytest expansion). This is useful when you’ve already run the test command once, and then want to loop on it. On zsh you can press TAB to expand the line to substitute the command, so you can see the full line that will run before you run it:

while pytest --reuse-db example/tests.py::ExampleTests::test_thing; do :; done

Noice.

Rerun Until Success

Useful for tests that suceed rarely. Use a negated while loop to run a command until it succeeds:

while ! <test>; do :; done

Again you can use !pytest for history expansion:

while ! !pytest; do :; done

Rerun N Times

This is useful for repeating a test that might be flaky. Use a for-loop with seq to count up however many runs you want:

for i in $(seq 10); do <test>; done

Again with history expansion:

for i in $(seq 10); do !pytest; done

Rerun Until Failure with Delay

Sometimes you want a little break between commands, perhaps to avoid overloading some resource. You can do this by modifying the loop body to use sleep, which takes a number of seconds:

while <test>; do sleep 1; done

Again with history expansion:

while !pytest; do sleep 1; done

Fin

May your flakes be easily fixed,

—Adam