Writing TCKs with JUnit Jupiter's Dynamic Tests

March 16, 2020 6 minutes read

java • junit • testing

When you develop a library that provides some abstraction, or write a spec, it is a good practice to include a TCK (Technology Compatibility Kit) that the implementers can use to verify the behavior of their implementation and how well it matches the spec.

You may find various TCKs out there, e.g. the Reactive Streams TCK.

I just need something to test

Let’s start with a simple example first. Consider we have the following contract defined:

interface Action<I, O> {

	/**
	 * Applies transformation to the provided stream.
	 * The result MUST NOT emit null.
	 * If error happens, it MUST be of {@link ActionException} type.
	 *
	 * @param input source {@link Stream}
	 * @return transformed {@link Stream}
	 * @throws ActionException on error
	 */
	Stream<O> perform(Stream<I> input) throws ActionException;
}

It looks like we have a framework that allows us to implement some actions on java.util.stream.Stream.

Let’s write such action:

class MappingAction<I, O> implements Action<I, O> {

	final Function<I, O> mapper;

	public MappingAction(Function<I, O> mapper) {
		this.mapper = mapper;
	}

	@Override
	public Stream<O> perform(Stream<I> input) {
		return input.map(it -> {
			O result = mapper.apply(it);
			if (result == null) {
				throw new ActionException(new NullPointerException("result"));
			}
			return result;
		});
	}
}

As you can see, we’ve attempted to implement it correctly after reading the javadoc. But documentation is not always up to date and may not have all the edge cases mentioned.

Let’s write a test for it!

class MappingActionTests {

	@Test
	void shouldMap() {
		var source = Stream.of("Hello", "World");

		var action = new MappingAction<>(it -> it.toUpperCase());

		var result = action.perform(source);

		assertThat(result).containsExactly("HELLO", "WORLD");
	}

Yay! It works! Here we test the concrete behavior of this Action.

But do we actually test that our Action conforms to the contract?
And, a better question would be: do we really want to write (or… repeat, in fact) tests for checking that?

This is where a TCK would be helpful!

Writing a TCK with JUnit

Some prefer to write TCKs with TestNG due to how flexible it is. While I have nothing against TestNG, I prefer to use JUnit.

Unfortunatelly, there aren’t many resources on how to write “abstact” test suites with JUnit.
(btw, this statement is always true. There can’t be too many resources about testing! =P)

⚠️ WARNING!
I will describe how I prefer to write such tests in JUnit Jupiter.
This is not the only way to do so (e.g. you can extend from a common class or use helpers).
You have been warned!

Since the TCK tests should be reusable, I will use JUnit’s support for interfaces and default methods.

We will test the tricky part first - exception handling:

public interface ActionExceptionsTest {

	Stream<Object> createFaultyInput();

	Action<Object, Object> createFailingAction();

	@Test
	default void shouldAlwaysWrapExceptions() {
		assertThrows(ActionException.class, () -> {
			Stream<Object> input = createFaultyInput();

			Action<Object, Object> action = createFailingAction();

            // Trigger it
			action.perform(input).limit(1).toArray();
		});
	}
}

As you can see, we need to ask the user to provide something that we can’t assume about the implementation:

faulty input that should trigger the error
an action that is supposed to fail

How can we run this test? Easy! Just implement it in MappingActionTests:

class MappingActionTests implements ActionExceptionsTest {

	@Override
	public Stream<Object> createFaultyInput() {
		return Stream.of(null);
	}

	@Override
	public Action<Object, Object> createFailingAction() {
		return new MappingAction<>(it -> it.toString());
	}

	@Test
	void shouldMap() {
        // ...
    }

But what if we need more parameters (e.g. expected output)? Let’s add a concept of “Scenario”:

public class Scenario {

    String name;

	Action<Object, Object> action;

	Stream<Object> input;

	Consumer<ListAssert<Object>> expect;

So that we can change out code to:

class MappingActionTests implements ActionExceptionsTest {

	@Override
	public Scenario createFailingScenario() {
		return Scenario.builder("returns null")
				.input(Stream.of("foo"))
				.action(new MappingAction<>(it -> null))
				.build();
	}

	@Test
	void shouldMap() {
        // ...
    }

It is passing again (since we handle null after applying the mapper). But we only check one scenario when an Action may throw, what about others?

Parameterize all the things!

We could create more tests, but they will follow the same pattern and it would be nice to parameterize them.

JUnit Jupiter does support Parameterized Tests, but, if we want the parameters factory to be a method, it must be static. It does not really work for us, since we let the end users implement them.

Instead, we will be using another feature of JUnit Jupiter - Dynamic Tests.

Before we change our ActionExceptionsTest test, I suggest introducing a base interface for our TCK tests:

public interface ActionTestSupport {

	Stream<Scenario> successfulScenarios();

	Stream<Scenario> failingScenarios();

	default Stream<DynamicTest> toDynamicTests(
        Stream<Scenario> scenarios,
        ThrowingConsumer<Scenario> executable
    ) {
		return scenarios.map(scenario -> {
			return dynamicTest(scenario.toString(), executable);
		});
	}
}

Note that we define two sets of scenarious that we can reuse later.

Now we can use it as follows:

public interface ActionExceptionsTest extends ActionTestSupport {

	@TestFactory
	default Stream<DynamicTest> shouldAlwaysWrapExceptions() {
		return toDynamicTests(failingScenarios(), scenario -> {
			assertThrows(ActionException.class, () -> {
				scenario.getAction().perform(scenario.getInput()).limit(1).toArray();
			});
		});
	}
}

Now we can implement more failing scenarios:

class MappingActionTests implements ActionTests {

	@Override
	public Stream<Scenario> failingScenarios() {
		return Stream.of(
				Scenario.builder("returns null")
				        .action(new MappingAction<>(it -> null))
				        .input(Stream.of("foo"))
				        .build(),

				Scenario.builder("receives null")
				        .action(new MappingAction<>(it -> it.toString()))
				        .input(Stream.of((Object) null))
				        .build(),

				Scenario.builder("throws")
				        .action(new MappingAction<>(it -> {
					        throw new RuntimeException("Ooops!");
				        }))
				        .input(Stream.of("foo"))
				        .build()
		);
	}

And guess what? Our implementation is only 33% correct!

@Override
public Stream<O> perform(Stream<I> input) {
    return input.map(it -> {
        O result = mapper.apply(it);
        if (result == null) {
            throw new ActionException(new NullPointerException("result"));
        }
        return result;
    });
}

We did wrap the NPE with ActionException but forgot to wrap mapper.apply which is a user-provided function that we cannot control.

Improving the DX

If you run the tests in the IDE, you may notice that the tests have nice names:

.. but it is impossible to go to the scenario definition from a test run.

No surprise, since we generate the test cases dynamically and IDE or JUnit have no idea about it.

To workaround that, we will use the following trick:

public class Scenario {

	public static ScenarioBuilder builder(String name) {
		StackTraceElement[] stackTrace = new Exception().getStackTrace();
		return new ScenarioBuilder().frame(stackTrace[1]).name(name);
	}

    // ...
}

public interface ActionTestSupport {

	default Stream<DynamicTest> toDynamicTests(
        Stream<Scenario> scenarios,
        ThrowingConsumer<Scenario> executable
    ) {
		return scenarios.map(scenario -> {
			return dynamicTest(scenario.toString(), () -> {
				System.out.println("Scenario defined at " + scenario.getFrame());
				executable.accept(scenario);
			});
		});
	}

Now every test will have a link to the scenario definition and some IDEs will make it clickable:

Going forward

It may not be the only approach to write TCKs with JUnit Jupiter (if you think that you have a better one, please do share it in the comments!), but I like how it works and how it scales.

You can easily add more cases:

public interface ParallelStreamsTest extends ActionTestSupport {

	@TestFactory
	default Stream<DynamicTest> shouldWorkWithParallelStreams() {
		return toDynamicTests(successfulScenarios(), scenario -> {
			Consumer<ListAssert<Object>> expect = scenario.getExpect();

			Assumptions.assumeTrue(expect != null, "expect is provided");

			Stream<Object> input = scenario.getInput().parallel();

			Stream<Object> result = scenario.getAction().perform(input);

			if (scenario.getLimit() != null) {
				result = result.limit(scenario.getLimit());
			}

			expect.accept(assertThat(result));
		});
	}
}

and you can “group” them too:

public interface ActionTests extends ActionExceptionsTest, ParallelStreamsTest {

}

class MappingActionTests implements ActionTests {

	@Override
	public Stream<Scenario> successfulScenarios() {
		return Stream.of(
            Scenario.builder("two items")
                    .action(new MappingAction<>(it -> it.toString().toUpperCase()))
                    .input(Stream.of("hello", "world"))
                    .expect(it -> it.containsExactlyInAnyOrder("HELLO", "WORLD"))
                    .build(),

            Scenario.builder("empty")
                    .action(new MappingAction<>(it -> it.toString().toUpperCase()))
                    .input(Stream.of())
                    .expect(it -> it.isEmpty())
                    .build(),

            Scenario.builder("infinite stream")
                    .action(new MappingAction<>(it -> it.toString().toUpperCase()))
                    .input(Stream.generate(() -> "hello"))
                    .limit(1)
                    .expect(it -> it.startsWith("HELLO"))
                    .build()
		);
	}

	@Override
	public Stream<Scenario> failingScenarios() {
        // ...
	}

    // ...
}