Yocto Project provides open source tools to help developers create custom Linux distributions, especially for embedded use. Hardware support is widely available from large chip vendors, and one could say that Yocto is currently the de-facto standard for creating embedded Linux ditributions. Yocto also provides a well-defined workflow for application development. However, despite its wide adoption and populatity, it seems to fall somewhat short in the are of unit testing.
Yocto provides mechanism to build a software-development-kit (SDK) that matches with the custom Linux distribution. The SDK contains the necessary compilers and tools to build for the target architecture, as well as all the necessary headers and libraries to cross-compile applications for the target platform. The SDK tooling and process are documented in the Yocto Project Application Development and the Extensible Software Development Kit (eSDK) Manual. In my experience, the SDK workflow works well and is a good way to separate platform and application development. In practice, this means that developers focusing on the applications do not need the Yocto development setup and can solely use the SDK package, which encapsulates the compilers, tools, headers and libraries needed.
Unit testing is an important aspect of modern software development and an integral part of developers’ daily workflow. Unit testing tools should be easy to use and allow for quick edit-build-test cycles to facilitate efficient development. There are many unit testing frameworks available, and choosing between the alternatives is not the focus of this post. Luckily, there are recipes available for many of them, making it easy to include them in the SDK package. However, things become a bit more complicated when selecting how to build and run those tests.
The following list summarizes the main alternatives for implementing unit testing in a Yocto-based application project:
- Run unit tests on target platform.
- Use the host environment for unit testing.
- Add native packages to the target SDK.
- Use a separate SDK for unit test builds.
Run unit tests on target platform
The first obvious solution is to use the target SDK as is and build the unit test binary for the target hardware. This is a straightforward approach that allows building the unit tests in the same way as the target application. It ensures the usage of the same versions for dependencies and tools as the software being tested, as both rely on the same SDK. However, the main drawback is that it does not allow running the tests directly on the host machine. Either real target hardware or an emulator such as QEMU needs to be used.
Having to dedicate real hardware (or use an emulator) for the unit tests complicates the edit-build-test cycle. It also complicates the integration into a CI pipeline for the same reason that hardware or emulator needs to be set up.
PROS:
- Same versions for tools and libraries as the target build.
- No additional setup needed since the target SDK is used
CONS:
- Tests cannot be run directly on host machine (real hardware or emulator needed)
Use host environment for unit tests
The next alternative is to use the existing host development environment without the Yocto SDK for the test build. In practice, this means installing the needed dependencies, tools and unit test framework to the host machine and building the project with those. This approach allows running the unit tests directly on the development machine and facilitates debugging using host tools.
At first glance, this approach seems promising. Unless the project uses some exotic tools, most dependencies can be either installed from the distribution package manager or built using existing project files. Therefore, setting up the host build is usually a straightforward task.
The problems of this approach lie in the dependency management. It would be beneficial to use exact same versions from the tools (such as compiler) and libraries as the target build being tested. However, managing this is not an easy tasks since there is no direct link between the versions available in the host distribution package manager and the versions used in Yocto. This approach can also lead to unexpected test failures if the host tools are updated, which is not ideal for reproducible builds.
While this approach is easy to setup and allows running tests directly on the host, it creates complicated dependency management issues. Some of these issues might be alleviated by using a unit test build container, but it still requires manual management of host dependencies.
PROS:
- Allows running unit tests on the host and using host debugging tools.
- Usually easy to set up.
CONS:
- Manual dependency management needed between target and host environments.
- Updates on the host machine may break tests.
- Differences between developer environments may cause test failures.
Add native packages to target SDK
It is possible to add native packages to the target SDK. This mechanism is usually used to bundle host tools to the SDK, separating them from the host tools. In other words, when you install the Yocto SDK, you will get known versions from the compiler and other tools, regardless of what is installed or available on the host machine. This is a good mechanism for bundling and sharing the development environment.
It is also possible to build other packages, such as generic libraries and unit testing frameworks, as native targets and have them bundled into the SDK. This is accomplished by building and installing nativesdk variants from the packages. For details, refer to the Customizing the Standard SDK section in the Yocto documentation.
With this approach, it is possible to add the native variants of the unit test framework and all dependencies needed by the application, bundling them in the target SDK. This way, the dependencies for the unit test build on host would use the exact same versions from libraries as the target build.
While this approach also seems promising, it has some issues. First, the SDK only has the environment-setup script for the target build, so managing the host build is quite manual and requires configuring include and linker paths to point to the SDK directories. The SDK also does not have the host compiler installed by default. When using the host compiler from the package manager, there can also be incompatibilities between the glibc version used in the Yocto build and the host. Although I believe this approach could be made to work, it is not very straightforward and easy.
PROS:
- Same versions for tools and libraries as target build.
- Host packages bundled to same SDK.
CONS:
- Host toolchain not included in target SDK by default.
- Host build needs to be configured manually (no host environment-setup script).
- Possible incompatibilities with the host if the host compiler is used.
Separate SDK for unit test build
Finally, there is an approach that is not perfect but seems to solve most of the challenges discussed above. It is possible to consider your host machine (probably an x86 PC) as a target and build a separate SDK for it. In this approach, you would set the target as MACHINE=genericx86-64 and use the bitbake -c populate_sdk command in the same way you would for the target. This will generate a separate SDK package that targets the PC environment.
You can then install this SDK normally and use the provided environment-setup script to build the unit tests. As long as you build this SDK from the same Yocto project that you are using for the target, you will automatically get the same compiler version as well as same versions of all the libraries.
There is one thing you might need to tweak in the build. To run the generated binaries using the Yocto SDK sysroot, you might want to set the rpath and dynamic-linker in the binary and set LD_LIBRARY_PATH environment variable when running the binary.
For example, in CMake you can set the RPATH as follows:
target_link_options(test-target PRIVATE
"LINKER:--rpath,${YOCTO_HOST_ROOT}/lib"
"LINKER:--dynamic-linker,${YOCTO_HOST_ROOT}/lib/ld-linux-x86-64.so.2"
)And to run the binary:
$> LD_LIBRARY_PATH=<path to SDK /usr/bin> ./test-targetThe unit tests are usually run directly in the developers’ machine or in the CI environment and are not shipped. These tweaks make it slightly easier to run the tests and prevent the host environment from interfering with them.
The main drawback of this approach is that it requires a full Yocto build to generate the x86 SDK. However, when using a CI to build the SDK, the extra build time is not a major issue. There is also some additional overhead in maintaining and shipping two separate SDKs. On the other hand, it solves the dependency issues outlined earlier and requires only minimal tweaks on the application development side to use. Therefore, this is the approach I am currently using when handling unit tests in the Yocto SDK-based application development workflow.
PROS:
- Same versions for tools and libraries as target build.
- Same compiler version as target build.
- Environment setup script and full sysroot available for host.
CONS:
- Separate host SDK and build.
Conclusions
In conclusion, while unit testing is an integral part of modern software development, there does not seem to be a standard way to handle it in the Yocto SDK application development workflow. Each of the alternatives mentioned has its own pros and cons, and none of them is perfect. The approach of using a separate SDK that targets the host machine (genericx86-64) seems to provide the least manual effort in managing dependencies and test framework configuration. However, it does require a full Yocto build to generate the x86 SDK and has some overhead in maintaining and shipping two separate SDKs.
More optimal way would be to bundle both the native and the target toolchains and sysroots into the same SDK, and have the environment setup scripts for both. This way the single SDK could be used for normal target builds as well as for the unit test builds on the host.
Is this already possible, or is there a better way to handle this case? If so, please let me know!
Page Fault Blog 