(!) Please ask about problems and questions regarding this tutorial on answers.ros.org. Don't forget to include in your question the link to this page, the versions of your OS & ROS, and also add appropriate tags.

Integrate tests in catkin

Description: In this tutorial you will learn why you should integrate tests in the build when using catkin.

Keywords: Unit Testing, Rostest, Catkin, Quality Assurance

Tutorial Level:

Contents

  1. Context
  2. Problem
    1. Example
  3. Solution
  4. Links
  5. Consequences
  6. Related patterns

Context

An application is unlikely to be developed only once, and the software is probably subject to continuous change either as part of the development process or as part of later maintenance and evolution. Thus, crucial behavior of the software can get affected and in worst case fail or yield wrong outputs.

Problem

In many cases error(s) will not arise right away, and at the time that a developer (or user) comes across this bug it might be difficult to track it down to its source. Even if the developer defined certain ad-hoc tests for the software, it is likely that they are not executed on a regular basis.

Example

An example could be that a team develops a library for path planning of a robot. One of the developers wants to clean up the code and decides to use matrices calculations instead of simple equations. However, he mismatches two entries of a matrix, or forgets a minus at some position. Afterwards, he pushes his changes to the common repository of the library and moves on to his next task. This will introduce a bug that might not be apparent at first, but could cause problems later on in specific cases.

Solution

A solution to avoid the problem in the aforementioned example is to write a test that is run regularly. This test could use inputs that has a known outcome, and then compare that outcome with the results from the calculations. If the difference is higher than a certain threshold (e.g. to account for numerical errors) then the test will fail.

Applications needs to be tested regularly to catch these types of errors. Thus, one should define explicit tests for the crucial behavior of the program. Common methods for this purpose are unit tests and rostests (an extension to roslaunch, which enables testing across multiple ROS nodes).

In order to run tests for a ROS package globally, they can be defined in the build system of ROS (i.e. catkin). Afterwards, the catkin tools offer to run all tests of the ROS workspace either at once or individually. This makes it more convenient to run tests on a regularly basis. Furthermore, they can be integrated into Continuous Integration to run the tests for example for every push to the Github repository.

Conceptual overview of catkin

Configuring and running unit tests in catkin

Unit testing

Rostest

Consequences

There is an initial extra effort to develop a suitable set of test cases for the software, and as best practice a Continuous Integration is required to run the tests on a regular basis.

Additionally, if tests are run frequently then it will likely encourage others to use the software, and also reduce potentially expensive bug hunting activities.

  • Continuous Integration with the public infrastructure
  • Continuous Integration with private repositories
  • Regression Testing (unit tests)
  • Replay testing

Wiki: Quality/Tutorials/Integrate tests in catkin (last edited 2018-03-29 07:06:08 by jontje)

Except where otherwise noted, the ROS wiki is licensed under the
Creative Commons Attribution 3.0