The SBFT Workshop offers a challenge for software testers who want to work with self-driving cars in the context of the usual tool competition.
The deadline to submit your tool is: November 11th 2023 December 1st 2023
The results of the evaluation will be communicated to participants on: December 21st 2023
The camera-ready paper describing your tool is due to: January 25th 2024
The competitors should propose a test generator that produces virtual roads to test a lane keeping assist system. The aim of the generation is to produce diverse failure-inducing tests, i.e., roads that make the lane keeping assist system drive out of the lane.
The ranking of the tools is based on coverage which measures the number of failed tests and their diversity. In fact, it is in the interest of any tester to provide as diverse failures as possible. This facilitates better root cause analysis.
The generated roads are evaluated in the BeamNG.tech driving simulator. This simulator is ideal for researchers due to its state-of-the-art soft-body physics simulation, ease of access to sensory data, and a Python API to control the simulation.
In the competition two lane keeping assist systems are used: BeamnNG.AI provided by the BeamnNG.tech simulator and DAVE-2 trained by the competition organizers (with a speed-limit of 25 km/h).
Note: BeamNG GmbH, the company developing the simulator, kindly offers it for free for researcher purposes upon registration (see Installation).
We make available a code pipeline that will integrate your test generator with the simulator by validating, executing and evaluating your test cases. Moreover, we offer some sample test generators to show how to use our code pipeline.
Deciding which test generator is the best is far from trivial and, currently, remains an open challenge. In the 2024 edition of the competition, we use different diversity metrics to rank the test generators. The first metric we use relies on feature maps and measures much each tool covers the map. Possible features to be used include:
- Direction Coverage (DirCov).
- Standard Deviation of the Steering Angle (StdSA).
- Maximum Curvature (MaxCurv).
- Mean Lateral Position (MLP).
- Standard Deviation of the Speed (StdSpeed).
The second metric relies on clustering techniques to measure the diversity of the trajectories of the ego vehicle. The more clusters a test generator covers, the more diverse its failures.
We expect that the submitted tools are stochastic in nature, so we compute the coverage as the total coverage over several repetitions of the tool.
More information can be found on the SBFT tool competition website: https://sbft24.github.io/tools/. See also the tool report of the previous competition edition: https://ieeexplore.ieee.org/document/10190377
Code pipeline: code that integrates your test generator with the simulator.
Self driving car testing library: library that helps the integration of the test input generators, our code pipeline, and the BeamNG simulator.
Scenario template: basic scenario used in this competition.
Documentation: contains the installation guide, detailed rules of the competition, and the frequently asked questions.
- Installation Guide: information about the prerequisites and how to install the code pipeline.
- Guidelines: goal and rules of the competition.
- FAQ: answers to the most frequent asked questions.
Sample test generators: sample test generators already integrated with the code pipeline for illustrative purposes.
Requirements: contains the list of the required packages.
The software we developed is distributed under GNU GPL license. See the LICENSE.md file.
Dr. Matteo Biagiola - Università della Svizzera italiana, Lugano, Switzerland - [email protected]
Dr. Stefan Klikovits - Johannes Kepler University, Linz, Austria - [email protected]