Writing a JQF test

This page is a quick reference for those who are already familiar with junit-quickcheck. For a tutorial that explains generator-based fuzzing from scratch, see Fuzzing with Zest.

JQF builds on junit-quickcheck, so do checkout the junit-quickcheck documentation for tips on writing generators for complex types. Let's summarize some of the differences between junit-quickcheck and JQF:

• Test methods must be annotated with @Fuzz instead of @Property. The @Property annotation allowed setting several configuration parameters such as the number of trials or the random seed to use; in JQF, we defer the running of the fuzzing loop to a separately configurable front-end. For example, when using AFL as the front-end, the fuzzing loop runs infinitely unless the AFL process is killed by an external user.
• Test classes must be annotated with @RunWith(JQF.class) instead of @RunWith(JunitQuickcheck.class). However, class JQF extends JunitQuickcheck, so a JQF test class can have a mix of JQF @Fuzz targets, Quickcheck @Property tests, and classic Junit @Test methods without any arguments.

All JQF tests can be part of your standard unit test suite -- JQF will sample 100 inputs at random every time the unit test suite runs (e.g. as part of mvn test). The same tests can be executed with code-coverage feedback for an extended fuzzing session (see fuzzing with Zest).

Adding Dependencies to JQF

You'll of course need to add JQF as a dependency to your test classes. You can either add that manually or using Maven.

Option 1: Command-line

Use the handy script classpath.sh that expands to the JQF classpath needed to compile your tests.

# Assume you have a test class in `MyTest.java`
javac -cp $(jqf/scripts/classpath.sh) MyTest.java

Option 2: Use Maven

JQF is in Maven central, so you can just depend on the jqf-fuzz artifact as follows:

    <dependencies>
        <dependency>
            <groupId>edu.berkeley.cs.jqf</groupId>
            <artifactId>jqf-fuzz</artifactId>
            <version>1.7</version>
        </dependency>
    </dependencies>

Note: You may want to double-check what is the latest released version, since this wiki page may go out of date. Use the version string after the prefix "jqf-" (e.g. if the release is "jqf-1.X" then use "1.X" as version).

Sample tests

Look at some examples bundled with JQF for sample tests.

Let's walk through some examples here.

JDK DateTimeFormatter

@RunWith(JQF.class)
public class DateFormatterTest {
    @Fuzz
    public void fuzzLocalDateTime(String date, String pattern) throws IllegalArgumentException, DateTimeParseException {
        LocalDateTime.parse(date, DateTimeFormatter.ofPattern(pattern));
    }
}

Here, we test the date formatting logic in the java.time package. JQF will generate the date and pattern automatically. Anything listed in the throws clause is assumed to be normal, i.e. we expect those exceptions to be thrown by our code, so they will not be marked as failures. Any other exceptions (e.g. ArrayIndexOutOfBoundsException or NullPointerException) will be marked as failures.

Maven - ModelReader

@RunWith(JQF.class)
public class ModelReaderTest {
    @Fuzz
    public void testWithSequence(InputStream in) throws IOException {
        ModelReader reader = new DefaultModelReader();
        Model model = reader.read(in, null);
        Assert.assertNotNull(model);
    }

    @Fuzz
    public void testWithGenerator(@From(XmlDocumentGenerator.class) @Dictionary("dictionaries/maven-model.dict") Document dom) throws IOException {
        testWithSequence(XmlDocumentGenerator.documentToInputStream(dom));
    }
}

In this example, we test Apache Maven's parsing of the pom.xml file. We can either make JQF provide a sequence of characters in an InputStream, which is the first variant -- testWithSequence -- or we can explicitly provide a generator of valid XML files using the W3C DOM API, the XmlDocumentGenerator, and then test the Maven POM parsing in testWithGenerator. For the DOM generator, we also provide a dictionary file comprising of literal strings scraped from the MavenXpp3Reader class.

Closure - Compiler

See Fuzzing a Compiler for an extended tutorial on writing a test driver for the Google Closure Compiler.