Skip to content

Commit

Permalink
Documentation: Rework the documentation and the examples
Browse files Browse the repository at this point in the history
  • Loading branch information
saroupille committed May 12, 2024
1 parent 0a126b9 commit 7637902
Show file tree
Hide file tree
Showing 21 changed files with 1,004 additions and 625 deletions.
109 changes: 92 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,28 +2,40 @@

## Overview

Bam is an OCaml library designed for writing property-based tests. It
simplifies the process of defining and using generators for tests,
offering a monad-like structure that integrates seamlessly with
shrinking strategies. This design aims to make shrinking both
predictable and effective, thereby enhancing the debugging experience.
Bam is an OCaml library designed for writing property-based tests. It simplifies
the process of defining and using generators for tests (a PPX is available),
offering a monad-like structure that integrates seamlessly with shrinking
strategies. This design aims to make shrinking both predictable and effective,
thereby enhancing the debugging experience.


## Key Features

- Standard Module: The {!module:Std} module provides some basic
generators with predefined shrinking strategies

- Monad-like Generators: The library enables easy creation of new
generators, following a monad-like pattern that works harmoniously
with shrinking mechanisms.

- Shrinking Strategies: Various default shrinking strategies are
available, aiding in the efficient identification of minimal
counter-examples.
- Standard Library: The `Std` module provides some basic generators with
multiple shrinking strategies. Thanks to the modularity provided by the
Monad-like generators, it should be easy to define your generators for your
custom types.

- A PPX deriving automatically generators for types. The deriver is
highly-customizable to ensure it can be integrated quite easily into your
codebase.

- A runner with the `Tezt` test framework providing a hopefully friendly UX,
especially with *hello debugging*.

- Custom Shrinkers: The {!Gen} module allows for the definition of
ad-hoc shrinkers.
- Internal shrinking: Various default shrinking strategies are available, aiding
in the efficient identification of minimal counter-examples. Each of the
shrinking strategy has the property to be "internal", meaning that during
shrinking, all the smaller counter-examples never generates new values, but
only smaller values defined by the shrinking strategy. This property is not
guaranteed by `QCheck2` for example.

- Custom shrinking: The {!Gen} module allows for the definition of
custom shrinkers that combine well with the already shrinking strategies

- Documentation on Shrinking: For those interested in understanding
the intricacies of shrinking within this library, a detailed primer
Expand All @@ -37,24 +49,87 @@ predictable and effective, thereby enhancing the debugging experience.
opam install bam tezt-bam
```


## Usage

A simple test can be run as follows:

```ocaml
open Tezt_bam
type t = Foo of {a: int; b : string} | Bar of int list[@@deriving gen]
(** The deriver creates a value [val gen : t Bam.Std.t]. *)
let register () =
let gen = Std.int () in
let property _x = Ok () in
let property = function
| Foo {a; b} ->
if a > 1_000 && String.contains b 'z' then
Error (`Fail "A counter-example was found")
else Ok ()
| Bar [1; 2; 3; 4] ->
Error `Bad_value
| Bar _ ->
Ok ()
in
Pbt.register ~__FILE__ ~title:"Simple example of bam" ~tags:["bam"; "simple"]
~gen ~property ()
```

or without using the PPX deriver, the same example could be written as follows:

```ocaml
open Tezt_bam
type t = Foo of {a: int; b : string} | Bar of int list
let gen =
let open Bam.Std.Syntax in
let gen_Foo =
let* a = Bam.Std.int () in
let* b = Bam.Std.string ~size:(Bam.Std.int ~max:10 ()) () in
return (Foo {a; b})
in
let gen_Bar =
let* arg_0 =
Bam.Std.list ~size:(Bam.Std.int ~max:10 ()) (Bam.Std.int ())
in
return (Bar arg_0)
in
Bam.Std.oneof [(1, gen_Foo); (1, gen_Bar)]
let register () =
let property = function
| Foo {a; b} ->
if a > 1_000 && String.contains b 'z' then
Error (`Fail "A counter-example was found")
else Ok ()
| Bar [1; 2; 3; 4] ->
Error `Bad_value
| Bar _ ->
Ok ()
in
Pbt.register ~__FILE__ ~title:"Simple example of bam" ~tags:["bam"; "simple"]
~gen ~property ()
```

This example and how it can be run are explained through the various
[examples](https://github.com/francoisthire/bam/tree/master/example). We invite
you to read them if you are interested in starting with the library!

## PPX

At the moment, the PPX support is partial but should cover a vast majority of
use cases. Moreover, the deriver supports many attributes to tune its behavior.
In particular, one can specify a generator when it is not supported by the
deriver.

More examples can be found [here](https://github.com/francoisthire/bam/tree/master/example).
There is no proper documentation for the PPX yet. Instead, we invite you to look
at the many examples
[here](https://github.com/francoisthire/bam/blob/master/test/ppx.ml) to see what
types are supported and how the deriver can be tuned.

Contributions are welcome!

## License

Expand Down
18 changes: 8 additions & 10 deletions example/2-simple-failure/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,11 @@ dune exec example/main.exe -- simple_failure
[15:30:55.229] Try again with: _build/default/example/main.exe --verbose --file example/2-simple-failure/simple_failure.ml --title 'Simple failure example with bam' --seed 555586205
```

This time `Tezt` is more verbose. On a failure, the counter-example
found is printed. By default, shrinking is not enabled, hence the
counter-example found can be rather big.
This time `Tezt` is more verbose. On a failure, the counter-example found is
printed. By default, shrinking is not enabled, hence the counter-example found
can be rather big and quite complex to use for debugging.

Moreover, the error raised associated to the counter-example is
printed.

If you want to run this example again using the shrinking of bam, you can use the option `--shrink`. To be sue the very same initial counter-example will be printed, you can use the seed given by Tezt:
If you want to run this example again using the shrinking of *Bam*, you can use the option `--shrink`. To be sure the very same initial counter-example will be printed, you can use the seed given by Tezt:

```ocaml
$ dune exec example/main.exe -- simple_failure --shrink --seed 555586205
Expand All @@ -57,9 +54,10 @@ $ dune exec example/main.exe -- simple_failure --shrink --seed 555586205

At this stage, we see the shrinking heuristic allowed us to find a
smaller-counter example which is `100`. Such a counter-example can be
a good candidate for starting debugging the test.
a good candidate when debugging the test.


**Next steps:**
- The next example
[3-debugging](https://github.com/francoisthire/bam/tree/master/example/3-debugging)
will investigate how bam can help for debugging
[3-writing-generators](https://github.com/francoisthire/bam/tree/master/example/3-writing-generators)
will explain how the monadic interface of *bam* is useful to write generators easily.
76 changes: 76 additions & 0 deletions example/3-writing-generators/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# 4-writing-generators

One can take advantage of the monadic interface of *bam* to define
generators.

```ocaml
open Tezt_bam
(** The deriver creates a value [val gen : t Bam.Std.t]. *)
type t = Foo of {a: int; b: string} | Bar of int list [@@deriving gen]
let gen =
let open Bam.Std.Syntax in
let gen_Foo =
let* a = Bam.Std.int () in
let* b = Bam.Std.string ~size:(Bam.Std.int ~max:10 ()) () in
return (Foo {a; b})
in
let gen_Bar =
let* arg_0 = Bam.Std.list ~size:(Bam.Std.int ~max:10 ()) (Bam.Std.int ()) in
return (Bar arg_0)
in
Bam.Std.oneof [(1, gen_Foo); (1, gen_Bar)]
let pp fmt = function
| Foo {a; b} ->
Format.fprintf fmt "Foo {x=%d;%s}" a b
| Bar list ->
Format.fprintf fmt "Bar %a"
(Format.pp_print_list ~pp_sep:Format.pp_print_space Format.pp_print_int)
list
let register () =
let property = function
| Foo {a; b} ->
if a > 1_000 && String.contains b 'z' then
Error (`Fail "A counter-example was found")
else Ok ()
| Bar [1; 2; 3; 4] ->
Error `Bad_value
| Bar _ ->
Ok ()
in
Pbt.register ~pp ~__FILE__ ~title:"Writing generators"
~tags:["bam"; "writing_generators"]
~gen ~property ()
```

And now run it:

```bash
$ dune exec example/main.exe -- writing_generators --shrink
[11:31:02.335] [pbt] \ | /
[11:31:02.335] [pbt] - BAM -
[11:31:02.335] [pbt] / | \
[11:31:02.335] [pbt] Counter example found:
[11:31:02.335] [pbt] A {x=1001;aaaaz}
[11:31:02.335] [error] Test failed with error:
[11:31:02.335] [error] A counter-example was found
[11:31:02.335] [FAILURE] (1/1, 1 failed) Writing generators
[11:31:02.335] Try again with: _build/default/example/main.exe --verbose --file example/4-writing-generators/writing_generators.ml --title 'Writing generators' --seed 493027010
```

With *bam* it is encouraged to extensively use and abuse of the
monadic interface. The library ensures it will behave well with
respect to shrinking.

However, one can notice that *bam* did not find the smallest counter-example on this example. The reason is due to the default shrinking strategy which is efficient. Fortunately, *bam* provides different shrinking strategies to help you.

**Next steps:**
- The next example
[4-deriving-generators](https://github.com/francoisthire/bam/tree/master/example/4-deriving-generators)
will investigate how `bam-ppx` can be used to generate derivers automatically.
- The next example
[5-shrinking-strategy](https://github.com/francoisthire/bam/tree/master/example/5-shrinking-strategy)
will investigate how we can modify the shrinking strategies to find better counter-examples.
File renamed without changes.
40 changes: 40 additions & 0 deletions example/3-writing-generators/writing_generators.ml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
open Tezt_bam

(** The deriver creates a value [val gen : t Bam.Std.t]. *)
type t = Foo of {a: int; b: string} | Bar of int list [@@deriving gen]

let gen =
let open Bam.Std.Syntax in
let gen_Foo =
let* a = Bam.Std.int () in
let* b = Bam.Std.string ~size:(Bam.Std.int ~max:10 ()) () in
return (Foo {a; b})
in
let gen_Bar =
let* arg_0 = Bam.Std.list ~size:(Bam.Std.int ~max:10 ()) (Bam.Std.int ()) in
return (Bar arg_0)
in
Bam.Std.oneof [(1, gen_Foo); (1, gen_Bar)]

let pp fmt = function
| Foo {a; b} ->
Format.fprintf fmt "Foo {x=%d;%s}" a b
| Bar list ->
Format.fprintf fmt "Bar %a"
(Format.pp_print_list ~pp_sep:Format.pp_print_space Format.pp_print_int)
list

let register () =
let property = function
| Foo {a; b} ->
if a > 1_000 && String.contains b 'z' then
Error (`Fail "A counter-example was found")
else Ok ()
| Bar [1; 2; 3; 4] ->
Error `Bad_value
| Bar _ ->
Ok ()
in
Pbt.register ~pp ~__FILE__ ~title:"Writing generators"
~tags:["bam"; "writing_generators"]
~gen ~property ()
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ One can take advantage of the `bam-ppx` library to derive automatically generato
```ocaml
open Tezt_bam
type t = A of {x: int; b: bool} | B of string
type t = A of {x: int; b: bool} | B of string [@weight 4]
[@@deriving gen] [@@size.min 1] [@@size.max 10]
let pp fmt = function
Expand All @@ -26,7 +26,16 @@ let register () =
~tags:["bam"; "deriving_generators"]
~gen ~property ()
```
The attribute `[@@deriving gen]` derives automatically a generator for the type attached. The generator derived with `gen` can be tuned via other `attributes` as shown in the example above. In this example, we have specified that the minimum size and the maximum size that should be used are `1` and `10`.
The attribute `[@@deriving gen]` derives automatically a generator for the type
attached. The generator derived with `gen` can be tuned via other `attributes`
as shown in the example above. In this example, we have specified that the
minimum size and the maximum size that should be used are `1` and `10`. And that
the weight of the constructor `B` if `4` (by default it is `1`). Meaning that
the variant `B` will be generated `4` times more often than variant `A`.

More examples can be found
[here](https://github.com/francoisthire/bam/blob/master/test/ppx.ml) to
understand how the deriver can be tuned.

And now if your un it:

Expand All @@ -43,5 +52,11 @@ $ dune exec example/main.exe -- deriving_generators --shrink
[23:37:37.020] Try again with: _build/default/example/main.exe --verbose --file example/5-deriving-generators/deriving_generators.ml --title 'Deriving generators' --seed 268040292
```


**Next steps:**
- The next example
[5-shrinking-strategy](https://github.com/francoisthire/bam/tree/master/example/5-shrinking-strategy)
will investigate how we can modify the shrinking strategies to find better counter-examples.
- The next example
[6-debugging](https://github.com/francoisthire/bam/tree/master/example/6-debugging)
will investigate how *tezt-bam* can help for debugging

27 changes: 27 additions & 0 deletions example/4-deriving-generators/deriving_generators.ml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
open Tezt_bam

type t = Foo of {a: int; b: string} | Bar of int list
[@@deriving gen] [@@size.min 1] [@@size.max 10]

let pp fmt = function
| Foo {a; b} ->
Format.fprintf fmt "Foo {x=%d;%s}" a b
| Bar list ->
Format.fprintf fmt "Bar %a"
(Format.pp_print_list ~pp_sep:Format.pp_print_space Format.pp_print_int)
list

let register () =
let property = function
| Foo {a; b} ->
if a > 1_000 && String.contains b 'z' then
Error (`Fail "A counter-example was found")
else Ok ()
| Bar [1; 2; 3; 4] ->
Error `Bad_value
| Bar _ ->
Ok ()
in
Pbt.register ~pp ~__FILE__ ~title:"Deriving generators"
~tags:["bam"; "deriving_generators"]
~gen ~property ()
File renamed without changes.
Loading

0 comments on commit 7637902

Please sign in to comment.