use testdata_dir when creating test_list

[gildor478]

This bug has been migrated from artifact #1475 on forge.ocamlcore.org. It was assigned to user102.

user163 posted on 2015-09-22 20:58:47:

As far as I can tell, it is impossible to make use of testdata_dir when creating a test_list. My set of tests is dictated by the files within a data directory. Each file corresponds to one test. Thus, I want to do a readdir on the directory, then create 1 test for each file that I find within that directory. I think this is impossible with the current API.

For concreteness, I want to do this:

let make_test (filename:string) : test =
(* create a test corresponding to the given file *)

let make_test_list (dir:string) : test_list =
test_list (List.map (Sys.readdir dir) make_test)

But I can't do the above. The only way to get the dir argument needed above is by using a test_ctxt, which OUnit2 only provides when constructing a test, not a test_list.

user102 replied on 2015-09-23 08:37:39:

testdata is data for tests and not tests.

Actually, you can achieve what you want to do in different ways:

Solution 1.

Have a test that tests your test list is complete:

let lst = [
dir1 >:: (fun test_ctxt -> ...);
dir2 >:: (fun test_ctxt -> ...)
]

let tests =
[
"include_all_dirs" >::
(fun test_ctxt ->
let lst1 = List.map fst lst (* extract dirX from lst )
let lst2 = Sys.readdir testdata_dir ( extract dir from filesystem *)
assert_equal lst1 lst2)
] @ lst

Solution 2.

Use OUnit2.non_fatal:

"my_big_test" >::
(fun tst_ctxt ->
Array.iter
(fun fn->
non_fatal test_ctxt
(fun test_ctxt ->
(* Test something about fn. *)
()))
(Sys.readdir testdata_dir))

S2 is more automatic than S1, but you get only 1 big test.

user163 replied on 2015-09-27 19:09:13:

I'm using Solution 1. Thanks for suggesting the extra test to check my hard coded list is complete. That helps.

I didn't like Solution 2's result of having just 1 big test.

testdata is data for tests and not tests.

Sure, but why. My need seems quite reasonable.

user102 replied on 2015-09-27 20:43:23:

testdata is data for tests and not tests.
Sure, but why. My need seems quite reasonable.

Your needs are totally reasonnable and I faced the same kind of problems for other projects.

Let's say solution 1. and 2. are reasonable options to your problem (I will assume you somehow agree). At least we have a solution.

Here are the various problem and solutions I have considered:

1. testdata dir can be changed on the commandline and through an environment variable. If you use it before run_test_tt_main (which contains command line parsing), how will you account changes done on the command line?

2. testdata is data for tests -> I said that because it is meant to be the thing you need to copy to a slave nodes to run a tests (and use testdata_dir on the command line to run the test). Nothing has really been implemented so far, but this was the initial idea.

3. if you really feel strong about it, you can use OUnitConf and OUnitTestdata and recode run_test_tt_main to do exactly what you need

user163 replied on 2015-09-29 16:29:15:

Ok, thanks for the explanations.