Testgroup verdicts, expectations
No due date
75% complete
For type: pass-fail
-
Apply
{input, output}_validator_flags
fromtestdata.yaml
to the respective testcase when validating - Collect testcase verdicts, infer testdata verdicts for each testgroup
- Update UI to report testgroup verdicts
For type: scoring
:
-
Allow testcase verdicts to have numerical
score
-
Apply
grading
flags fromtestdata.yaml
to respective …
For type: pass-fail
- Apply
{input, output}_validator_flags
fromtestdata.yaml
to the respective testcase when validating - Collect testcase verdicts, infer testdata verdicts for each testgroup
- Update UI to report testgroup verdicts
For type: scoring
:
- Allow testcase verdicts to have numerical
score
- Apply
grading
flags fromtestdata.yaml
to respective testcase scores - Apply aggregation rules from
grading
intestdata.yaml
- Update UI to report aggregate scores for testgroups
Expectations:
- Read and parse
expectations.yaml
- Compare testcase verdicts with expectations
- Compare testgroup verdicts with expectations