Skip to content

Testgroup verdicts, expectations

No due date 75% complete

For type: pass-fail

  • Apply {input, output}_validator_flags from testdata.yaml to the respective testcase when validating
  • Collect testcase verdicts, infer testdata verdicts for each testgroup
  • Update UI to report testgroup verdicts

For type: scoring:

  • Allow testcase verdicts to have numerical score
  • Apply grading flags from testdata.yaml to respective …

For type: pass-fail

  • Apply {input, output}_validator_flags from testdata.yaml to the respective testcase when validating
  • Collect testcase verdicts, infer testdata verdicts for each testgroup
  • Update UI to report testgroup verdicts

For type: scoring:

  • Allow testcase verdicts to have numerical score
  • Apply grading flags from testdata.yaml to respective testcase scores
  • Apply aggregation rules from grading in testdata.yaml
  • Update UI to report aggregate scores for testgroups

Expectations:

  • Read and parse expectations.yaml
  • Compare testcase verdicts with expectations
  • Compare testgroup verdicts with expectations
Loading