judges and ground truth

[geo-awad]

Hello,
Sorry if I missed this information. But, I am trying to find how to setup the ground truth , judging process, etc.
Is there any documentation or guide for this?

Thanks
George

[lucaro]

Dear George,

in order to make use of the judgement mechanism, the following steps are required:

1. Import the relevant media collection as described in https://github.com/dres-dev/DRES/wiki/Collection-Management. For the judgement to work, it is important for videos to also import the segmentation information as described at the bottom of the page.
2. Create a 'Competition' with a task type that uses 'Judgement' as a validation mechanism and one or more task categories using that type. Also add all teams that will participate in this evaluation.
3. Create the tasks with the relevant task descriptions. These tasks do not currently support pre-defined ground-truth information, but that's on the TODO list, see Pre-specify known judgements #321.
4. Start a run for the relevant competition.
5. Have teams submit their results. There is currently no way to import previously generated submissions through the UI, but we did such things before through the API. So this would be generally possible, but there is not yet a straight-forward workflow for it.
6. As soon as there is at least one result, use the judgement viewer to view individual submissions and assess them as correct, wrong, or undecidable.

I hope this is helpful as a brief overview, just let us know if you'd need different functionality and we can see how to best support that.

Best regards,
Luca