Discrepancy in Reproduced Results (Zuco Dataset)

[FishCatCake]

Hi Lukas,

Thank you for the valuable work and for making the code and datasets publicly available. Your research is very innovative and inspiring!

I recently went through your paper DETRtime and tried to reproduce the results using the code from this repository and the provided Zuco dataset. While I followed the steps as described in the README and the paper, the validation results I obtained were notably different from those reported in the paper.

Hyperparameter setup follows the table 6 in the paper appendix:

Obtained results for zuco:

1. Below are the f1 scores for each class (0, 1, 2), orange for training, blue for validation.

2. Below are the weighted average f1 scores and macro average f1 scores, respectively. Orange for training, blue for validation.

Reported result for zuco in the paper:

I'm reaching out to see if there might be any additional steps, configurations, or updates that I might have missed. I'm eager to understand the potential reasons for this discrepancy. Any insights or suggestions would be highly appreciated.

Thanks.
Yicen

[Onlyknight]

Hi Lukas,

Thank you for the valuable work and for making the code and datasets publicly available. Your research is very innovative and inspiring!

I recently went through your paper DETRtime and tried to reproduce the results using the code from this repository and the provided Zuco dataset. While I followed the steps as described in the README and the paper, the validation results I obtained were notably different from those reported in the paper.

Hyperparameter setup follows the table 6 in the paper appendix:

Obtained results for zuco:

1. Below are the f1 scores for each class (0, 1, 2), orange for training, blue for validation.

2. Below are the weighted average f1 scores and macro average f1 scores, respectively. Orange for training, blue for validation. **Reported result for zuco in the paper:**

I'm reaching out to see if there might be any additional steps, configurations, or updates that I might have missed. I'm eager to understand the potential reasons for this discrepancy. Any insights or suggestions would be highly appreciated.

Thanks. Yicen

Hello,I have meet the same problem with you!

[Onlyknight]

Hi Lukas,
When I was replicating your article using the zuco dataset, I found that all the final prediction boxes were overlapping in one place, and I couldn't reproduce the experimental results as effectively as you did in your paper. May I ask if there are any specific training techniques you can recommend?