Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reject Test Instance Submissions with "bad" YAML #24

Open
wmwolf opened this issue Nov 18, 2020 · 0 comments
Open

Reject Test Instance Submissions with "bad" YAML #24

wmwolf opened this issue Nov 18, 2020 · 0 comments
Assignees
Labels

Comments

@wmwolf
Copy link
Member

wmwolf commented Nov 18, 2020

Test instances that are missing mandatory information (broadly defined), like a test case name, or a failure_type (if passed is false) should never make it into the database and should be spat back to the submitter while accepting the rest. Might also spit back unrecognized keys since they are likely wrong, or at least as a reminder to the developer to make them do something on the backend.

Background

mesa handles creating the YAML files, and mesa_test handles submitting them. In principle, mesa_test could check the YAML files for consistency so it never makes it to the server, but that would require mesa_test to be "smart", and more importantly, updated. Kicking this up to the testhub means that mesa_test can remain a blameless messenger between the testing scripts of mesa and the database in the testhub.

@wmwolf wmwolf added the bug label Nov 18, 2020
@wmwolf wmwolf self-assigned this Nov 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant