Backend: Intelligently Evaluate Available Data in Repo Even When A Tailhash is Missing

[frozenfrank]

Overview

In #420, we fixed an issue that caused the commit verification system to reject grading and fail to save a submission if a tail hash was missing during evaluation. Now the student will still be failed, but a submission object will be created which allows for manual intervention to apply the score for the student.

However, in all the relevant use-cases, the students did not really mean to nuke their repo. Typically, they are still setting up the project for the first time, run in to issues, and reset everything in order to fix it (after submitting the GitHub assignment). There is no need for manual intervention in these use-cases. There should be enough information for us to make a fair grading decision without manual intervention even if the tail commit is no longer available.

Solution

Currently, we filter the commits in the main branch by only the ones that were not included in the previous submission (this requires a valid tail commit) and then exclude other corner cases by verifying the commit timestamp. This double verification is the most robust, but we could relax the verification and still feel confident that the system is behaving as it should when a tail commit is not available.

This relaxation of requirements could be easily accomplished using existing architecture in the codebase. The GitHelper.getMostRecentPassingSubmission() method is responsible for identifying the latest submission commit and the latest submission timestamp and then merging the two pieces of information together into a CommitThreshold record. Currently, the logic tracks them together and updates both values when possible, but we could separate these concerns to update the latest timestamp, and update the latest submission hash only when it exists. The rest of the logic down the line already applies both of the filters independent of each other. This would result in a relaxation of the requirements and save students from needing a manual intervention if they blow up their repo and restart. It will still verify that the new commits were authored after the latest submission and avoid double-counting commits towards the minimum requirements.

Potential Concerns

1. There is a strong use-case for providing this relaxation on Phase 0 (immediately following the GitHub assignment), but we may not want to have this potential loophole open for later phases where there is no demanding use-case.
   • This is not opening up an actual loophole since we are verifying author timestamps, and those author timestamps have been verified the entire time.
   • We could limit this relaxation as an exception on the Phase 0 phase. (If a student were to have this issue on a future phase, they would need a TA to approve the score.)

Moving Forward

Feedback is welcomed. Please comment and weigh in on this issue.