Exit with special code upon unexpected error

[zarifmahfuz]

We set up ci-queue workers to soft fail on every exit status. We do this mainly to ignore infrastructure failures. Currently, when a test fails due a problem with the test framework, it has an exit status 1 and is ignored. We want to have a unique exit status for such failures to stop ignoring them as soft failure.

[ChrisBr]

Can you please provide some tophats before we merge it

[ChrisBr]

Suggested change

	rescue => e
	rescue => error

[ChrisBr]

I wonder if a better approach would be to let the summary reporter handle this and we just set a tombstone / error when this happens in Redis?

cc @casperisfine @zarifmahfuz

[ChrisBr]

^^ Reason for this is that I would like to experiment with setting the commit status only on the summary step, this wouldn't work anymore if we don't ignore the exit status of a worker (we can still return a different exit code though). If we put the error into Redis we can still ignore all workers and succeed if all tests passed and no worker crashed.

[zarifmahfuz]

I wonder if a better approach would be to let the summary reporter handle this and we just set a tombstone / error when this happens in Redis?

Yeah I imagine with group status checks, we need to do that that instead

[zarifmahfuz]

Merging this in for now because this created an incident today.