[WIP] Create a framework to exercise the go test output

[knz]

Prior to this patch, there was no way to exercise the behavior of
-rewrite nor how errors and sub-tests would be reported in the
output of go test -v.

This patch introduces such "reflective" tests, where a test can
compare the output and modified file through running go test
recursively.

---

This change is 

[knz]

What do you think? I was missing a way to properly test all the features of the datadriven package. I think this way we can do this.

If you think this is a good idea, I will add unit tests for the features already there - error handling, sub-tests, parsing errors, etc.

[RaduBerinde]

Not crazy about running go from the test.. Couldn't we change the infrastructure to take a testing.TB and provide an instrumented implementation for the purposes of testing?

[knz]

Couldn't we change the infrastructure to take a testing.TB and provide an instrumented implementation for the purposes of testing?

That was my first idea actually, but I found it unsatisfying for two reasons.

The first reason is that it's quite a large amount of work. Not just the interface change, which is minor; the implementation of a mock testing.TB is quite the challenge, as there is a complex interplay between goroutines (one per test), parent-child goroutines (subtests) and the channels used to synchronize (t.Fatal/t.Skip use runtime.Goexit to terminate early). Just the mock would implement 30-40% of the testing package, which is quite a lot.

The second reason (and for me the main reason) is that I really want to test the output produced by failing datadriven tests when ran under go test:

• the command line flag parsing
• how the file/lineno reference look like in error messages
• whether the sub-test names are properly assembled (I just found a bug in this area, PR upcoming)

It looks to me like the amount of refactoring / mock injection needed to test these aspects is unreasonably large compared to the (relative) simplicity of the approach I took here instead.

What do you think?

[RaduBerinde]

I was thinking the implementation would still use a *testing.T underneath (the one from our test), you'd just intercept some of the calls we care about (Error,Fatal). I might be missing some subtleties though, I'll leave it to you to decide since you spent more time thinking about this. I don't have strong objections given that it's testing code and not even in the main repo. I'll look more closely at the final change.

[knz]

you'd just intercept some of the calls we care about (Error,Fatal). I might be missing some subtleties though

I'd like to understand more: how do you mock something that properly casts down to *testing.T in some places, but uses a mock in others?

[RaduBerinde]

type myT struct { *testing.T }

[knz]

Oh TIL.

Ok I'll experiment with that.

[RaduBerinde]

It wouldn't cast down, but it would implement testing.TB. Hm though I guess that doesn't have everything we need (eg Run). We'd need to define our own interface and that will make the API more complicated. Your solution is probably better.

[knz]

It doesn't have Run and also implementing Fatal and Skip is annoying (goroutines + defers + weird synchronization)