Cut down on noise in expected strings

[ktbarrett]

Many times the full expectation string is long and includes low value information. I'd like to introduce some way to cut down on that noise. One thought is to make the expected string a fullmatch regex. This could also handle #45.

- case: test_logic_promotion
  main: |
    from hdltypes.logic import StdLogic, X01Z, Bit

    a: StdLogic
    b: X01Z
    c: Bit

    reveal_type(c & a) # N: .* "hdltypes.logic.StdLogic(?:\\*|`-?\\d+)"
    reveal_type(c & b) # N: .* "hdltypes.logic.X01Z(?:\\*|`-?\\d+)"
    reveal_type(c & c) # N: .* "hdltypes.logic.Bit(?:\\*|`-?\\d+)"

It might also be possible to make the expected strings templates to enable reuse, and perhaps package some of the common patterns with the tool.

- case: test_logic_promotion
  main: |
    # ...

    reveal_type(c & a) # N: {reveal} "hdltypes.logic.StdLogic{_}"
    reveal_type(c & b) # N: {reveal} "hdltypes.logic.X01Z{_}"
    reveal_type(c & c) # N: {reveal} "hdltypes.logic.Bit{_}"
  meta:
    reveal: "Revealed type is"
    _: "(?:\\*|`-?\\d+)"

The language here is strings, so using the common tools for that: regexes and templates, makes a lot of sense.

[sobolevn]

Related #45

[brunns]

I'd be happy to work on a PR for this if it would be welcome.

[sobolevn]

@brunns it is! Thank you!

[brunns]

Is there a one-liner for running the tests, or do I create and activate a venv by hand, then run pytest?

[sobolevn]

Yes, activate your venv then https://github.com/typeddjango/pytest-mypy-plugins/blob/master/.github/workflows/test.yml#L24-L29

[brunns]

I'm assuming we'd want a flag to activate this behavior. Otherwise,, the change wouldn't be backward compatible.

[sobolevn]

@brunns yes 👍

[brunns]

So, two options really. There could be have a flag level test, or the comment could be annotated somehow, so we could mix the two styles. Or, I suppose, both? I've made a start on the former approach in #55 (though not actually implemented the functionality yet!) but it would be easy enough to allow both options.

[brunns]

OK, #55 now implements regex matching of the messages, with a flag at the test level. Can someone let me know if I'm heading in the right direction? If so, I'll look at allowing regexes for specific messages, and also maybe the template idea.

[brunns]

Templates - would we want those at the level of a single test, or a single set of tests across the whole test file? I'm thinking the latter, given that the aim is to enable reuse.

[sobolevn]

What templates are you talking about in this context? 🤔

[brunns]

It might also be possible to make the expected strings templates to enable reuse, and perhaps package some of the common patterns with the tool.

From the original issue report.

[sobolevn]

Oh, I've missed this part. What kind of common-case templates do you think we need?

[brunns]

I don't need them myself, but the OP's example included being able to specify a reveal template which expands to "Revealed type is", and another template which expands to the regex "(?:\*|`-?\d+)".

[brunns]

It was with this in mind that I did #57, since it would presumably use the same engine.

[brunns]

#58 contains a sketch of a solution. BTW, I believe that if templates at the individual test level is prefered, you can already do that using parametrized and a single set of values.

[brunns]

Are you planning a release, @sobolevn? I'd find this very useful over at https://github.com/hamcrest/PyHamcrest !

[sobolevn]

Yes, I will reelase it this week! Thanks again!

[sobolevn]

Done! Thanks a lot @brunns