Fix for #54 - allow regexes when matching expected message text #55
Conversation
…d output rather than a string.
brunns
force-pushed
the
regex-message-assertions
branch
from
3518b79 to
ced9be4
Compare
README.md
Outdated
| @@ -62,6 +62,7 @@ On top of that, each case must comply to following types: | |||
| | `env` | `Optional[Dict[str, str]]={}` | Environmental variables to be provided inside of test run | | |||
| | `parametrized` | `Optional[List[Parameter]]=[]`\* | List of parameters, similar to [`@pytest.mark.parametrize`](https://docs.pytest.org/en/stable/parametrize.html) | | |||
| | `skip` | `str` | Expression evaluated with following globals set: `sys`, `os`, `pytest` and `platform` | | |||
| | `regex` | `str` | Allow regular expressions in comments to be matched against actual output. | | |||
There was a problem hiding this comment.
Let's say that the default is no
There was a problem hiding this comment.
It does indeed default to "no" - I'll mention that.
brunns
force-pushed
the
regex-message-assertions
branch
from
c239a2d to
9811300
Compare
| @@ -6,3 +6,11 @@ | |||
|
|
|||
| reveal_type(a) # N: Revealed type is "builtins.int" | |||
| reveal_type(b) # N: .*str.* | |||
|
|
|||
| - case: expected_message_regex_with_out | |||
There was a problem hiding this comment.
We also need a test without regex attribute and with regex: no where we try to use regex output
There was a problem hiding this comment.
I'm not sure how to do negative tests - ie tests where we expect a failure. Are there any examples of this kind of test?
There was a problem hiding this comment.
For example, if I added this test:
- case: should_fail
regex: no
main: |
a = 'hello'
reveal_type(a) # N: .*str.*I'd expect the test to fail - but that fails the build. Is there a way to express that I expect this to fail?
There was a problem hiding this comment.
Nope, see #46
The easiest way for now is to add all failing tests into a single non-collectable file. And run it with pytest after the main test case. We can grep that failed X tests
There was a problem hiding this comment.
Can you approve me to run workflows? I have something which works locally, but I'd like to see if it works in CI.
I suppose it could be merged now, but there are a couple of things we might want to add. Firstly, the template idea from #54, and secondly, some way of making a single expectation a regex rather than the whole test. What do you think? Re #45, I think this PR works for that use case, though in a slightly different way. |
Maybe we can use |
brunns
force-pushed
the
regex-message-assertions
branch
from
83a56e1 to
8defa6a
Compare
|
I'm not sure the template idea belongs in this PR, so I think it's ready to go AFAIC. |
brunns
force-pushed
the
regex-message-assertions
branch
from
9acfd71 to
848282e
Compare
| a = 'hello' | ||
| reveal_type(a) # N: .*str.* | ||
|
|
||
| - case: rexex_but_turned_off |
There was a problem hiding this comment.
Let's add one more test that should fail: with wrong regex. Something that does not match the output 👍
|
Thanks a lot for your help! |
No description provided.