Discovery works but execution fails with file not found error

[abadrinath15]

Hello,

I didn't seem to have this problem before, but something seems to have gone a bit wonky this week (no changes to config on my end).
Using the vscode Testing pane, I'm seeing tests successfully discovered, but if i try to run (or debug them), they automatically fail with a

ERROR: file or directory not found: <package/module/submodule_1/submodule_2/tests/test_file.py>

Our layout is a bit interesting, but at a high level it's something like

repo\
    lib\
        package\

The terminal spawns at repo, but I have an .env file with PYTHONPATH=lib. All this seemed to have worked until say... Thursday? :) . Perhaps I had it setup wrong. In the debug output, I do see it now showing:
Recieved JSON data: [package/module/submodule_1/submodule_2/tests/test_file.py::tests_blahblah

I've never seen that Received JSON line before; i do wonder if something on the paths is getting a little messed up from the latest version? Apologies for the mediocre output trace; trying to abstract it as much as I can. Let me know what else I could do to be helpful?

I'm using v2023.12.0.

And of course running something like pytest lib\package\module\submodule_1\submodule_2\tests\test_file.py works perfectly fine from terminal!

Thanks.

[juandiago]

I can confirm this same issue on my end with versions above 2023.8.0, had to pin the extension version to 2023.8.0 on my devcontainer.json to be able to work with the Testing panel again. Looks like a regression got introduced on releases after 2023.8.0.

From early investigation, seems like pytest's rootdir is getting changed between the discovery phase and the run phase of the tests.

pytest.ini

[pytest]
DJANGO_SETTINGS_MODULE = app.settings
pythonpath = . src
env =
    ENV=test
addopts = -v --color=yes -p no:warnings

[juandiago]

flagging some similar issues over here:

#21637
#21632

[akarev0]

@juandiago downgraded extension version from 2023.12 to 2023.8 and I can verify that it works again, thanks :)

[eleanorjboyd]

Thank you for your issue report. We are looking into this now! In the meantime, you are likely on the new testing rewrite and this is why you saw a change in behavior since we just moved to this experiment to a larger group of people receiving it. You can opt out of the rewrite as I get this fix in by setting this in your user settings: "python.experiments.optOutFrom": ["pythonTestAdapter"],. If this doesn't work let me know as this will mean it is a different issue. Thank you and I will send updates in this thread as I get a fix in.

[eleanorjboyd]

Hello! Would people agree that #21632 has a minimal repro? I have tried this with version v2023.8.0 and it is still not working for me. Can anyone provide any ideas on what might be going wrong? Here is my minimal repro as a zip
path_issue.zip

[juandiago]

@eleanorjboyd some more data points:

• Disabling the pythonTestAdapter, allows me to use the Testing panel again, so it does seems related to that
• The output I get on the console is not the same as test explorer cannot import .env #21632, I get a file-not-found error
• Checking the pytest conf on the test logs, the rootdir changes between the detection phase & the run phase of the test, resulting(I presume) on the file-not-found error

[abadrinath15]

@eleanorjboyd some more data points:

• Disabling the pythonTestAdapter, allows me to use the Testing panel again, so it does seems related to that

• The output I get on the console is not the same as test explorer cannot import .env #21632, I get a file-not-found error

• Checking the pytest conf on the test logs, the rootdir changes between the detection phase & the run phase of the test, resulting(I presume) on the file-not-found error

I just wanted to hop-in and say my experience is the same as @juan on points 2/3, and I notice the rootdir thing as well. I unfortunately can't test 1 / the repro until later, but, rolling back to 2023.8 did work for me as well.

[eleanorjboyd]

Using the zip below as a minimal repro and still not seeing it (sorry!). What is listed in your settings for the pytest args? Anything else I set up incorrectly? I appreciate all the help- thanks! path_changing.zip

[abadrinath15]

@eleanorjboyd hello and thank you for your help!

The problem I diagnosed is on my work computer so it's a bit tricky to try to do replicate/do it with the zip you've attached. I tried it with my personal computer (which is also a different OS, fun) -- i see slightly different stuff in that discovery fails as well, but the error it reports looks familiar. The major I change i made to your repro was to use an .env file I suspect (other basic layout stuff but I doubt it matters).

repo.zip

let me know if i did anything super stupid, and thank you!

[eleanorjboyd]

Hello! Upon further investigation I have opened up this issue upstream on pytest as I think there is an issue with testIDs and the rootdir. Will look to see if I can think on a fix upstream or a patch. Thanks

[eleanorjboyd]

@abadrinath15 thank you for the example repo, got an error when I ran it even from the command line. Did I set up something wrong? Seems like pytest is having trouble finding the package even without the extension.

(.venv) eleanorboyd@Eleanors-MacBook-Pro repo % ./.venv/bin/python -m pytest --collect-only lib 
=========================================================================================================================== test session starts ============================================================================================================================
platform darwin -- Python 3.9.6, pytest-7.4.0, pluggy-1.2.0
rootdir: /Users/eleanorboyd/testingFiles/from_users/repo
collected 0 items / 1 error                                                                                                                                                                                                                                                

================================================================================================================================== ERRORS ==================================================================================================================================
______________________________________________________________________________________________________________ ERROR collecting lib/package/tests/test_ex.py _______________________________________________________________________________________________________________
ImportError while importing test module '/Users/eleanorboyd/testingFiles/from_users/repo/lib/package/tests/test_ex.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
lib/package/tests/test_ex.py:1: in <module>
    from package.example import do_nothing
E   ModuleNotFoundError: No module named 'package'
========================================================================================================================= short test summary info ==========================================================================================================================
ERROR lib/package/tests/test_ex.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=================================================================================================================== no tests collected, 1 error in 0.05s ===================================================================================================================
(.venv) eleanorboyd@Eleanors-MacBook-Pro repo %

UPDATE:
I think the issue is pytest doesn't use .env files so you just have to do the same functionality a different way and that is why the lib path in the env file isn't working

pytest core doesn't come with any support for .env files - but there are plugins which add that functionality, such as [pytest-dotenv](https://github.com/quiqua/pytest-dotenv) or [pytest-env](https://github.com/MobileDynasty/pytest-env).
comment link

[eleanorjboyd]

Also if someone could confirm the zip I uploaded here: pytest-dev/pytest#11235 is a minimal repro that would be great. If not please let me know what I got wrong as the bug I found could be only partially the cause of your bug.

[nleroy917]

Downgrading to 2023.8 worked for me as well. Thanks @akarev0

[juandiago]

@eleanorjboyd saw you guys reverted the fix, should we reopen this issue?

[eleanorjboyd]

Good catch @juandiago, thanks! Yes, the fix was reverted due to an error that appeared in the testing behavior but this commit was not actually the culprit so we will be re-merging it soon. Thanks

[schaveyt]

I am also impacted by this issue. It seems to be only impact a limited set of machines and we cannot figure out why. "python.experiments.optOutFrom": ["All"] which takes the new testAdaptor out of the picture.

I can see it invoke

C:\Python311\python.exe c:\Users\{myusername}\.vscode\extensions\ms-python.python-2023.14.0\pythonFiles\lib\python\debugpy\adapter

And the test is attempts to run is still off by 1 folder level. Ugh

[eleanorjboyd]

Hi @schaveyt, are you saying that opting out from the experiment fixes the issue or it is still present outside of the experiment?

[schaveyt]

akarev0 how do you downgrade to that earlier version mentioned above?

[akarev0]

@schaveyt
you could choose version right here, into installed extansion list

[schaveyt]

karev0 - Awesome! I installed the 2023.08 and everything works. This will do for now i suppose

[github-actions]

Hey @eleanorjboyd, this issue might need further attention.

@abadrinath15, you can help us out by closing this issue if the problem no longer exists, or adding more information.

[abadrinath15]

Hey @eleanorjboyd, this issue might need further attention.

@abadrinath15, you can help us out by closing this issue if the problem no longer exists, or adding more information.

I think @eleanorjboyd and team have fixed it so happy to close. Thank you!

[akarev0]

@abadrinath15 not they not, it's still there

[abadrinath15]

@abadrinath15 not they not, it's still there

Oh sorry! Undoing that.

[akarev0]

@abadrinath15 yeah, look fixed, sorry :)

[abadrinath15]

@abadrinath15 yeah, look fixed, sorry :)

ok!

[eleanorjboyd]

Hi, what is the status on this? Did people try it and it wasn't fixed?

[abadrinath15]

Hi @eleanorjboyd! To my understanding it was fixed and that's why I closed it.

[juandiago]

Can confirm resolved, everything green & working