Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "discovery" as a kind for TestRunProfiles #203468

Open
eleanorjboyd opened this issue Jan 25, 2024 · 7 comments
Open

Add "discovery" as a kind for TestRunProfiles #203468

eleanorjboyd opened this issue Jan 25, 2024 · 7 comments
Assignees
Labels
api testing Built-in testing support under-discussion Issue is under discussion for relevance, priority, approach

Comments

@eleanorjboyd
Copy link
Member

eleanorjboyd commented Jan 25, 2024

The testing API currently allows for TestRunProfiles with 3 different options for kind: run, debug, or coverage. With the Python extensions upcoming changes to custom args which expands functionality, we would need a discovery kind of profile (see proposal for custom args here). Looking to have this profile be configurable the same way the run button is in the test view when you have multiple configs. Have the discover button (the refresh button) have a dropdown that lists the different configuration options, a default, and a click through option to edit the configs.

@connor4312
Copy link
Member

connor4312 commented Jan 26, 2024

I'm not clear on why we need a "discovery" test run profile kind. What kind of results would be reported from a discovery test run? Why does the refreshHandler not already solve this?

@eleanorjboyd
Copy link
Member Author

The "discovery" run kind would allow the user to have different args configured for their test discovery. It wouldn't necessarily be a "run" event as much as it would just trigger the same "discover tests" event which is triggered by the "refresh" button in the test explorer. From my understanding the refreshHandler does not pass in config I could somehow reference to understand which discovery config in the settings.json the user is referring to.

@connor4312
Copy link
Member

This seems like we're trying to squeeze test profiles into a place where they don't really belong.

Would it be acceptable for the extension to instead show a quick pick in its refreshHandler if there's >1 profile configured so that the user can pick which profile(s) to refresh that way?

@eleanorjboyd
Copy link
Member Author

Which is the refreshHandler? That seems like it would be fine if the refreshHandler quick pick / selection was in an intuitive place for users.

@connor4312
Copy link
Member

connor4312 commented Jan 30, 2024

The refreshHander is defined on the TestController and is what's invoked when the "refresh" button in the Test Explorer view is clicked.

@eleanorjboyd
Copy link
Member Author

Yes this sounds good! Good point that it is not a TesetRunProfile but instead a different style of profile since discovery is a different process.

Would it be possible to have UI where a user can select a default for their discovery profile? My worry is the flow is not as clean if users must click "refresh" -> "select profile" to run every time instead of just "refresh" and then change the default if they do want to switch up their discovery command. We could also opt to do the quickpick for a MVP and make changes later based on what users actual usage looks like.

@connor4312
Copy link
Member

From discussion: we can have the refreshHandler refresh all configurations, and an extension-contributed command can run discovery with individual configurations. If telemetry usage shows people are using the per-configuration discovery command often, we can consider making some API changes for this.

@rzhao271 rzhao271 added api under-discussion Issue is under discussion for relevance, priority, approach testing Built-in testing support labels Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api testing Built-in testing support under-discussion Issue is under discussion for relevance, priority, approach
Projects
None yet
Development

No branches or pull requests

3 participants