Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow redirect of IO for PIP #4

Open
peardox opened this issue Jul 20, 2022 · 2 comments
Open

Allow redirect of IO for PIP #4

peardox opened this issue Jul 20, 2022 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@peardox
Copy link
Contributor

peardox commented Jul 20, 2022

Looking at PyPackage.Manager.Pip.pas and cross-referencing with PythonEnvironments' PyTools.ExecCmd.pas there are methods defined to capture IO but these are unused as everything calls the Run(AOutput) version

It seems desirable to use the capture if a (unimplemented) PIP/Conda IO event were available. This would possibly allow the use of the Progress option to monitor what the install was doing. This would be especially useful for very large packages (e.g. Torch-GPU takes 4 mins to download with no way to inform the user what's going on)

@lmbelo lmbelo self-assigned this Jul 22, 2022
@lmbelo lmbelo added the enhancement New feature or request label Jul 22, 2022
@lmbelo
Copy link
Member

lmbelo commented Jul 22, 2022

@peardox it was in our plans to make it available on events. What do you have in mind?

@peardox
Copy link
Contributor Author

peardox commented Jul 22, 2022

From looking at what PIP does it shows a progress bar when installing a new module.
This is displayed on one line being overwritten periodically looking like this...

Using pip 22.1.2 from

C:\Users\simon\AppData\Local\Programs\Python\Python39\lib\site-packages\pip (python 3.9)
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cu116
Collecting torch
Downloading https://download.pytorch.org/whl/cu116/torch-1.12.0%2Bcu116-cp39-cp39-win_amd64.whl (2388.0 MB)
-- ------------------------------------- 0.2/2.4 GB 6.6 MB/s eta 0:05:36

The very last line is the interesting one as it changes showing the progress of the download. Try it yourself with this...

pip install -v torch --extra-index-url https://download.pytorch.org/whl/cu116

Yoiu might need to "pip uninstall torch" and possibly "pip cache purge" to get a proper idea of the issue.

What I'm hoping is that the last line can be parsed to return some useful info on install progress in the 5 mins 36 secs mine says it's taking. This could then be used to drive a Delphi progress bar etc.

While I'm using an extreme example here this also applies to smaller packages - it just isn't quite as bad as this case...

A lot of ML Python stuff relies on large quantities of data - I mean CoCo 2017 (117k training images) is 18Gb for example. Those I can handle as it's just files from the Internet but PIP has no mechanism for developer-friendly progress information.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants