Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Struggling with permissions issues for the AllUsers settings on Windows #828

Open
2 tasks done
moorepants opened this issue Jul 31, 2024 · 24 comments
Open
2 tasks done
Labels
type::bug describes erroneous operation, use severity::* to classify the type

Comments

@moorepants
Copy link

moorepants commented Jul 31, 2024

Checklist

  • I added a descriptive title
  • I searched open reports and couldn't find a duplicate

What happened?

We build a constructor Windows installer annually and install for AllUsers in silent mode. This year many/most of the installed scripts in <install-directory>/Scripts/ have permissions that prevent execution. For example, opening the Anaconda prompt and typing Jupyter lab gives Error executing Jupyter command 'lab': [WinError 5] Access is denied. Similar errors are produced when trying to launch lab from the Anaconda Navigator. Running spyder from the prompt or Navigator works. We tried building the new installer with the constructor version (3.4.5) we used last year but still get the permission errors. We have many details of our trial and error in this issue: https://gitlab.ewi.tudelft.nl/bhmgerritsen/anaconda-dee-config/-/issues/64 but I'll summarize below.

Constructor setup:

If we download the latest official Anaconda windows installer and run it in silent mode the permissions are correct. But our installer leaves many files with restricted permissions.

Changes from last year:

We have run out of ideas as to what causes the permissions errors. Do you have any suggestions for what we may be doing wrong?

Conda Info

NA

Conda Config

NA

Conda list

NA

Additional Context

No response

@moorepants moorepants added the type::bug describes erroneous operation, use severity::* to classify the type label Jul 31, 2024
@github-project-automation github-project-automation bot moved this to 🆕 New in 🧭 Planning Jul 31, 2024
@marcoesters
Copy link
Contributor

I downloaded your 2024.01 and ran the GUI installation - everything looks good on my end. jupyter lab launches from the prompt without problems. I ran it on a Windows 11 machine where I'm the only user.

The fact that this also breaks with the old constructor makes me think that there is a specific interaction with your system. Does it maybe restrict what unsigned installers and/or binaries can do? The Anaconda and, since recently, the binaries on Windows are all signed by Anaconda.

You could also check if installing the package from defaults instead of conda-forge helps.

@moorepants
Copy link
Author

Thanks @marcoesters, good to know that it worked for you. We can look more closely at things in the environment we are installing to. I'll ask out IT if we need the signing and we're already working on checking the installs from defaults. I'll report back on our success/failure.

@moorepants
Copy link
Author

@marcoesters did you do a "JustMe" installation? Because that does seem to work for us. It is the "AllUsers" option that has the borked permissions.

@moorepants
Copy link
Author

One other important note I forgot from above, is that the installer we built last year works just fine with and AllUsers install on the computers this year. So that is odd.

@subsense33
Copy link

To reproduce the error, it is important to start the installer as the local SYSTEM account, not just as a local administrator. Our software deployment mechanism also installs software using the local SYSTEM user.

You can use Microsoft psexec.exe to start an elevated command prompt as the SYSTEM user.

  1. Open an elevated Command Prompt.
  2. Run the following command: psexec.exe -s -i cmd
  3. A new Command Prompt window will open. To verify your current user, type whoami.

@moorepants
Copy link
Author

To reproduce the error, it is important to start the installer as the local SYSTEM account

Does that mean you have reproduced the error? This is exactly how we are installing using the SYSTEM account.

Oddly, the installer we build last year installs with correct permissions with the SYSTEM account but the new installer (built with either constructor 3.4.5 or the latest version) has incorrect permissions with the SYSTEM account.

@moorepants
Copy link
Author

Ah, I didn't realize @subsense33 was my colleague here at TU Delft. He's just explaining what we've done. I assumed @subsense33 was another person commenting on this.

@moorepants
Copy link
Author

I just installed the v2024.01 release from https://gitlab.ewi.tudelft.nl/bhmgerritsen/anaconda-dee-config/-/releases on a regular Windows PC. I selected the "All Users" option and to "Register Python". I elevated admin rights to run the installer and then afterwards checked running spyder, jupyter, python console, ipython both from command line and anaconda navigator app and everything worked correctly with no permissions issues. This is presumably what @marcoesters did.

So, this points towards some kind of issue installing with the local SYSTEM account. (except that last year's installer has no issue installing with the SYSTEM account)

@marcoesters
Copy link
Contributor

marcoesters commented Aug 1, 2024

Thanks @subsense33 for these instructions! I was able to reproduce the problem now, but I suspect this is not related to constructor or your installer, but to jupyterlab. Here is what I did to come to that conclusion:

  • Open Anaconda Prompt
  • Run jupyter lab --> WinError 5. I also ran jupyter notebook with the same issue. Other executables like spyder open without errors.
  • Create an environment in a location writable to me: conda create -p %USERPROFILE%\testenv
  • conda activate %USERPROFILE%\testenv
  • conda install -y jupyterlab -c conda-forge
  • Run jupyter lab --> jupterlab opens without error

My hypothesis is that jupyterlab tries to write files into its own install location, which is write-protected. This would also explain why rebuilding the installer with an older constructor version doesn't work either. Your installer from last year (I assume) contains an older version of jupyterlab, which may not exhibit this behavior.

@marcoesters
Copy link
Contributor

Of course, executing jupyterlab from the new environment means that I have full permissions to that executable as well. However, the fact that I can execute all other binaries in the base environment tells me that jupyter tries to do some kind of write operation into a location that it's not allowed to do.

@marcoesters
Copy link
Contributor

I also manually checked the permissions inside the Scripts folder and they are what I expect: as a user, I only have read and execute permissions. The WinError is also a python error, so it reaches and executes the python interpreter somehow. If this was a permission error of your directory, I'd expect the error to be emitted by Windows, not python.

@moorepants
Copy link
Author

Thanks for the insight. The permission error also occurs when trying to run ipython from the anaconda prompt.

We could install an older version of IPython and Jupyter as a stopgap.

@moorepants
Copy link
Author

Here is an issue on the IPython tracker that implies that IPython tries to create something in the user's home directory:

ipython/ipython#12079

It is from 2020 though, so that doesn't align with our last year's installer working.

@moorepants
Copy link
Author

Searching for "jupyter" and "access is denied" gives lots of hits, like: https://stackoverflow.com/questions/61747187/jupyter-lab-in-anaconda-not-working-error-executing-jupyter-command-lab-win

@moorepants
Copy link
Author

It's worth noting that we see this:
image

indicating that we don't even have read and execute permissions for all the binaries.

@marcoesters
Copy link
Contributor

marcoesters commented Aug 1, 2024

You are right, alembic and other binaries don't have the correct permissions. jupyter does though.

For AllUsers installations, the permissions are supposed to be fixed here: https://github.com/conda/constructor/blob/main/constructor/nsis/main.nsi.tmpl#L1240-L1253

You can monitor the installation and see that you will lose read access to the entire directory during the installation, but get it back shortly after. Maybe the permissions changes are not performed recursively? I will have to investigate what those commands do exactly, but that change is also two years old, so this cannot be the only factor.

Alembic doesn't have read access throughout the entire installation process, even before the permissions are fixed in the lines above.

You may be running into a few things at once here:

  • jupyter has the correct permissions, so you may see the problem I described earlier. This isn't something constructor will be able to change, but maybe you can configure jupyter/ipython in a post-install script to write somewhere else. (the second point is more likely)
  • jupyter calls another binary that doesn't have the correct permissions. I don't know enough about the inner workings of jupyter though.
  • As for alembic et al., constructor may have to retroactively change the permissions recursively if this isn't done already. I will have to think about whether this comes with security risks, even though read and execute privileges should be relatively safe.

@moorepants
Copy link
Author

I will have to think about whether this comes with security risks, even though read and execute privileges should be relatively safe.

Our current work around is to give everything in <install-dir>/Scripts read/exec permissions in a post install script.

@subsense33
Copy link

subsense33 commented Aug 2, 2024

I will have to think about whether this comes with security risks, even though read and execute privileges should be relatively safe.

Our current work around is to give everything in <install-dir>/Scripts read/exec permissions in a post install script.

Yesterday I changed the work-around script to reset the owner on <install-dir>. Not just <install-dir>/Scripts. Does no harm, just to be sure everything is set correctly.
The script resets the owner of the entire tree to the local group "Administrators", that is how it's supposed to be. (Like the Anaconda deployment of past years).

Normal users have only read and execute permissions, that is safe. So something weird is happening with the "owner" of certain files. I use the Alembic binary just to check if the problem is there, but more files are affected.

@marcoesters
Copy link
Contributor

While that workaround probably works, the ownership of the files is not different from the Anaconda installer. I am using jupyter-kernel.exe as an example here. File permissions are listed using Get-Acl | fl.

Anaconda installer, installed as SYSTEM:

Path   : Microsoft.PowerShell.Core\FileSystem::C:\ProgramData\anaconda3\Scripts\jupyter-kernel.exe
Owner  : BUILTIN\Administrators
Group  : NT AUTHORITY\SYSTEM
Access : BUILTIN\Users Allow  ReadAndExecute, Synchronize
         NT AUTHORITY\SYSTEM Allow  FullControl
         BUILTIN\Administrators Allow  FullControl
Audit  :
Sddl   : O:BAG:SYD:AI(A;ID;0x1200a9;;;BU)(A;ID;FA;;;SY)(A;ID;FA;;;BA)

TU Delft installer, installed as SYSTEM:

Path   : Microsoft.PowerShell.Core\FileSystem::C:\ProgramData\tudelft-anaconda\Scripts\jupyter-kernel.exe
Owner  : BUILTIN\Administrators
Group  : NT AUTHORITY\SYSTEM
Access : NT AUTHORITY\SYSTEM Allow  FullControl
         BUILTIN\Administrators Allow  ReadAndExecute, Synchronize
Audit  :
Sddl   : O:BAG:SYD:PAI(A;;FA;;;SY)(A;;0x1200a9;;;BA)

So, the owner is the same, but for some reason, access isn't granted the same way. I'm wondering if this is because of differences between the two build systems (Anaconda vs. conda-forge).

I'd be really curious to see if the problem persists if you build the installer using only defaults packages.

@moorepants
Copy link
Author

This was my attempt at building the new installer the same way we did last year (defaults as priority w/ conda forge packages on top):

https://github.com/moorepants/anaconda-dee-config/actions/runs/10185441100

We have the same permissions issue with it.

I can build an installer without any packages from conda forge, but I just have to go through an make a new construct.yaml that eliminates all packages from conda forge. Last year's approach included the anaconda package (as does the one I linked just above). Are you suggesting to build an installer only from defaults package but without using the anaconda package or to include it?

Note that I switched to the "conda forge only" approach because it has long been recommend to not mix conda forge packages with defaults packages by conda forge. Our users need packages only available on conda forge, so I went the "pure" conda forge route to ensure binary compatibility.

@marcoesters
Copy link
Contributor

I suggest to not use conda-forge as a channel for this test, yes. I'd also reduce the number of packages since we know which packages are problematic: the packages that install alembic or jupyter-kernel. That just makes debugging faster.

If that results in the same problem, I'm out of ideas. The workflow looks okay. The only difference I see is that you use PowerShell instead of bash (as Miniforge and Anaconda do), but that can't be the reason, can it?

@moorepants
Copy link
Author

I built an installer without conda-forge packages in the base environment. You can download here:

https://github.com/moorepants/anaconda-dee-config/actions/runs/10242877472

Note that my install action fails with:

Error: Unable to locate executable file: C:\Users\runneradmin\miniconda3\condabin\conda.bat. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also verify the file has a valid extension for an executable file.

I'm not sure why. I did not include the anaconda package.

@moorepants
Copy link
Author

PowerShell instead of bash (as Miniforge and Anaconda do), but that can't be the reason, can it?

I can test a build using bash to call the constructor related commands.

@marcoesters
Copy link
Contributor

I built an installer without conda-forge packages in the base environment. You can download here:

https://github.com/moorepants/anaconda-dee-config/actions/runs/10242877472

Note that my install action fails with:

Error: Unable to locate executable file: C:\Users\runneradmin\miniconda3\condabin\conda.bat. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also verify the file has a valid extension for an executable file.

I'm not sure why. I did not include the anaconda package.

Probably because the installation has failed. I'd run the GUI with the artifact and/or create an environment with your specs to see if there is a package conflict.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type::bug describes erroneous operation, use severity::* to classify the type
Projects
Status: 🆕 New
Development

No branches or pull requests

3 participants