Symlinks in uploaded run tar prevents dearchiving

[TShapinsky]

No description provided.

[TShapinsky]

It is fairly easy to tall the python tarfile library to turn symlinks into their target files. However when you do this e+ can no longer find supporting files for running the simulation.

Additionally, to save space we heavily lean on symlinks to system python for our virtual environments.

[TShapinsky]

@axelstudios can you provide more context on what issue you're running into on windows when trying to dearchive runs? I wonder if there may be a different solution to just losing symlinks.

[axelstudios]

Sure - this is the error you'll see when attempting to extract the contents of a completed runs' .tar.gz archive:

That's because it contains serveral symlinks, which require admin rights on Windows to create (and in general you really don't want to be using symlinks on Windows unless you have a great reason):

When creating the tarballs it should be easy enough to have it archive actual files instead of symlinks: https://superuser.com/a/137391/1074025

[TShapinsky]

Yeah, I can dereference but it breaks the worker, and also when we have python venvs there are symbolic links to files which are not in the run directory.