Allow assignment cache to be saved to and loaded from datadir

[pvanheus]

• Support download assignment cache to datadir and using from datadir
• Switch to using pip for all installs (thanks for tip from Wolfgang Maier)
• Change logic for finding constellations files to use latest version

[wm75]

Very cool work @pvanheus !

[wm75]

One general remark: I know you didn't touch this in this PR, but I'm somewhat unhappy with the earlier decision to accept a --datadir only if the pango data in it is newer than the package-level one.

I think if a user specifies --datadir, they are fully responsible for its contents. If they want to use an older version of data, let them do so. In fact, for Galaxy the current version check creates some extra effort because with data managers we want to be able to enforce use of an old data version for reproducibility of prior jobs. Right now we're cheating by prepending the datadir version with a "v" to make it appear more recent to the check:
https://github.com/galaxyproject/tools-iuc/blob/bfc2dbccaec2ebeaf235ac833edbcead2758d792/data_managers/data_manager_pangolearn/data_manager/pangolearn_dm.py#L77
;)

With the changes in this PR, it could also make sense to use whatever data components are found in --datadir even if there is no pangolin-data in it. E.g. a user might want to use only a datadir constellations, but go with the installed versions of everything else.

[AngieHinrichs]

Thanks so much for this @pvanheus!

I agree with @wm75's comments (yikes about prepending the "v"). I think it would be fine to simply accept what the user provides in datadir; also fine to warn the user if the datadir version is older than the installed version; but heavy-handed to ignore datadir if the version is equal to or less than the installed version, and not OK to ignore without warning the user, as I think is done for constellations unless I'm missing something.

[aineniamh]

I also agree, I can see cases where people would want to use historical data. My only concern is that there isn't always going to be backwards compatibility, but I guess if it finds the correct files then it'll work!

[pvanheus]

I also agree, I can see cases where people would want to use historical data. My only concern is that there isn't always going to be backwards compatibility, but I guess if it finds the correct files then it'll work!

I have added a flag (--use_old_datadir) to allow data to be used from datadir even if it is older than the pre-installed data. This lets people use custom versions if they like and otherwise get the latest version.

In terms of backwards compatibility, can a __schema_version__ be added to the various data modules (pangolin_data, constellations and pangolin_assignments) that can be incremented when the data schema changes?

[AngieHinrichs]

Thanks so much for the changes, @pvanheus, this is looking much cleaner now!

I have one more question and a couple nitpicky requests, but it's looking really good now with the more uniform version-finding and pip usage.

[AngieHinrichs]

This looks terrific now, thanks so much Peter for all of your work on this and Wolfgang for the helpful tips!

[aineniamh]

Thanks so much @pvanheus, @wm75 and @AngieHinrichs! I think this will be really useful for pangolin users. 😁