Adding PyTorch support in data_utils #79
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… (either torch or tensorflow).
…acement. Have not tested
…rols whether to use tensorflow or pytorch as the backend
…ng to this_self inside the nested class IterableTorchDataset
… and that saving to NumPy still works.
…ng their outputs to NumPy
jerrylin96
approved these changes
Feb 19, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this pull request, I add the option of using PyTorch for the
data_utilsclass. This should allow users to get out a PyTorch dataset via theload_ncdata_with_generatorfunction.The following changes occurred:
data_utils.pywhere one can choose anml_backend(currently either PyTorch or Tensorflow, defaulting to Tensorflow) which will be used inload_ncdata_with_generatorand thus elsewhere in the class.setup.pywhere one can now set an environment variable to install PyTorch over Tensorflow. Note that one can runsetup.pynaively as usual and it will install Tensorflow.testing_data_utils_with_backends.pywhich is a small script to test that one can indeed use the backends correctly and that one can still save things to NumPy arrays.I was also informed that a new testing framework is coming -- I am happy to do another PR at that time with proper testing. For now, I have tested the logic in
data_utils.pywith the scripttesting_data_utils_with_backends.py(including a comparison of output arrays) and the changes tosetup.pyvia the use of different virtual environments.