use_deepspeed giving me trouble on Windows10/RTX 2070

[anthonyj-codestuff]

I just wiped out my tortoise setup and reinstalled everything. It doesn't take THAT long to generate a phrase (~10 min/350 chars), but I'd still like an extra boost if I can get it.

• Installed VS 2019 C++ x64/x86 build tools and Cuda Toolkit 12.1as per the instructions laid out here
• Installed PyTorch as per the website
  E:\Projects\tts>pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
  I skipped the bit about setting up a "pythonenv" environment because the base "tortoise" env is already set up for Python 3.9.20
• installed miniconda and activated the base env
• Ran the commands from the readme, except the package installation was changed to match my Cuda version
  (tortoise) E:\Projects\tts>conda install pytorch torchvision torchaudio pytorch-cuda=12.1 transformers -c pytorch -c nvidia
• BEFORE running setup.py install, I downloaded DeepSpeed 0.8.3, built a shiny new deepspeed-0.8.3+unknown-cp39-cp39-win_amd64.whl, and installed it from the "tortoise" environment
• Last, I ran the install script

NOW I can run python ...\do_tts.py and it works just fine, but if I try to run something like
(tortoise) E:\Projects\tts>python "..\Tortoise TTS\tortoise\do_tts.py" --text "test voice says a thing" --voice random --preset standard --output_path .\audio --candidates 1 --use_deepspeed true
I get the lengthy error

C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\huggingface_hub\file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\transformers\modeling_utils.py:460: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  return torch.load(checkpoint_file, map_location="cpu")
Some weights of the model checkpoint at jbetker/wav2vec2-large-robust-ft-libritts-voxpopuli were not used when initializing Wav2Vec2ForCTC: ['wav2vec2.encoder.pos_conv_embed.conv.weight_g', 'wav2vec2.encoder.pos_conv_embed.conv.weight_v']
- This IS expected if you are initializing Wav2Vec2ForCTC from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing Wav2Vec2ForCTC from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at jbetker/wav2vec2-large-robust-ft-libritts-voxpopuli and are newly initialized: ['wav2vec2.encoder.pos_conv_embed.conv.parametrizations.weight.original1', 'wav2vec2.encoder.pos_conv_embed.conv.parametrizations.weight.original0']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
E:\Projects\Tortoise TTS\tortoise\api.py:221: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  self.autoregressive.load_state_dict(torch.load(get_model_path('autoregressive.pth', models_dir)), strict=False)
C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\pydantic\_internal\_config.py:341: UserWarning: Valid config keys have changed in V2:
* 'allow_population_by_field_name' has been renamed to 'populate_by_name'
* 'validate_all' has been renamed to 'validate_default'
  warnings.warn(message, UserWarning)
C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\deepspeed\runtime\zero\linear.py:53: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  def forward(ctx, input, weight, bias=None):
C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\deepspeed\runtime\zero\linear.py:79: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
  def backward(ctx, grad_output):
C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\pydantic\_internal\_fields.py:161: UserWarning: Field "model_persistence_threshold" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\pydantic\_internal\_config.py:341: UserWarning: Valid config keys have changed in V2:
* 'validate_all' has been renamed to 'validate_default'
  warnings.warn(message, UserWarning)
Traceback (most recent call last):
  File "E:\Projects\Tortoise TTS\tortoise\do_tts.py", line 31, in <module>
    tts = TextToSpeech(models_dir=args.model_dir, use_deepspeed=args.use_deepspeed, kv_cache=args.kv_cache, half=args.half)
  File "E:\Projects\Tortoise TTS\tortoise\api.py", line 222, in __init__
    self.autoregressive.post_init_gpt2_config(use_deepspeed=use_deepspeed, kv_cache=kv_cache, half=self.half)
  File "C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\tortoise\models\autoregressive.py", line 380, in post_init_gpt2_config
    import deepspeed
  File "C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\deepspeed\__init__.py", line 15, in <module>
    from . import module_inject
  File "C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\deepspeed\module_inject\__init__.py", line 3, in <module>
    from .replace_module import replace_transformer_layer, revert_transformer_layer, ReplaceWithTensorSlicing, GroupQuantizer, generic_injection
  File "C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\deepspeed\module_inject\replace_module.py", line 803, in <module>
    from ..pipe import PipelineModule
  File "C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\deepspeed\pipe\__init__.py", line 3, in <module>
    from ..runtime.pipe import PipelineModule, LayerSpec, TiedLayerSpec
  File "C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\deepspeed\runtime\pipe\__init__.py", line 3, in <module>
    from .module import PipelineModule, LayerSpec, TiedLayerSpec
  File "C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\deepspeed\runtime\pipe\module.py", line 16, in <module>
    from ..activation_checkpointing import checkpointing
  File "C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\deepspeed\runtime\activation_checkpointing\checkpointing.py", line 25, in <module>
    from deepspeed.runtime.config import DeepSpeedConfig
  File "C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\deepspeed\runtime\config.py", line 30, in <module>
    from ..monitor.config import get_monitor_config
  File "C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\deepspeed\monitor\config.py", line 70, in <module>
    class DeepSpeedMonitorConfig(DeepSpeedConfigModel):
  File "C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\deepspeed\monitor\config.py", line 82, in DeepSpeedMonitorConfig
    def check_enabled(cls, values):
  File "C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\pydantic\deprecated\class_validators.py", line 234, in root_validator
    return root_validator()(*__args)  # type: ignore
  File "C:\Users\me\miniconda3\envs\tortoise\lib\site-packages\pydantic\deprecated\class_validators.py", line 240, in root_validator
    raise PydanticUserError(
pydantic.errors.PydanticUserError: If you use `@root_validator` with pre=False (the default) you MUST specify `skip_on_failure=True`. Note that `@root_validator` is deprecated and should be replaced with `@model_validator`.

For further information visit https://errors.pydantic.dev/2.8/u/root-validator-pre-skip

I'm not sure what the torch.load with weights_only=False thing is all about, but it started popping up after I updated the repo and rebuilt the "tortoise" env. It doesn't seem to affect the audio generation. Not sure how to approach the DeepSpeedMonitorConfig error, but based on the suggestion here, it seems like maybe uninstalling Pydantic before installing Tortoise could work, but that just led to issues with tokenizers not cooperating with the install

Am I missing something obvious?