Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BugFix,Feature] Allow non-tensor data in envs #1944

Merged
merged 11 commits into from
Jun 19, 2024
Merged

[BugFix,Feature] Allow non-tensor data in envs #1944

merged 11 commits into from
Jun 19, 2024

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Feb 21, 2024 •
edited
Loading

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Feb 21, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1944

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 2 Unrelated Failures

As of commit e1b38b3 with merge base 47a2627 (image):

NEW FAILURES - The following jobs have failed:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 21, 2024
@vmoens vmoens linked an issue Feb 21, 2024 that may be closed by this pull request
[BUG] A dataclass in "next" doesn't get copied over in step_mdp #1919
Closed
3 tasks
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 21, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 91. Improved: $\large\color{#35bf28}4$. Worsened: $\large\color{#d91a1a}14$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1122s 59.1493ms 16.9064 Ops/s 18.0008 Ops/s $\textbf{\color{#d91a1a}-6.08\%}$
test_sync 32.0513ms 30.3505ms 32.9483 Ops/s 32.5993 Ops/s $\color{#35bf28}+1.07\%$
test_async 58.1409ms 29.1910ms 34.2571 Ops/s 34.5667 Ops/s $\color{#d91a1a}-0.90\%$
test_simple 0.3777s 0.3758s 2.6607 Ops/s 2.6787 Ops/s $\color{#d91a1a}-0.67\%$
test_transformed 0.5278s 0.5261s 1.9008 Ops/s 1.8906 Ops/s $\color{#35bf28}+0.54\%$
test_serial 1.3151s 1.2622s 0.7923 Ops/s 0.7929 Ops/s $\color{#d91a1a}-0.08\%$
test_parallel 1.1350s 1.0726s 0.9323 Ops/s 0.9191 Ops/s $\color{#35bf28}+1.44\%$
test_step_mdp_speed[True-True-True-True-True] 0.1680ms 22.4828μs 44.4784 KOps/s 47.0309 KOps/s $\textbf{\color{#d91a1a}-5.43\%}$
test_step_mdp_speed[True-True-True-True-False] 41.5770μs 13.0563μs 76.5914 KOps/s 77.5674 KOps/s $\color{#d91a1a}-1.26\%$
test_step_mdp_speed[True-True-True-False-True] 34.5440μs 13.2639μs 75.3925 KOps/s 78.7324 KOps/s $\color{#d91a1a}-4.24\%$
test_step_mdp_speed[True-True-True-False-False] 39.8840μs 7.7702μs 128.6968 KOps/s 129.0506 KOps/s $\color{#d91a1a}-0.27\%$
test_step_mdp_speed[True-True-False-True-True] 78.1840μs 23.9616μs 41.7334 KOps/s 43.9501 KOps/s $\textbf{\color{#d91a1a}-5.04\%}$
test_step_mdp_speed[True-True-False-True-False] 43.1500μs 14.3943μs 69.4720 KOps/s 70.0694 KOps/s $\color{#d91a1a}-0.85\%$
test_step_mdp_speed[True-True-False-False-True] 45.9060μs 14.4713μs 69.1025 KOps/s 72.0249 KOps/s $\color{#d91a1a}-4.06\%$
test_step_mdp_speed[True-True-False-False-False] 31.1180μs 8.9049μs 112.2971 KOps/s 113.9121 KOps/s $\color{#d91a1a}-1.42\%$
test_step_mdp_speed[True-False-True-True-True] 78.7870μs 25.2083μs 39.6694 KOps/s 41.7428 KOps/s $\color{#d91a1a}-4.97\%$
test_step_mdp_speed[True-False-True-True-False] 44.3130μs 15.7902μs 63.3304 KOps/s 64.9630 KOps/s $\color{#d91a1a}-2.51\%$
test_step_mdp_speed[True-False-True-False-True] 39.7440μs 14.4310μs 69.2952 KOps/s 72.4592 KOps/s $\color{#d91a1a}-4.37\%$
test_step_mdp_speed[True-False-True-False-False] 41.2170μs 8.9816μs 111.3384 KOps/s 112.5698 KOps/s $\color{#d91a1a}-1.09\%$
test_step_mdp_speed[True-False-False-True-True] 64.2700μs 26.3262μs 37.9849 KOps/s 40.4049 KOps/s $\textbf{\color{#d91a1a}-5.99\%}$
test_step_mdp_speed[True-False-False-True-False] 61.1150μs 17.6407μs 56.6871 KOps/s 60.6153 KOps/s $\textbf{\color{#d91a1a}-6.48\%}$
test_step_mdp_speed[True-False-False-False-True] 43.9420μs 15.5294μs 64.3940 KOps/s 67.4531 KOps/s $\color{#d91a1a}-4.54\%$
test_step_mdp_speed[True-False-False-False-False] 35.9170μs 10.1703μs 98.3254 KOps/s 101.4436 KOps/s $\color{#d91a1a}-3.07\%$
test_step_mdp_speed[False-True-True-True-True] 81.8130μs 25.2752μs 39.5645 KOps/s 42.2252 KOps/s $\textbf{\color{#d91a1a}-6.30\%}$
test_step_mdp_speed[False-True-True-True-False] 40.7960μs 15.8787μs 62.9775 KOps/s 65.6854 KOps/s $\color{#d91a1a}-4.12\%$
test_step_mdp_speed[False-True-True-False-True] 44.7240μs 17.0488μs 58.6553 KOps/s 62.6270 KOps/s $\textbf{\color{#d91a1a}-6.34\%}$
test_step_mdp_speed[False-True-True-False-False] 32.0000μs 10.3489μs 96.6282 KOps/s 99.6596 KOps/s $\color{#d91a1a}-3.04\%$
test_step_mdp_speed[False-True-False-True-True] 53.9910μs 26.3855μs 37.8997 KOps/s 39.8213 KOps/s $\color{#d91a1a}-4.83\%$
test_step_mdp_speed[False-True-False-True-False] 46.6370μs 17.0404μs 58.6842 KOps/s 60.2214 KOps/s $\color{#d91a1a}-2.55\%$
test_step_mdp_speed[False-True-False-False-True] 45.9960μs 17.9424μs 55.7340 KOps/s 58.4326 KOps/s $\color{#d91a1a}-4.62\%$
test_step_mdp_speed[False-True-False-False-False] 44.4430μs 11.3715μs 87.9389 KOps/s 89.3722 KOps/s $\color{#d91a1a}-1.60\%$
test_step_mdp_speed[False-False-True-True-True] 55.8850μs 27.7643μs 36.0175 KOps/s 38.5992 KOps/s $\textbf{\color{#d91a1a}-6.69\%}$
test_step_mdp_speed[False-False-True-True-False] 42.2480μs 18.1913μs 54.9714 KOps/s 56.3148 KOps/s $\color{#d91a1a}-2.39\%$
test_step_mdp_speed[False-False-True-False-True] 92.8640μs 17.9594μs 55.6812 KOps/s 58.9765 KOps/s $\textbf{\color{#d91a1a}-5.59\%}$
test_step_mdp_speed[False-False-True-False-False] 39.3340μs 11.3554μs 88.0637 KOps/s 90.4949 KOps/s $\color{#d91a1a}-2.69\%$
test_step_mdp_speed[False-False-False-True-True] 54.5520μs 29.5704μs 33.8176 KOps/s 35.9358 KOps/s $\textbf{\color{#d91a1a}-5.89\%}$
test_step_mdp_speed[False-False-False-True-False] 42.8900μs 19.2844μs 51.8553 KOps/s 53.5318 KOps/s $\color{#d91a1a}-3.13\%$
test_step_mdp_speed[False-False-False-False-True] 49.6430μs 18.8878μs 52.9442 KOps/s 55.4883 KOps/s $\color{#d91a1a}-4.59\%$
test_step_mdp_speed[False-False-False-False-False] 45.0240μs 12.5053μs 79.9662 KOps/s 82.0150 KOps/s $\color{#d91a1a}-2.50\%$
test_values[generalized_advantage_estimate-True-True] 9.7782ms 9.3801ms 106.6082 Ops/s 105.8255 Ops/s $\color{#35bf28}+0.74\%$
test_values[vec_generalized_advantage_estimate-True-True] 35.5799ms 33.0069ms 30.2967 Ops/s 30.0750 Ops/s $\color{#35bf28}+0.74\%$
test_values[td0_return_estimate-False-False] 0.2301ms 0.1645ms 6.0796 KOps/s 5.9374 KOps/s $\color{#35bf28}+2.40\%$
test_values[td1_return_estimate-False-False] 25.3872ms 23.4561ms 42.6329 Ops/s 41.3361 Ops/s $\color{#35bf28}+3.14\%$
test_values[vec_td1_return_estimate-False-False] 34.4385ms 33.0862ms 30.2240 Ops/s 30.0359 Ops/s $\color{#35bf28}+0.63\%$
test_values[td_lambda_return_estimate-True-False] 34.2885ms 34.0787ms 29.3439 Ops/s 28.5945 Ops/s $\color{#35bf28}+2.62\%$
test_values[vec_td_lambda_return_estimate-True-False] 34.5332ms 33.1219ms 30.1915 Ops/s 30.0439 Ops/s $\color{#35bf28}+0.49\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 8.3540ms 8.1590ms 122.5638 Ops/s 120.2082 Ops/s $\color{#35bf28}+1.96\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.1529ms 1.7945ms 557.2660 Ops/s 506.3198 Ops/s $\textbf{\color{#35bf28}+10.06\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.4235ms 0.3492ms 2.8633 KOps/s 2.7722 KOps/s $\color{#35bf28}+3.29\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 46.1276ms 45.0026ms 22.2209 Ops/s 21.9477 Ops/s $\color{#35bf28}+1.25\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 3.9750ms 3.0233ms 330.7591 Ops/s 330.2966 Ops/s $\color{#35bf28}+0.14\%$
test_dqn_speed 6.0616ms 1.3572ms 736.8355 Ops/s 741.4725 Ops/s $\color{#d91a1a}-0.63\%$
test_ddpg_speed 3.5396ms 2.8635ms 349.2207 Ops/s 349.7033 Ops/s $\color{#d91a1a}-0.14\%$
test_sac_speed 8.8096ms 8.3617ms 119.5932 Ops/s 118.4106 Ops/s $\color{#35bf28}+1.00\%$
test_redq_speed 15.4516ms 13.2488ms 75.4787 Ops/s 75.4705 Ops/s $\color{#35bf28}+0.01\%$
test_redq_deprec_speed 16.4961ms 13.4117ms 74.5616 Ops/s 75.3876 Ops/s $\color{#d91a1a}-1.10\%$
test_td3_speed 16.6059ms 8.4536ms 118.2930 Ops/s 118.3687 Ops/s $\color{#d91a1a}-0.06\%$
test_cql_speed 37.9769ms 36.6997ms 27.2482 Ops/s 27.0457 Ops/s $\color{#35bf28}+0.75\%$
test_a2c_speed 8.1459ms 7.4991ms 133.3486 Ops/s 134.1456 Ops/s $\color{#d91a1a}-0.59\%$
test_ppo_speed 8.8627ms 7.7692ms 128.7134 Ops/s 129.3487 Ops/s $\color{#d91a1a}-0.49\%$
test_reinforce_speed 8.0189ms 6.7127ms 148.9706 Ops/s 150.2630 Ops/s $\color{#d91a1a}-0.86\%$
test_iql_speed 33.8972ms 32.7661ms 30.5193 Ops/s 30.1138 Ops/s $\color{#35bf28}+1.35\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 5.3507ms 3.5808ms 279.2704 Ops/s 281.3358 Ops/s $\color{#d91a1a}-0.73\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.9380ms 0.5033ms 1.9870 KOps/s 1.9930 KOps/s $\color{#d91a1a}-0.30\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6891ms 0.4767ms 2.0978 KOps/s 2.1057 KOps/s $\color{#d91a1a}-0.38\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 4.0866ms 3.5896ms 278.5835 Ops/s 292.2538 Ops/s $\color{#d91a1a}-4.68\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.9144ms 0.4943ms 2.0230 KOps/s 2.0275 KOps/s $\color{#d91a1a}-0.22\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.5982ms 0.4688ms 2.1330 KOps/s 2.1188 KOps/s $\color{#35bf28}+0.67\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 2.1252ms 1.7323ms 577.2764 Ops/s 575.5977 Ops/s $\color{#35bf28}+0.29\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 1.9167ms 1.6336ms 612.1593 Ops/s 607.6980 Ops/s $\color{#35bf28}+0.73\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.3909ms 3.6657ms 272.7991 Ops/s 281.6461 Ops/s $\color{#d91a1a}-3.14\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.1099s 0.6970ms 1.4347 KOps/s 1.6324 KOps/s $\textbf{\color{#d91a1a}-12.11\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8669ms 0.5925ms 1.6877 KOps/s 1.6964 KOps/s $\color{#d91a1a}-0.51\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.7807ms 3.6064ms 277.2852 Ops/s 287.4180 Ops/s $\color{#d91a1a}-3.53\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.0647ms 0.4995ms 2.0019 KOps/s 1.9836 KOps/s $\color{#35bf28}+0.93\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6377ms 0.4783ms 2.0907 KOps/s 2.0659 KOps/s $\color{#35bf28}+1.20\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 4.8510ms 3.5660ms 280.4284 Ops/s 284.6356 Ops/s $\color{#d91a1a}-1.48\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.6761ms 0.4963ms 2.0148 KOps/s 2.0205 KOps/s $\color{#d91a1a}-0.28\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3.8491ms 0.4802ms 2.0823 KOps/s 1.9152 KOps/s $\textbf{\color{#35bf28}+8.73\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.6089ms 3.7116ms 269.4277 Ops/s 279.1165 Ops/s $\color{#d91a1a}-3.47\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.2834ms 0.6131ms 1.6309 KOps/s 1.6237 KOps/s $\color{#35bf28}+0.44\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.7637ms 0.5918ms 1.6897 KOps/s 1.6995 KOps/s $\color{#d91a1a}-0.58\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1204s 6.0344ms 165.7179 Ops/s 174.9978 Ops/s $\textbf{\color{#d91a1a}-5.30\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 15.0335ms 12.6920ms 78.7898 Ops/s 69.1455 Ops/s $\textbf{\color{#35bf28}+13.95\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 1.1588ms 1.0487ms 953.5791 Ops/s 868.1816 Ops/s $\textbf{\color{#35bf28}+9.84\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1054s 7.7756ms 128.6067 Ops/s 176.3166 Ops/s $\textbf{\color{#d91a1a}-27.06\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 15.2617ms 12.6261ms 79.2011 Ops/s 80.8066 Ops/s $\color{#d91a1a}-1.99\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 4.6857ms 1.1118ms 899.4105 Ops/s 896.9326 Ops/s $\color{#35bf28}+0.28\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1261s 6.4052ms 156.1240 Ops/s 170.1338 Ops/s $\textbf{\color{#d91a1a}-8.23\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 15.6122ms 12.8678ms 77.7136 Ops/s 79.7931 Ops/s $\color{#d91a1a}-2.61\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 1.9723ms 1.2053ms 829.6367 Ops/s 826.2474 Ops/s $\color{#35bf28}+0.41\%$

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 21, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 94. Improved: $\large\color{#35bf28}3$. Worsened: $\large\color{#d91a1a}4$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1770s 0.1259s 7.9423 Ops/s 7.9836 Ops/s $\color{#d91a1a}-0.52\%$
test_sync 0.1037s 0.1013s 9.8711 Ops/s 9.5135 Ops/s $\color{#35bf28}+3.76\%$
test_async 0.1816s 91.4315ms 10.9371 Ops/s 12.6128 Ops/s $\textbf{\color{#d91a1a}-13.29\%}$
test_single_pixels 0.1309s 0.1300s 7.6946 Ops/s 7.6004 Ops/s $\color{#35bf28}+1.24\%$
test_sync_pixels 85.2294ms 83.4744ms 11.9797 Ops/s 11.9860 Ops/s $\color{#d91a1a}-0.05\%$
test_async_pixels 0.1553s 68.3391ms 14.6329 Ops/s 14.5060 Ops/s $\color{#35bf28}+0.87\%$
test_simple 0.8954s 0.8327s 1.2010 Ops/s 1.1981 Ops/s $\color{#35bf28}+0.24\%$
test_transformed 1.1519s 1.0917s 0.9160 Ops/s 0.9132 Ops/s $\color{#35bf28}+0.31\%$
test_serial 2.5781s 2.5178s 0.3972 Ops/s 0.3955 Ops/s $\color{#35bf28}+0.41\%$
test_parallel 2.4259s 2.3987s 0.4169 Ops/s 0.4225 Ops/s $\color{#d91a1a}-1.33\%$
test_step_mdp_speed[True-True-True-True-True] 0.1189ms 35.4163μs 28.2356 KOps/s 28.5744 KOps/s $\color{#d91a1a}-1.19\%$
test_step_mdp_speed[True-True-True-True-False] 39.6200μs 20.4286μs 48.9511 KOps/s 48.4404 KOps/s $\color{#35bf28}+1.05\%$
test_step_mdp_speed[True-True-True-False-True] 48.3710μs 20.0218μs 49.9456 KOps/s 50.4275 KOps/s $\color{#d91a1a}-0.96\%$
test_step_mdp_speed[True-True-True-False-False] 26.6510μs 11.5840μs 86.3258 KOps/s 87.3106 KOps/s $\color{#d91a1a}-1.13\%$
test_step_mdp_speed[True-True-False-True-True] 52.5810μs 36.7491μs 27.2116 KOps/s 27.4517 KOps/s $\color{#d91a1a}-0.87\%$
test_step_mdp_speed[True-True-False-True-False] 39.7100μs 21.9422μs 45.5744 KOps/s 44.2938 KOps/s $\color{#35bf28}+2.89\%$
test_step_mdp_speed[True-True-False-False-True] 40.1400μs 21.6856μs 46.1134 KOps/s 45.5837 KOps/s $\color{#35bf28}+1.16\%$
test_step_mdp_speed[True-True-False-False-False] 37.6600μs 13.4539μs 74.3281 KOps/s 74.0415 KOps/s $\color{#35bf28}+0.39\%$
test_step_mdp_speed[True-False-True-True-True] 60.9310μs 38.2492μs 26.1444 KOps/s 25.9293 KOps/s $\color{#35bf28}+0.83\%$
test_step_mdp_speed[True-False-True-True-False] 49.0500μs 24.3528μs 41.0631 KOps/s 40.3396 KOps/s $\color{#35bf28}+1.79\%$
test_step_mdp_speed[True-False-True-False-True] 41.2200μs 21.8872μs 45.6888 KOps/s 45.4865 KOps/s $\color{#35bf28}+0.44\%$
test_step_mdp_speed[True-False-True-False-False] 33.5610μs 13.5898μs 73.5847 KOps/s 73.6417 KOps/s $\color{#d91a1a}-0.08\%$
test_step_mdp_speed[True-False-False-True-True] 72.3910μs 40.8748μs 24.4650 KOps/s 24.7105 KOps/s $\color{#d91a1a}-0.99\%$
test_step_mdp_speed[True-False-False-True-False] 44.6010μs 26.1198μs 38.2851 KOps/s 37.5598 KOps/s $\color{#35bf28}+1.93\%$
test_step_mdp_speed[True-False-False-False-True] 63.0800μs 23.4038μs 42.7281 KOps/s 42.3609 KOps/s $\color{#35bf28}+0.87\%$
test_step_mdp_speed[True-False-False-False-False] 30.0710μs 15.2649μs 65.5099 KOps/s 64.7654 KOps/s $\color{#35bf28}+1.15\%$
test_step_mdp_speed[False-True-True-True-True] 60.4900μs 38.3883μs 26.0496 KOps/s 25.7671 KOps/s $\color{#35bf28}+1.10\%$
test_step_mdp_speed[False-True-True-True-False] 40.5100μs 24.1103μs 41.4761 KOps/s 40.2484 KOps/s $\color{#35bf28}+3.05\%$
test_step_mdp_speed[False-True-True-False-True] 51.2400μs 25.8827μs 38.6359 KOps/s 38.2734 KOps/s $\color{#35bf28}+0.95\%$
test_step_mdp_speed[False-True-True-False-False] 35.0710μs 15.6766μs 63.7893 KOps/s 63.8577 KOps/s $\color{#d91a1a}-0.11\%$
test_step_mdp_speed[False-True-False-True-True] 59.0410μs 40.7059μs 24.5665 KOps/s 24.9984 KOps/s $\color{#d91a1a}-1.73\%$
test_step_mdp_speed[False-True-False-True-False] 43.9510μs 26.3530μs 37.9464 KOps/s 37.8356 KOps/s $\color{#35bf28}+0.29\%$
test_step_mdp_speed[False-True-False-False-True] 47.5910μs 27.0564μs 36.9598 KOps/s 35.9601 KOps/s $\color{#35bf28}+2.78\%$
test_step_mdp_speed[False-True-False-False-False] 41.5310μs 17.1816μs 58.2017 KOps/s 56.6107 KOps/s $\color{#35bf28}+2.81\%$
test_step_mdp_speed[False-False-True-True-True] 63.9200μs 42.1609μs 23.7186 KOps/s 23.5813 KOps/s $\color{#35bf28}+0.58\%$
test_step_mdp_speed[False-False-True-True-False] 44.5200μs 28.0813μs 35.6109 KOps/s 35.1855 KOps/s $\color{#35bf28}+1.21\%$
test_step_mdp_speed[False-False-True-False-True] 44.6100μs 27.2735μs 36.6656 KOps/s 36.2865 KOps/s $\color{#35bf28}+1.04\%$
test_step_mdp_speed[False-False-True-False-False] 34.9900μs 17.4684μs 57.2462 KOps/s 56.6430 KOps/s $\color{#35bf28}+1.06\%$
test_step_mdp_speed[False-False-False-True-True] 64.4810μs 44.7923μs 22.3253 KOps/s 22.7639 KOps/s $\color{#d91a1a}-1.93\%$
test_step_mdp_speed[False-False-False-True-False] 70.7410μs 29.9939μs 33.3401 KOps/s 33.4632 KOps/s $\color{#d91a1a}-0.37\%$
test_step_mdp_speed[False-False-False-False-True] 50.9010μs 29.3214μs 34.1047 KOps/s 34.9499 KOps/s $\color{#d91a1a}-2.42\%$
test_step_mdp_speed[False-False-False-False-False] 43.3310μs 19.3312μs 51.7299 KOps/s 51.6678 KOps/s $\color{#35bf28}+0.12\%$
test_values[generalized_advantage_estimate-True-True] 25.4582ms 24.9178ms 40.1319 Ops/s 40.3242 Ops/s $\color{#d91a1a}-0.48\%$
test_values[vec_generalized_advantage_estimate-True-True] 89.6960ms 2.6935ms 371.2651 Ops/s 370.4466 Ops/s $\color{#35bf28}+0.22\%$
test_values[td0_return_estimate-False-False] 97.8510μs 67.1365μs 14.8950 KOps/s 14.5395 KOps/s $\color{#35bf28}+2.45\%$
test_values[td1_return_estimate-False-False] 56.0207ms 55.8090ms 17.9182 Ops/s 17.9836 Ops/s $\color{#d91a1a}-0.36\%$
test_values[vec_td1_return_estimate-False-False] 1.2893ms 1.0857ms 921.0314 Ops/s 915.3037 Ops/s $\color{#35bf28}+0.63\%$
test_values[td_lambda_return_estimate-True-False] 88.4348ms 88.0821ms 11.3531 Ops/s 10.8762 Ops/s $\color{#35bf28}+4.38\%$
test_values[vec_td_lambda_return_estimate-True-False] 1.2302ms 1.0797ms 926.1552 Ops/s 912.8916 Ops/s $\color{#35bf28}+1.45\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 25.2245ms 24.9884ms 40.0185 Ops/s 38.0778 Ops/s $\textbf{\color{#35bf28}+5.10\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 0.9782ms 0.7201ms 1.3887 KOps/s 1.3796 KOps/s $\color{#35bf28}+0.66\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.7602ms 0.6735ms 1.4847 KOps/s 1.4848 KOps/s $-0.01\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.5119ms 1.4695ms 680.5027 Ops/s 677.9308 Ops/s $\color{#35bf28}+0.38\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.7427ms 0.6871ms 1.4554 KOps/s 1.4493 KOps/s $\color{#35bf28}+0.42\%$
test_dqn_speed 1.9208ms 1.4968ms 668.1142 Ops/s 672.1315 Ops/s $\color{#d91a1a}-0.60\%$
test_ddpg_speed 3.3863ms 3.0204ms 331.0807 Ops/s 328.8797 Ops/s $\color{#35bf28}+0.67\%$
test_sac_speed 9.8753ms 8.6144ms 116.0851 Ops/s 117.6190 Ops/s $\color{#d91a1a}-1.30\%$
test_redq_speed 0.1029s 11.8393ms 84.4642 Ops/s 93.5939 Ops/s $\textbf{\color{#d91a1a}-9.75\%}$
test_redq_deprec_speed 12.1965ms 11.7331ms 85.2291 Ops/s 85.7994 Ops/s $\color{#d91a1a}-0.66\%$
test_td3_speed 8.5466ms 8.4281ms 118.6513 Ops/s 116.3634 Ops/s $\color{#35bf28}+1.97\%$
test_cql_speed 27.5974ms 25.9471ms 38.5400 Ops/s 38.4209 Ops/s $\color{#35bf28}+0.31\%$
test_a2c_speed 5.9872ms 5.6930ms 175.6542 Ops/s 176.2211 Ops/s $\color{#d91a1a}-0.32\%$
test_ppo_speed 6.2461ms 5.9892ms 166.9675 Ops/s 166.7157 Ops/s $\color{#35bf28}+0.15\%$
test_reinforce_speed 5.3407ms 4.6584ms 214.6642 Ops/s 214.5781 Ops/s $\color{#35bf28}+0.04\%$
test_iql_speed 20.6972ms 19.9985ms 50.0037 Ops/s 51.0056 Ops/s $\color{#d91a1a}-1.96\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 4.9010ms 4.7848ms 208.9933 Ops/s 214.1906 Ops/s $\color{#d91a1a}-2.43\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.1063s 0.6925ms 1.4441 KOps/s 1.6692 KOps/s $\textbf{\color{#d91a1a}-13.48\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7764ms 0.5773ms 1.7321 KOps/s 1.7389 KOps/s $\color{#d91a1a}-0.39\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 5.0540ms 4.7539ms 210.3517 Ops/s 214.7943 Ops/s $\color{#d91a1a}-2.07\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.3688ms 0.5973ms 1.6742 KOps/s 1.6800 KOps/s $\color{#d91a1a}-0.35\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7570ms 0.5697ms 1.7553 KOps/s 1.7373 KOps/s $\color{#35bf28}+1.04\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 3.3775ms 2.1940ms 455.7986 Ops/s 468.5341 Ops/s $\color{#d91a1a}-2.72\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 3.1038ms 2.0133ms 496.6937 Ops/s 493.1909 Ops/s $\color{#35bf28}+0.71\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 4.9189ms 4.7908ms 208.7321 Ops/s 209.7710 Ops/s $\color{#d91a1a}-0.50\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.8484ms 0.7233ms 1.3825 KOps/s 1.3768 KOps/s $\color{#35bf28}+0.42\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 4.8902ms 0.7072ms 1.4140 KOps/s 1.4097 KOps/s $\color{#35bf28}+0.31\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 4.8065ms 4.7628ms 209.9604 Ops/s 214.2303 Ops/s $\color{#d91a1a}-1.99\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7117ms 0.6004ms 1.6655 KOps/s 1.6638 KOps/s $\color{#35bf28}+0.10\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7961ms 0.5762ms 1.7356 KOps/s 1.7348 KOps/s $\color{#35bf28}+0.05\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 4.9010ms 4.7352ms 211.1848 Ops/s 216.7775 Ops/s $\color{#d91a1a}-2.58\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.3884ms 0.5949ms 1.6809 KOps/s 1.6851 KOps/s $\color{#d91a1a}-0.25\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7114ms 0.5705ms 1.7530 KOps/s 1.7340 KOps/s $\color{#35bf28}+1.09\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 4.9757ms 4.8157ms 207.6559 Ops/s 208.1632 Ops/s $\color{#d91a1a}-0.24\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.8762ms 0.7298ms 1.3703 KOps/s 1.3743 KOps/s $\color{#d91a1a}-0.29\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8799ms 0.7085ms 1.4115 KOps/s 1.4222 KOps/s $\color{#d91a1a}-0.75\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1239s 7.4851ms 133.5980 Ops/s 132.9035 Ops/s $\color{#35bf28}+0.52\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 18.4577ms 15.7739ms 63.3959 Ops/s 61.9385 Ops/s $\color{#35bf28}+2.35\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 2.3674ms 1.3039ms 766.9020 Ops/s 772.5510 Ops/s $\color{#d91a1a}-0.73\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1254s 9.7909ms 102.1353 Ops/s 134.0951 Ops/s $\textbf{\color{#d91a1a}-23.83\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 18.2607ms 15.8038ms 63.2758 Ops/s 53.9183 Ops/s $\textbf{\color{#35bf28}+17.35\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 2.3708ms 1.2914ms 774.3330 Ops/s 713.7742 Ops/s $\textbf{\color{#35bf28}+8.48\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1236s 7.5790ms 131.9440 Ops/s 131.5491 Ops/s $\color{#35bf28}+0.30\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 18.5065ms 15.8843ms 62.9554 Ops/s 61.0796 Ops/s $\color{#35bf28}+3.07\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 7.5392ms 1.5830ms 631.7236 Ops/s 617.4548 Ops/s $\color{#35bf28}+2.31\%$

@vmoens vmoens added the enhancement New feature or request label Feb 21, 2024
@vmoens vmoens linked an issue Jun 11, 2024 that may be closed by this pull request
[BUG] NonTensorData disappears from environment after step_mdp() #2171
Closed
3 tasks
@vmoens
Merge remote-tracking branch 'origin/main' into fix-env-nontensor
78d910f 
# Conflicts:
#	test/mocking_classes.py
#	test/test_env.py
#	torchrl/data/tensor_specs.py
#	torchrl/envs/batched_envs.py
#	torchrl/envs/utils.py
@vmoens
Copy link
Contributor Author

vmoens commented Jun 11, 2024

@maximilianigl @Michael-C-Strobel I'm going back to this
For a single env it is easy to patch.
For batched envs there are two ways of doing this:

  • We used to rely entirely on buffers in batched envs to share data across processes. We can't do that generically with non-tensor data atm. So I was thinking to share the non-tensor data separately through the pipes and the rest on the buffers. This is awfully complex but it could work
  • The simpler idea could be to prevent users from using buffers altogether. Envs with non-tensor data will be slow to be executed in parallel envs, but that's going to be the price to pay to get them running.

I guess my question is this: how much do you guys care about parallel envs runtime when it comes to envs with non tensor data?

@vmoens
Copy link
Contributor Author

vmoens commented Jun 11, 2024

I fixed both options (with and without buffers) and all are incorporated in the tests.
There may still be some incompatibility with collectors, but I think this should belong to a separate PR.

@maximilianigl
Copy link

For my project I've hacked together an "encoder"/"decoder" step that encodes non-tensor data into ints and stores the values in a dictionary. That allowed me to use buffers and just "decode" the data in places I need it.
Regarding your question: If I still had the problem, I'd care about runtime. But not sure how much complexity that justifies.

@Michael-C-Strobel
Copy link

I think parallel envs will end up being pretty important to us. In our case our environment initialization instantiates an object from another package but neither the environment nor agent modifies it (the agent just needs access to some of its methods/attributed).

For us, a constant NonTensorSpec would probably work and I suspect wouldn’t cause any issues for the collector or replay buffer.

@vmoens
Copy link
Contributor Author

vmoens commented Jun 11, 2024

Ok so I think that now runtime should be ok-ish (at least not drastically slower)
Next steps after this PR are collectors and buffers.

@vmoens vmoens merged commit 038a615 into main Jun 19, 2024
46 of 52 checks passed
@vmoens vmoens deleted the fix-env-nontensor branch June 19, 2024 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. enhancement New feature or request
Projects
None yet
Milestone
No milestone
4 participants
@vmoens @maximilianigl @Michael-C-Strobel @facebook-github-bot