Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BugFix] Fix collectors with non tensors #2232

Merged
merged 7 commits into from
Jun 19, 2024
Merged

[BugFix] Fix collectors with non tensors #2232

merged 7 commits into from
Jun 19, 2024

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Jun 15, 2024

Based on pytorch/tensordict#817

Code to test:

from torchrl.envs import GymEnv, Transform, check_env_specs
from torchrl.data import NonTensorSpec, TensorSpec
from torchrl.collectors import SyncDataCollector
from tensordict import TensorDictBase, TensorDict, NonTensorData

class AddNontTensorData(Transform):
    def _call(self, tensordict: TensorDictBase) -> TensorDictBase:
        tensordict["nt"] = "a string!"
        return tensordict

    def _reset(
        self, tensordict: TensorDictBase, tensordict_reset: TensorDictBase
    ) -> TensorDictBase:
        return tensordict_reset.set("nt", NonTensorData("reset!"))

    def transform_observation_spec(self, observation_spec: TensorSpec) -> TensorSpec:
        observation_spec["nt"] = NonTensorSpec(shape=())
        return observation_spec

env = GymEnv("CartPole-v1").append_transform(AddNontTensorData())
print(env.rollout(3))
check_env_specs(env)

collector = SyncDataCollector(env, frames_per_batch=10)
for data in collector:
    print(data)
    break

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jun 15, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2232

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 13 Unrelated Failures

As of commit 93ee894 with merge base 45ab9de (image):

NEW FAILURES - The following jobs have failed:

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 15, 2024
@vmoens vmoens marked this pull request as draft June 15, 2024 07:09
This was linked to issues Jun 15, 2024
[BUG] NonTensorSpec calls NonTensorData with unexpected shape argument #2168
Closed
[BUG] NonTensorData disappears from environment after step_mdp() #2171
Closed
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 15, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 91. Improved: $\large\color{#35bf28}16$. Worsened: $\large\color{#d91a1a}1$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1197s 59.9147ms 16.6904 Ops/s 17.1821 Ops/s $\color{#d91a1a}-2.86\%$
test_sync 34.6752ms 31.2297ms 32.0208 Ops/s 29.2211 Ops/s $\textbf{\color{#35bf28}+9.58\%}$
test_async 57.6006ms 29.4117ms 34.0001 Ops/s 33.8806 Ops/s $\color{#35bf28}+0.35\%$
test_simple 0.3812s 0.3805s 2.6279 Ops/s 2.5637 Ops/s $\color{#35bf28}+2.50\%$
test_transformed 0.5301s 0.5284s 1.8924 Ops/s 1.8389 Ops/s $\color{#35bf28}+2.91\%$
test_serial 1.3199s 1.2620s 0.7924 Ops/s 0.7820 Ops/s $\color{#35bf28}+1.33\%$
test_parallel 1.1038s 1.0587s 0.9446 Ops/s 0.9023 Ops/s $\color{#35bf28}+4.68\%$
test_step_mdp_speed[True-True-True-True-True] 0.1810ms 21.2144μs 47.1378 KOps/s 46.3472 KOps/s $\color{#35bf28}+1.71\%$
test_step_mdp_speed[True-True-True-True-False] 44.9820μs 12.8937μs 77.5576 KOps/s 74.9329 KOps/s $\color{#35bf28}+3.50\%$
test_step_mdp_speed[True-True-True-False-True] 31.7300μs 12.5404μs 79.7422 KOps/s 77.8596 KOps/s $\color{#35bf28}+2.42\%$
test_step_mdp_speed[True-True-True-False-False] 26.9310μs 7.5838μs 131.8596 KOps/s 127.6285 KOps/s $\color{#35bf28}+3.32\%$
test_step_mdp_speed[True-True-False-True-True] 49.9840μs 22.4324μs 44.5783 KOps/s 42.8819 KOps/s $\color{#35bf28}+3.96\%$
test_step_mdp_speed[True-True-False-True-False] 34.9960μs 14.2282μs 70.2830 KOps/s 69.0741 KOps/s $\color{#35bf28}+1.75\%$
test_step_mdp_speed[True-True-False-False-True] 45.6360μs 13.7840μs 72.5478 KOps/s 71.1295 KOps/s $\color{#35bf28}+1.99\%$
test_step_mdp_speed[True-True-False-False-False] 61.8370μs 8.8576μs 112.8972 KOps/s 111.1495 KOps/s $\color{#35bf28}+1.57\%$
test_step_mdp_speed[True-False-True-True-True] 0.1485ms 24.2527μs 41.2324 KOps/s 40.8254 KOps/s $\color{#35bf28}+1.00\%$
test_step_mdp_speed[True-False-True-True-False] 39.6050μs 15.4364μs 64.7818 KOps/s 63.2101 KOps/s $\color{#35bf28}+2.49\%$
test_step_mdp_speed[True-False-True-False-True] 64.9420μs 13.6863μs 73.0657 KOps/s 71.7598 KOps/s $\color{#35bf28}+1.82\%$
test_step_mdp_speed[True-False-True-False-False] 29.5060μs 8.7831μs 113.8551 KOps/s 110.4712 KOps/s $\color{#35bf28}+3.06\%$
test_step_mdp_speed[True-False-False-True-True] 51.2060μs 24.8032μs 40.3173 KOps/s 39.1515 KOps/s $\color{#35bf28}+2.98\%$
test_step_mdp_speed[True-False-False-True-False] 42.3800μs 16.5716μs 60.3444 KOps/s 58.8433 KOps/s $\color{#35bf28}+2.55\%$
test_step_mdp_speed[True-False-False-False-True] 49.9240μs 14.7409μs 67.8383 KOps/s 65.6330 KOps/s $\color{#35bf28}+3.36\%$
test_step_mdp_speed[True-False-False-False-False] 29.0140μs 9.8601μs 101.4190 KOps/s 97.5129 KOps/s $\color{#35bf28}+4.01\%$
test_step_mdp_speed[False-True-True-True-True] 48.5720μs 23.6828μs 42.2247 KOps/s 40.8876 KOps/s $\color{#35bf28}+3.27\%$
test_step_mdp_speed[False-True-True-True-False] 51.1460μs 15.3954μs 64.9547 KOps/s 63.1917 KOps/s $\color{#35bf28}+2.79\%$
test_step_mdp_speed[False-True-True-False-True] 42.6200μs 15.8565μs 63.0657 KOps/s 60.6879 KOps/s $\color{#35bf28}+3.92\%$
test_step_mdp_speed[False-True-True-False-False] 42.7110μs 9.8777μs 101.2386 KOps/s 96.7759 KOps/s $\color{#35bf28}+4.61\%$
test_step_mdp_speed[False-True-False-True-True] 63.2090μs 24.7560μs 40.3942 KOps/s 38.9342 KOps/s $\color{#35bf28}+3.75\%$
test_step_mdp_speed[False-True-False-True-False] 48.7920μs 16.4478μs 60.7983 KOps/s 58.3334 KOps/s $\color{#35bf28}+4.23\%$
test_step_mdp_speed[False-True-False-False-True] 42.5300μs 16.9282μs 59.0729 KOps/s 56.5972 KOps/s $\color{#35bf28}+4.37\%$
test_step_mdp_speed[False-True-False-False-False] 35.4770μs 11.1115μs 89.9965 KOps/s 86.7695 KOps/s $\color{#35bf28}+3.72\%$
test_step_mdp_speed[False-False-True-True-True] 61.1950μs 25.9229μs 38.5759 KOps/s 36.4807 KOps/s $\textbf{\color{#35bf28}+5.74\%}$
test_step_mdp_speed[False-False-True-True-False] 42.9410μs 17.7467μs 56.3486 KOps/s 53.5613 KOps/s $\textbf{\color{#35bf28}+5.20\%}$
test_step_mdp_speed[False-False-True-False-True] 41.2270μs 16.8635μs 59.2997 KOps/s 55.7363 KOps/s $\textbf{\color{#35bf28}+6.39\%}$
test_step_mdp_speed[False-False-True-False-False] 40.2250μs 11.1137μs 89.9793 KOps/s 85.6723 KOps/s $\textbf{\color{#35bf28}+5.03\%}$
test_step_mdp_speed[False-False-False-True-True] 40.8770μs 27.6879μs 36.1169 KOps/s 34.4935 KOps/s $\color{#35bf28}+4.71\%$
test_step_mdp_speed[False-False-False-True-False] 48.7210μs 18.8989μs 52.9132 KOps/s 50.5287 KOps/s $\color{#35bf28}+4.72\%$
test_step_mdp_speed[False-False-False-False-True] 69.1910μs 18.0220μs 55.4876 KOps/s 52.9456 KOps/s $\color{#35bf28}+4.80\%$
test_step_mdp_speed[False-False-False-False-False] 45.7870μs 12.2768μs 81.4542 KOps/s 78.5968 KOps/s $\color{#35bf28}+3.64\%$
test_values[generalized_advantage_estimate-True-True] 9.6236ms 9.3866ms 106.5347 Ops/s 104.6781 Ops/s $\color{#35bf28}+1.77\%$
test_values[vec_generalized_advantage_estimate-True-True] 36.4431ms 35.2056ms 28.4046 Ops/s 28.3631 Ops/s $\color{#35bf28}+0.15\%$
test_values[td0_return_estimate-False-False] 0.1990ms 0.1777ms 5.6268 KOps/s 5.6890 KOps/s $\color{#d91a1a}-1.09\%$
test_values[td1_return_estimate-False-False] 24.3021ms 23.8104ms 41.9985 Ops/s 39.4520 Ops/s $\textbf{\color{#35bf28}+6.45\%}$
test_values[vec_td1_return_estimate-False-False] 48.0842ms 35.5334ms 28.1425 Ops/s 28.0282 Ops/s $\color{#35bf28}+0.41\%$
test_values[td_lambda_return_estimate-True-False] 37.4899ms 34.5470ms 28.9461 Ops/s 28.1503 Ops/s $\color{#35bf28}+2.83\%$
test_values[vec_td_lambda_return_estimate-True-False] 36.9595ms 35.4010ms 28.2478 Ops/s 28.0848 Ops/s $\color{#35bf28}+0.58\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 11.0091ms 8.2845ms 120.7075 Ops/s 116.6147 Ops/s $\color{#35bf28}+3.51\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.0918ms 1.8866ms 530.0530 Ops/s 504.3696 Ops/s $\textbf{\color{#35bf28}+5.09\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.6381ms 0.3597ms 2.7799 KOps/s 2.7587 KOps/s $\color{#35bf28}+0.77\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 46.6221ms 44.5474ms 22.4480 Ops/s 22.7859 Ops/s $\color{#d91a1a}-1.48\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 3.6909ms 3.0428ms 328.6477 Ops/s 326.3876 Ops/s $\color{#35bf28}+0.69\%$
test_dqn_speed 6.1135ms 1.3722ms 728.7340 Ops/s 738.8447 Ops/s $\color{#d91a1a}-1.37\%$
test_ddpg_speed 4.3684ms 2.9096ms 343.6876 Ops/s 349.3017 Ops/s $\color{#d91a1a}-1.61\%$
test_sac_speed 9.0863ms 8.5108ms 117.4971 Ops/s 116.9543 Ops/s $\color{#35bf28}+0.46\%$
test_redq_speed 17.5801ms 13.2719ms 75.3471 Ops/s 74.0277 Ops/s $\color{#35bf28}+1.78\%$
test_redq_deprec_speed 93.0242ms 14.5549ms 68.7056 Ops/s 71.6346 Ops/s $\color{#d91a1a}-4.09\%$
test_td3_speed 8.6189ms 8.4218ms 118.7388 Ops/s 117.4807 Ops/s $\color{#35bf28}+1.07\%$
test_cql_speed 38.0164ms 36.6791ms 27.2635 Ops/s 26.9322 Ops/s $\color{#35bf28}+1.23\%$
test_a2c_speed 8.7068ms 7.4825ms 133.6452 Ops/s 130.7672 Ops/s $\color{#35bf28}+2.20\%$
test_ppo_speed 9.1705ms 7.7287ms 129.3881 Ops/s 125.5260 Ops/s $\color{#35bf28}+3.08\%$
test_reinforce_speed 7.4473ms 6.6457ms 150.4723 Ops/s 148.5387 Ops/s $\color{#35bf28}+1.30\%$
test_iql_speed 38.7926ms 33.3692ms 29.9678 Ops/s 30.1265 Ops/s $\color{#d91a1a}-0.53\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.6952ms 3.4633ms 288.7427 Ops/s 273.9643 Ops/s $\textbf{\color{#35bf28}+5.39\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.6032ms 0.4969ms 2.0126 KOps/s 1.9668 KOps/s $\color{#35bf28}+2.33\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7711ms 0.4770ms 2.0965 KOps/s 1.8237 KOps/s $\textbf{\color{#35bf28}+14.95\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.8531ms 3.4175ms 292.6157 Ops/s 276.8254 Ops/s $\textbf{\color{#35bf28}+5.70\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.6450ms 0.4903ms 2.0396 KOps/s 1.9805 KOps/s $\color{#35bf28}+2.98\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3.4137ms 0.4682ms 2.1361 KOps/s 2.0586 KOps/s $\color{#35bf28}+3.76\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 2.4323ms 1.7342ms 576.6411 Ops/s 573.0608 Ops/s $\color{#35bf28}+0.62\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 2.2514ms 1.6417ms 609.1365 Ops/s 599.8438 Ops/s $\color{#35bf28}+1.55\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.8969ms 3.6126ms 276.8126 Ops/s 255.8624 Ops/s $\textbf{\color{#35bf28}+8.19\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.0091ms 0.6158ms 1.6240 KOps/s 1.5507 KOps/s $\color{#35bf28}+4.73\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8839ms 0.5854ms 1.7081 KOps/s 1.6291 KOps/s $\color{#35bf28}+4.85\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.6931ms 3.4329ms 291.2979 Ops/s 267.2862 Ops/s $\textbf{\color{#35bf28}+8.98\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.9540ms 0.4984ms 2.0065 KOps/s 1.8996 KOps/s $\textbf{\color{#35bf28}+5.63\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6858ms 0.4760ms 2.1008 KOps/s 2.0049 KOps/s $\color{#35bf28}+4.78\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 4.4014ms 3.4439ms 290.3690 Ops/s 265.3226 Ops/s $\textbf{\color{#35bf28}+9.44\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.5845ms 0.4922ms 2.0318 KOps/s 1.9445 KOps/s $\color{#35bf28}+4.49\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3.5387ms 0.4710ms 2.1230 KOps/s 2.0410 KOps/s $\color{#35bf28}+4.01\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.1645ms 3.5857ms 278.8871 Ops/s 261.1065 Ops/s $\textbf{\color{#35bf28}+6.81\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.1401ms 0.6189ms 1.6159 KOps/s 1.5874 KOps/s $\color{#35bf28}+1.79\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.7198ms 0.5872ms 1.7030 KOps/s 1.6591 KOps/s $\color{#35bf28}+2.64\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1095s 7.8667ms 127.1180 Ops/s 117.7979 Ops/s $\textbf{\color{#35bf28}+7.91\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 14.3807ms 12.2965ms 81.3240 Ops/s 80.1550 Ops/s $\color{#35bf28}+1.46\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 1.1731ms 1.0345ms 966.6604 Ops/s 948.9799 Ops/s $\color{#35bf28}+1.86\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1024s 5.6818ms 176.0004 Ops/s 168.1489 Ops/s $\color{#35bf28}+4.67\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 14.7626ms 12.3160ms 81.1952 Ops/s 79.7102 Ops/s $\color{#35bf28}+1.86\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 4.3777ms 1.0919ms 915.7964 Ops/s 949.0716 Ops/s $\color{#d91a1a}-3.51\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1093s 5.9348ms 168.4989 Ops/s 166.5289 Ops/s $\color{#35bf28}+1.18\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 14.9479ms 12.4555ms 80.2859 Ops/s 78.9406 Ops/s $\color{#35bf28}+1.70\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 0.1054s 3.2904ms 303.9104 Ops/s 773.7237 Ops/s $\textbf{\color{#d91a1a}-60.72\%}$

@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 15, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 94. Improved: $\large\color{#35bf28}5$. Worsened: $\large\color{#d91a1a}5$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1218s 0.1187s 8.4229 Ops/s 8.3900 Ops/s $\color{#35bf28}+0.39\%$
test_sync 0.1050s 0.1044s 9.5758 Ops/s 9.5697 Ops/s $\color{#35bf28}+0.06\%$
test_async 0.2001s 78.8886ms 12.6761 Ops/s 10.5391 Ops/s $\textbf{\color{#35bf28}+20.28\%}$
test_single_pixels 0.1290s 0.1288s 7.7614 Ops/s 7.6403 Ops/s $\color{#35bf28}+1.58\%$
test_sync_pixels 85.2986ms 82.3905ms 12.1373 Ops/s 12.0448 Ops/s $\color{#35bf28}+0.77\%$
test_async_pixels 0.1616s 70.4018ms 14.2042 Ops/s 14.8632 Ops/s $\color{#d91a1a}-4.43\%$
test_simple 0.8346s 0.8230s 1.2151 Ops/s 1.1900 Ops/s $\color{#35bf28}+2.11\%$
test_transformed 1.1072s 1.1044s 0.9055 Ops/s 0.9243 Ops/s $\color{#d91a1a}-2.03\%$
test_serial 2.6652s 2.5900s 0.3861 Ops/s 0.3964 Ops/s $\color{#d91a1a}-2.60\%$
test_parallel 2.4009s 2.3516s 0.4252 Ops/s 0.4236 Ops/s $\color{#35bf28}+0.39\%$
test_step_mdp_speed[True-True-True-True-True] 0.3381ms 34.2032μs 29.2370 KOps/s 28.8159 KOps/s $\color{#35bf28}+1.46\%$
test_step_mdp_speed[True-True-True-True-False] 41.8500μs 20.4286μs 48.9510 KOps/s 46.7979 KOps/s $\color{#35bf28}+4.60\%$
test_step_mdp_speed[True-True-True-False-True] 0.2187ms 19.5117μs 51.2513 KOps/s 48.9775 KOps/s $\color{#35bf28}+4.64\%$
test_step_mdp_speed[True-True-True-False-False] 35.0110μs 11.5939μs 86.2525 KOps/s 81.1660 KOps/s $\textbf{\color{#35bf28}+6.27\%}$
test_step_mdp_speed[True-True-False-True-True] 0.2296ms 36.2653μs 27.5745 KOps/s 27.5980 KOps/s $\color{#d91a1a}-0.08\%$
test_step_mdp_speed[True-True-False-True-False] 0.2342ms 22.4250μs 44.5931 KOps/s 44.0284 KOps/s $\color{#35bf28}+1.28\%$
test_step_mdp_speed[True-True-False-False-True] 0.2423ms 21.5155μs 46.4781 KOps/s 45.4706 KOps/s $\color{#35bf28}+2.22\%$
test_step_mdp_speed[True-True-False-False-False] 32.7200μs 13.4783μs 74.1935 KOps/s 70.9571 KOps/s $\color{#35bf28}+4.56\%$
test_step_mdp_speed[True-False-True-True-True] 0.2298ms 37.9015μs 26.3842 KOps/s 25.6785 KOps/s $\color{#35bf28}+2.75\%$
test_step_mdp_speed[True-False-True-True-False] 0.2236ms 24.5146μs 40.7920 KOps/s 40.7158 KOps/s $\color{#35bf28}+0.19\%$
test_step_mdp_speed[True-False-True-False-True] 50.5010μs 21.4028μs 46.7228 KOps/s 45.5478 KOps/s $\color{#35bf28}+2.58\%$
test_step_mdp_speed[True-False-True-False-False] 0.2029ms 13.4874μs 74.1431 KOps/s 71.2936 KOps/s $\color{#35bf28}+4.00\%$
test_step_mdp_speed[True-False-False-True-True] 0.2279ms 39.7711μs 25.1439 KOps/s 25.0726 KOps/s $\color{#35bf28}+0.28\%$
test_step_mdp_speed[True-False-False-True-False] 0.2241ms 26.1144μs 38.2931 KOps/s 38.2618 KOps/s $\color{#35bf28}+0.08\%$
test_step_mdp_speed[True-False-False-False-True] 44.3200μs 23.0414μs 43.4002 KOps/s 41.9007 KOps/s $\color{#35bf28}+3.58\%$
test_step_mdp_speed[True-False-False-False-False] 0.2183ms 15.4078μs 64.9022 KOps/s 63.0852 KOps/s $\color{#35bf28}+2.88\%$
test_step_mdp_speed[False-True-True-True-True] 0.2392ms 37.7625μs 26.4813 KOps/s 26.2525 KOps/s $\color{#35bf28}+0.87\%$
test_step_mdp_speed[False-True-True-True-False] 0.2138ms 23.9013μs 41.8387 KOps/s 40.5504 KOps/s $\color{#35bf28}+3.18\%$
test_step_mdp_speed[False-True-True-False-True] 45.3610μs 25.3603μs 39.4318 KOps/s 38.3816 KOps/s $\color{#35bf28}+2.74\%$
test_step_mdp_speed[False-True-True-False-False] 0.3130ms 15.4756μs 64.6179 KOps/s 61.2072 KOps/s $\textbf{\color{#35bf28}+5.57\%}$
test_step_mdp_speed[False-True-False-True-True] 0.2275ms 39.7371μs 25.1654 KOps/s 25.4413 KOps/s $\color{#d91a1a}-1.08\%$
test_step_mdp_speed[False-True-False-True-False] 0.2116ms 25.8948μs 38.6178 KOps/s 38.1593 KOps/s $\color{#35bf28}+1.20\%$
test_step_mdp_speed[False-True-False-False-True] 47.5510μs 27.0290μs 36.9973 KOps/s 35.7006 KOps/s $\color{#35bf28}+3.63\%$
test_step_mdp_speed[False-True-False-False-False] 0.2324ms 17.1949μs 58.1569 KOps/s 56.8976 KOps/s $\color{#35bf28}+2.21\%$
test_step_mdp_speed[False-False-True-True-True] 0.2285ms 41.3631μs 24.1761 KOps/s 24.0361 KOps/s $\color{#35bf28}+0.58\%$
test_step_mdp_speed[False-False-True-True-False] 0.2515ms 28.3735μs 35.2441 KOps/s 35.1503 KOps/s $\color{#35bf28}+0.27\%$
test_step_mdp_speed[False-False-True-False-True] 56.3700μs 27.2492μs 36.6983 KOps/s 35.8404 KOps/s $\color{#35bf28}+2.39\%$
test_step_mdp_speed[False-False-True-False-False] 0.2040ms 17.2944μs 57.8222 KOps/s 56.5468 KOps/s $\color{#35bf28}+2.26\%$
test_step_mdp_speed[False-False-False-True-True] 66.0910μs 44.1486μs 22.6508 KOps/s 23.1760 KOps/s $\color{#d91a1a}-2.27\%$
test_step_mdp_speed[False-False-False-True-False] 0.2356ms 30.0558μs 33.2715 KOps/s 32.8983 KOps/s $\color{#35bf28}+1.13\%$
test_step_mdp_speed[False-False-False-False-True] 0.2219ms 28.3912μs 35.2222 KOps/s 33.9835 KOps/s $\color{#35bf28}+3.65\%$
test_step_mdp_speed[False-False-False-False-False] 37.8610μs 18.8918μs 52.9329 KOps/s 51.5065 KOps/s $\color{#35bf28}+2.77\%$
test_values[generalized_advantage_estimate-True-True] 26.7202ms 25.6999ms 38.9107 Ops/s 39.8816 Ops/s $\color{#d91a1a}-2.43\%$
test_values[vec_generalized_advantage_estimate-True-True] 88.3654ms 2.6700ms 374.5362 Ops/s 371.3613 Ops/s $\color{#35bf28}+0.85\%$
test_values[td0_return_estimate-False-False] 94.2210μs 68.3830μs 14.6235 KOps/s 14.9480 KOps/s $\color{#d91a1a}-2.17\%$
test_values[td1_return_estimate-False-False] 59.5483ms 57.6678ms 17.3407 Ops/s 17.8954 Ops/s $\color{#d91a1a}-3.10\%$
test_values[vec_td1_return_estimate-False-False] 1.2942ms 1.0844ms 922.1290 Ops/s 916.4126 Ops/s $\color{#35bf28}+0.62\%$
test_values[td_lambda_return_estimate-True-False] 94.2791ms 93.4101ms 10.7055 Ops/s 11.2988 Ops/s $\textbf{\color{#d91a1a}-5.25\%}$
test_values[vec_td_lambda_return_estimate-True-False] 1.2654ms 1.0827ms 923.6478 Ops/s 921.6671 Ops/s $\color{#35bf28}+0.21\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 26.3334ms 25.4097ms 39.3551 Ops/s 40.6598 Ops/s $\color{#d91a1a}-3.21\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 0.9769ms 0.7308ms 1.3684 KOps/s 1.3725 KOps/s $\color{#d91a1a}-0.30\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.7790ms 0.6975ms 1.4337 KOps/s 1.4784 KOps/s $\color{#d91a1a}-3.03\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.6444ms 1.4806ms 675.3799 Ops/s 676.6091 Ops/s $\color{#d91a1a}-0.18\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.7624ms 0.7058ms 1.4169 KOps/s 1.4547 KOps/s $\color{#d91a1a}-2.60\%$
test_dqn_speed 79.0521ms 1.6746ms 597.1682 Ops/s 674.7902 Ops/s $\textbf{\color{#d91a1a}-11.50\%}$
test_ddpg_speed 3.2739ms 3.0156ms 331.6055 Ops/s 330.1403 Ops/s $\color{#35bf28}+0.44\%$
test_sac_speed 9.3777ms 8.6241ms 115.9546 Ops/s 117.2239 Ops/s $\color{#d91a1a}-1.08\%$
test_redq_speed 11.5650ms 10.7312ms 93.1862 Ops/s 95.0754 Ops/s $\color{#d91a1a}-1.99\%$
test_redq_deprec_speed 12.4624ms 11.8675ms 84.2641 Ops/s 86.5079 Ops/s $\color{#d91a1a}-2.59\%$
test_td3_speed 17.8538ms 8.5927ms 116.3772 Ops/s 116.6601 Ops/s $\color{#d91a1a}-0.24\%$
test_cql_speed 27.7132ms 26.0367ms 38.4073 Ops/s 38.0314 Ops/s $\color{#35bf28}+0.99\%$
test_a2c_speed 5.8486ms 5.6421ms 177.2395 Ops/s 172.6937 Ops/s $\color{#35bf28}+2.63\%$
test_ppo_speed 6.5886ms 5.9839ms 167.1144 Ops/s 164.7370 Ops/s $\color{#35bf28}+1.44\%$
test_reinforce_speed 4.9831ms 4.6301ms 215.9785 Ops/s 211.5937 Ops/s $\color{#35bf28}+2.07\%$
test_iql_speed 20.0129ms 19.4027ms 51.5391 Ops/s 50.6160 Ops/s $\color{#35bf28}+1.82\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 5.0283ms 4.7189ms 211.9122 Ops/s 213.4861 Ops/s $\color{#d91a1a}-0.74\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7989ms 0.5920ms 1.6891 KOps/s 1.6912 KOps/s $\color{#d91a1a}-0.12\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 4.6153ms 0.5702ms 1.7537 KOps/s 1.7548 KOps/s $\color{#d91a1a}-0.06\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 5.0073ms 4.6962ms 212.9377 Ops/s 214.8037 Ops/s $\color{#d91a1a}-0.87\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.8943ms 0.5857ms 1.7074 KOps/s 1.6971 KOps/s $\color{#35bf28}+0.61\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 4.5820ms 0.5676ms 1.7617 KOps/s 1.7755 KOps/s $\color{#d91a1a}-0.78\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 2.3050ms 2.1149ms 472.8274 Ops/s 468.8320 Ops/s $\color{#35bf28}+0.85\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 5.9784ms 2.0157ms 496.1172 Ops/s 495.1814 Ops/s $\color{#35bf28}+0.19\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.0458ms 4.8538ms 206.0262 Ops/s 209.2571 Ops/s $\color{#d91a1a}-1.54\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.5480ms 0.7182ms 1.3923 KOps/s 1.3913 KOps/s $\color{#35bf28}+0.08\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8073ms 0.6918ms 1.4455 KOps/s 1.4316 KOps/s $\color{#35bf28}+0.97\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 4.9238ms 4.7146ms 212.1083 Ops/s 213.8317 Ops/s $\color{#d91a1a}-0.81\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.1899ms 0.5931ms 1.6861 KOps/s 1.6846 KOps/s $\color{#35bf28}+0.09\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.8781ms 0.5710ms 1.7513 KOps/s 1.7383 KOps/s $\color{#35bf28}+0.75\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 4.8761ms 4.6920ms 213.1290 Ops/s 214.2363 Ops/s $\color{#d91a1a}-0.52\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.8632ms 0.5864ms 1.7054 KOps/s 1.6970 KOps/s $\color{#35bf28}+0.49\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6808ms 0.5620ms 1.7792 KOps/s 1.7787 KOps/s $\color{#35bf28}+0.03\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.0576ms 4.8650ms 205.5513 Ops/s 208.5507 Ops/s $\color{#d91a1a}-1.44\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.3888ms 0.7175ms 1.3937 KOps/s 1.3849 KOps/s $\color{#35bf28}+0.64\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8725ms 0.6941ms 1.4407 KOps/s 1.4383 KOps/s $\color{#35bf28}+0.16\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1314s 7.5391ms 132.6419 Ops/s 130.1348 Ops/s $\color{#35bf28}+1.93\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 18.4571ms 16.1059ms 62.0891 Ops/s 60.7139 Ops/s $\color{#35bf28}+2.27\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 7.3767ms 1.4805ms 675.4520 Ops/s 782.3438 Ops/s $\textbf{\color{#d91a1a}-13.66\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1244s 7.4180ms 134.8070 Ops/s 133.9132 Ops/s $\color{#35bf28}+0.67\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 18.6336ms 15.9724ms 62.6081 Ops/s 53.1020 Ops/s $\textbf{\color{#35bf28}+17.90\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 6.7735ms 1.4277ms 700.4256 Ops/s 779.7386 Ops/s $\textbf{\color{#d91a1a}-10.17\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1265s 10.0361ms 99.6402 Ops/s 131.0914 Ops/s $\textbf{\color{#d91a1a}-23.99\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 18.7535ms 16.3139ms 61.2975 Ops/s 60.0942 Ops/s $\color{#35bf28}+2.00\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 2.5080ms 1.4916ms 670.4144 Ops/s 637.2536 Ops/s $\textbf{\color{#35bf28}+5.20\%}$

@vmoens vmoens changed the title [WIP] Fix collectors with non tensors [BugFix] Fix collectors with non tensors Jun 19, 2024
@vmoens vmoens marked this pull request as ready for review June 19, 2024 15:41
@vmoens vmoens added the bug Something isn't working label Jun 19, 2024
@vmoens vmoens merged commit 47a2627 into main Jun 19, 2024
16 of 21 checks passed
@vmoens vmoens deleted the fix-collector-nontensor branch June 19, 2024 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
bug Something isn't working CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
2 participants
@vmoens @facebook-github-bot