Batch inference？ #124
-
|
Is it possible to perform inference in batches? I have observed that the inference time for a single instance often requires at least 60 seconds. If the dataset is large, the cumulative waiting time becomes significantly prolonged. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
Hi @mengmeng233 no, that's not practically feasible - typical bottleneck is memory. In case you have only small proteins, you can just start several processes in parallel, don't expect significant speed up - GPU utilization is quite ok. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your reply! What if my GPU has sufficient memory? When I run small proteins, the GPU memory usage is less than 15G (num_diffn_timesteps=80). If the lengths of the small proteins in my dataset are similar (allowing for loading the same size model), can I perform batch inference? I am using an A100 with 80G of memory. If the memory is sufficient, can I perform batch inference just like during model training? |
Beta Was this translation helpful? Give feedback.
Hi @mengmeng233
no, that's not practically feasible - typical bottleneck is memory.
In case you have only small proteins, you can just start several processes in parallel, don't expect significant speed up - GPU utilization is quite ok.