Deallocate the memory which is not in demand

[KotomiHacker]

Is it possible to deallocate the memory when the ortValue is out of it's lifetime. it will save a lot of memory usage for large models?
I saw there is some information about lifetime of each ortValue tracked in the code.

[snnn]

We can divide the ortValues into 3 categories:

1. Model inputs/outputs
2. Model weights(const initializers)
3. Intermediate ort values. They are the outputs of some internal nodes.

The third one has the potential for reuse. You can search "enable_memory_reuse" to see how it is implemented in ONNX Runtime. Overall, this is an NP problem. There is no best answer.

[KotomiHacker]

@snnn
Yes, I went through the code and found it is implemented based on reuse count and freelist_. Any relation between reuse count and lifetime. When I analyze the unet model from stable diffusion, I found only 0.2 GB out of 3.6 GB is reused for the intermediate ort value. So for rest of the ort value, can we release their memory directly at the end of their lifetime, rather than putting them in the freelist, which most of them are not reused.