Is a completely flattened latent token in THW the optimal solution?

[LinB203]

We are currently exploring relevant aspects in this regard. On one hand, a completely flattened latent token in THW requires a significant amount of memory to allow for the operation of attention blocks. On the other hand, we have observed that its convergence is not particularly fast.

Additionally, as a side note, OpenAI's Sora does not explicitly state that their transformer has visibility of every spatiotemporal token, which would be computationally expensive for generating 2k videos. Is it possible that only the encoder is spatiotemporal, while diffusion is 2+1D?

[ttttt12345]

orphan