Visione's Capabilities

[MalikAhmed2]

I'm curious about Visione's abilities. Can Visione identify faces and logos in videos? Can I search for videos based on a specific face or logo? And, does Visione work with audio too, like turning speech into text?

[fabiocarrara]

Hi @MalikAhmed2 ,

right now, VISIONE's visual search is supported by the following global image descriptors: OpenCLIP, ALADIN, CLIP2Video, and DINOv2.
None of them are specifically designed to identify/recognize faces or logos.

For faces, you might be able to search for some celebrities and other public figures that ended up in CLIP's training set, but that's all... no custom face search.
For logos, you might also be able to search for custom ones using VISIONE's query-by-example powered by DINOv2, which performs reasonably well in instance retrieval tasks.

Concerning audio, it is not analyzed in VISIONE (yet).

[MalikAhmed2]

Is it possible to add different modules, such as face recognition, to VISIONE?

[fabiocarrara]

Yes, the analysis part is kind of modular (based on dockerized services, see visione/services/analysis) and thought for extension, but it might get tricky to integrate it with the indexing and UI modules. If you are willing to contribute with a PR, we can provide support on this.

[MalikAhmed2]

I'm eager to contribute in the future when my schedule allows. Keep up the excellent work!