Implement modern Image recognition model(s)

[TechupBusiness]

Describe the feature you'd like to request

The AI space is evolving quite fast and there are fantastic models coming out every few months. I would really like to see that this extension starts using modern AI models to improve tagging capabilities. Ideally make this extension model agnostic so that later everybody can use the model they like.

Describe the solution you'd like

A good candidate is Recognize Anything Model - RAM - on huggingface. Please see here https://huggingface.co/spaces/xinyu1205/recognize-anything
You can directly test it also on the website.

Describe alternatives you've considered

I dont see alternatives for proper AI models :)

[bonswouar]

I see the models haven't been updated for some years now, that would be really great to be able to implement some new ones!

As far as I understand this app uses tensorflow-js to run models.
I've just checked the available tfjs models (https://github.com/tensorflow/tfjs-models) and tensorflow models (https://www.kaggle.com/models), and it seems there haven't been any new/better ones in image classification / object detection categories unfortunately.

If someone has any suggestion, would be great!

I've found this kinda leaderboard on image classification : https://paperswithcode.com/sota/image-classification-on-imagenet
EfficientNetV2 is ranked 101. I've check a couple of "better" models with less than 1B params and couldn't find any tensorflow implementation.
I guess if someone really wants & know how to do it, converting a pytorch model to tensorflow (and then to tensorflowjs) would be doable : https://paperswithcode.com/sota/image-classification-on-imagenet

[bonswouar]

FYI I did some tests with the latest https://github.com/vikhyat/moondream , it's a really impressive model for it's size (1.8B parameters), but unfortunately it's a pretty general vision LLM, thus not made for image classification specifically.

I've tried to pass a list of keywords in the input but I didn't have great results, I guess long text inputs aren't well handled by such a small LLM.
Although testing some keywords one by one gives pretty excellent results!
For example I made this simple prompt; Does this picture contain "{keyword}"? Yes or No. No if you're not sure. and it seems to reliably gives accurate Yes or No responses. But it would be very time intensive to test all keywords one by one... A fine-tuned version for image classification could be pretty amazing though.

Anyway I'm not sure it's even possible to convert it to tensorflow model, but seeing the progress those kind of models have had recently the current Recognize model really feels old now :/

[marcelklehr]

seeing the progress those kind of models have had recently the current Recognize model really feels old now :/

I agree. Nextcloud GmbH is unlikely to spend efforts on revamping the models, sadly. I'm more than happy to guide any contributor willing to commit to implementing new stuff. (A good thing to work on would be #73 )

[bonswouar]

(A good thing to work on would be #73 )

That could be interesting, didn't notice this feature proposal before!
Although - and I don't know much about recent classification models (but it seems less active than general LLMs unfortunately, on the open-weight side at least) - it could be nice to aim for a specific model, it would be a good motivation for contributors to see how much better it could be (:
(Inference speed isn't really a criteria currently for me, even if I have ~10K pictures and running on the cpu it's fast enough for my use case - but an external device could mean a bigger model)

[bonswouar]

Kinda related, in recent news Mozilla is releasing an image recognition model for generating images alt text: https://hacks.mozilla.org/2024/05/experimenting-with-local-alt-text-generation-in-firefox-nightly/

The current model is apparently quite good and very small, and the training / model creation code is also open.
Looking at the dataset it was fined tuned on, it's obviously made for alt_text, but it could be quite easy to refactor this dataset to have a list of keywords instead.

[JartanFTW]

I can see a use-case in including harmful content keywords for automated flagging/removal purposes for organizations.

[TechupBusiness]

Wow Microsofts newest Florence 2 model looks awesome (Open Source). Just not sure how helpful it is, but it could extract a description and do an object detection.

[JartanFTW]

Meta's brand new open-source Segment Anything Model 2 may be very useful for object segmentation for recognition.