Gender metadata

[mkaflowski]

Is it possible to have some metadata of gender of a speaker? It would be very helpful for making translations for languages that have different forms for genders.
For example:
English - I would like
Polish - Chciałbym (form man) / Chciałabym (for woman)

[glangford]

whisper does not do gender identification. FYI below in case these other efforts are helpful.

Gender Recognition by Voice and Speech Analysis
https://github.com/PrathamSolanki/gender-recognition-by-voice

Kaggle - Gender Recognition by Voice
https://www.kaggle.com/datasets/primaryobjects/voicegender

[JaesungHuh]

For gender identification, you could use the repo below.

https://github.com/JaesungHuh/voice-gender-classifier

[HubKing]

I transcribed French with the "large" model, and the speaker was obviously a female with very feminine voice. I came here because the transcription was in the male form. E.g., she said "désolée", but the transcription was "désolé". This is kind of problematic for French, and probably almost all European languages.

[whicks1]

YMMV but the whisper-at variant includes a secondary data structure that potentially contains gender info insofar as the audioset dataset has male/female feature labels. One could, I suppose, limit to those two features during the recognize step and then, based on the output (default 10 second blocks), run a post processing step to look for and transform gender specific content. I’ve run a handful of voices over that project and it can be a bit hit or miss though and I don’t recall lots of gendered voices being recognized beyond the standard speech level but it might be worth the shot for seemingly clear cut audio (mine was much less so)