Transcribing and translating offline with whisper

[hilderonny]

Hello everybody,

this week I created a python task for IPED which uses Whisper to transcribe audio and video files in various languages. It then translates the results into english. Finally the script uses Argos Translate to translate the english result into german for our collegues.
The results are attached to the evidence items as metadata and look like that:

audio:translation:language: ar  

audio:translation:original: سبحان الله والحمد لله ولا إله إلا الله والله أكبر ولا حول ولا قوة إلا بالله العلي العظيم  

audio:translation:en: Glory to Allah and praise be to Allah, and there is no god but Allah, and Allah is the Greatest, and there is no power except with Allah, the Greatest.

audio:translation:de: Herrlichkeit zu Allah und Lob sei zu Allah, und es gibt keinen Gott außer Allah, und Allah ist der Größte, und es gibt keine Macht außer mit Allah, dem Größten.

The script needs some additional installations and downloads but it rans completely offline which is important for our department.

If you are interested in this solution I could share the scripts and an installation instruction. Please tell me where and how.

Best regards,
Ronny

[lfcnassif]

Hello @hilderonny!

This is very useful and would be a great addition to the project! We have a recently opened improvement to integrate Whisper:
#1823

I was conducting tests to compare Whisper portuguese results to our current Wav2Vec2 portuguese transcription:
#1335 (comment)

I didn't finish it yet. But, actually, we don't have to wait for it and we can integrate Whisper and let users decide the algorithm choice (accuracy x speed).

My idea would be to integrate Whisper in a slight different way, similar to the way we run Wav2Vec2 today, so IPED's transcription cluster service could use Whisper too, not sure if you know about it:
https://github.com/sepinf-inc/IPED/wiki/User-Manual#remote-transcription-service-side

For this, we would need a script like below. We could read a configurable parameter from conf/AudioTranscriptConfig.txt to let user choose the Wav2Vec2 or Whisper script.
#1335 (comment)

At the moment, I think about integrating Faster-Whisper, not sure which Whisper implementation you are using...

About translation, that is also very useful and would be an awesome addition! We have an old stale feature request about this:
#1180

It can be very useful, I think not only for audios, but to translate documents too. So I think it could fit better as a separate task, independent of transcription, so users could translate documents independently of audio processing. Do you think it would be possible to split your script into 2, so translation would be a separate task? If yes and if you could submit a pull request, I would be happy to review and test!

PS: I know Whisper does translation to english automatically, I think it would be possible to return the english transcription version from the cluster service too, changing a bit the communication protocol.

Coincidently, today a colleague of mine shared this to me:
https://github.com/facebookresearch/seamless_communication

I don't know how it compares to Argos about accuracy x speed...

Thank you very much for sharing your work and for trying to help the project!

Best regards,
Luís Nassif

[hilderonny]

Hi @lfcnassif!

Thank you for your reply and your detailed descriptions. The reason why I use whisper is that I do not know the language which ist spoken in the audio files to process. When I am right, the default transcription functionality need to know the language of the file, right?

I also use faster-whisper because it is ... faster. But I had the problem with the parallel processing of different files. My machine has only 8 GB of GPU RAM and can handle only one transscription at once. But when IPED processes my cases it can happen that multiple audio files need to be processed the same time.

The solution was for me equally to your transcription server but not so elegant. My IPED task simply puts the audio files into a shared folder naming it with its MD5-hash. Then the task waits until a JSON file with the same name shows up in an output folder and processes it.

On the other side I have a number of machines which have a look into the shared folder and take a file as soon as it gets created. The machines have simple faster-whisper scripts (python) and transcribe/translate the file. After finishing they put the result JSON into the shared output folder.

Doing so I have a distributed cluster for transcribing without the need for a fully fledged server or internet solution (working offline in the department).

If you are interested I could post the scripts I made in the next month (still in alpha :) ). Simply tell me how you would like to get the code (attachment, fork, issue, ... )

Thank you for the link to the link to SeamlessM4T. It sounds very interesting and maybe it will be a good alternative.

Best regards,
Ronny Hildebrandt

[lfcnassif]

Hi @hilderonny!

When I am right, the default transcription functionality need to know the language of the file, right?

Yes, that's right.

I also use faster-whisper because it is ... faster.

Yes, although faster, I understand faster-whisper can return slightly worse results, that's why I was running comparison tests on #1335 (not finished yet).

The solution was for me equally to your transcription server but not so elegant. My IPED task simply puts the audio files into a shared folder naming it with its MD5-hash. Then the task waits until a JSON file with the same name shows up in an output folder and processes it.
On the other side I have a number of machines which have a look into the shared folder and take a file as soon as it gets created. The machines have simple faster-whisper scripts (python) and transcribe/translate the file. After finishing they put the result JSON into the shared output folder.
Doing so I have a distributed cluster for transcribing without the need for a fully fledged server or internet solution (working offline in the department).

This is a valid approach! For example, we have a similar solution to automate the whole IPED processing using multiple processing servers in our lab, they monitor a shared folder for new created DD/E01 disk images or Cellebrite *.ufdr reports. But about the transcription solution, since we wanted to provide an offline transcription service for all forensic labs country wide, we found the current cluster implementation using TCP sockets very useful and it is meeting our needs.

If you are interested I could post the scripts I made in the next month (still in alpha :) ). Simply tell me how you would like to get the code (attachment, fork, issue, ... )

At the moment we have a starting draft implementation to integrate whisper here: #1335 (comment)

I am personally interested in your translation code, if you could share it. As I said, I think it could be even more useful as an independent module. To integrate a new functionallity into the project, ideally a Pull Request is the best way, since it keeps authorship, makes reviewing and testing easier by multiple developers, etc. But I know creating a PR takes more time. Simply posting your scripts here should help, including the transcription part, I think it can be useful for users with a similar infrastructure setup as yours.

Thank you again for sharing your work!

Best regards,
Luis Nassif