Twitch.tv live transcription

[ggerganov]

We now have a sample script for transcribing radio livestream, such as BBC radio:

$ ./examples/livestream.sh http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/nonuk/sbr_low/ak/bbc_world_service.m3u8 10 small.en
[+] Transcribing stream with model 'small.en', step_s 10 (press Ctrl+C to stop):

Buffering audio. Please wait...

 dug out enough of this trench one. I thought they can't go back now but it's because it's um you know the car parks now a bit of a mess. So that day was...
 - Today was a very special day for me, very special. - Very quickly, there are remains found. How do you feel when you're told that there are (buzzer)
 in fact bones where you have been digging. Yeah, you know, again, it was remarkable because we haven't been digging for long. It was just...
 few hours and it was in trench one and we uncovered a lower leg bone or it looked to be a human lower leg bone and
 it was right in the area where I'd had this intuitive experience and it was right beside the letter R on the tarmac and all of that. So I was really...
 Excited. After 11 days of careful excavation they exhumed a skeleton. And that's when everyone got very interested very quickly because it's
 looked like substantial battle wounds particularly on the skull and the skeleton showed a spinal abnormality which we then discovered was
 scoliosis. And the curved spine, the scoliosis, was significant because some historical accounts had described the king as having unequal shoulders. And then after months of 10-
 the University of Leicester confirmed what everyone suspected. The skeleton was Richard III. Beyond reasonable doubt, the individual exhumed...
 Greyfriars in September 2012 is indeed Richard III, the last Plantagenet King of England. [Applause]
 BBC News with Marion Marshall. Scientists in England.

It would be cool if we could transcribe https://twitch.tv livestreams in a similar way.
I tried to understand how to read the livestream audio using ffmpeg but couldn't figure it out.
I think it shouldn't be very difficult to achieve.

[ggerganov]

This is now fully functional - thanks to @keyehzy !
Simply run:

# build whisper.cpp and download model
git clone https://github.com/ggerganov/whisper.cpp
cd whisper.cpp
make base.en

# transcribe some chess
./examples/twitch.sh https://www.twitch.tv/chess

Piping from streamlink url=https://www.twitch.tv/chess model=base.en step=10 threads=4
Buffering stream... (this should take 10 seconds)
Starting...
 Yeah, it's just c5. - And Bishop back. - I like what White's got going on here, a little f5 action.
 Okay, I was gonna say Queen D1, but fine. I'm just worried that when push comes to shove, White's king is gonna end up being
 Yeah, and Queenie1 is covered by the stupid knight on G2. Amazing defender.
 Oh my god. Oh my gosh, you got like 84 for white when you want it. So some Queen C4 stuff. Oh my gosh. You can- NO!

[ooobo]

Thanks for this @ggerganov, enjoying playing around with this and the livestream.sh example on live radio.
Removing --no-timestamps does seem to kill the transcribing, assuming this is because of the way the audio is piped out of ffmpeg? Curious if I can keep timestamps, or even global UNIX time?

[LiveRock]

I couldn't get it to work on the sample example. I am on a Mac Book Pro M1 Pro.
Here's the output:

./examples/twitch.sh https://www.twitch.tv/chess
Piping from streamlink url=https://www.twitch.tv/chess model=base.en step=10 threads=4
Buffering stream... (this should take 10 seconds)
Starting...

Nothing comes out after Starting..

[ggerganov]

Maybe the stream is offline now?
I just tried it using another online stream and it works:

./examples/twitch.sh https://www.twitch.tv/grandmastergauri
Piping from streamlink url=https://www.twitch.tv/grandmastergauri model=base.en step=10 threads=4
Buffering stream... (this should take 10 seconds)
Starting...
 It's just my playlist. Just random YouTube stuff.
 [Music] Bishop C5 threatening mate in one.
 (music)
 Just looking at it. (soft music)
 But, okay. So you're at the upper piece, but your king's wide open. (upbeat music) (music)
 He wants to go for mate ideas. So you're gonna go key here.

[ooobo]

Thanks for this @ggerganov, enjoying playing around with this and the livestream.sh example on live radio. Removing --no-timestamps does seem to kill the transcribing, assuming this is because of the way the audio is piped out of ffmpeg? Curious if I can keep timestamps, or even global UNIX time?

should add that timestamps now work as of latest version.

[nickswiss]

Sweet! exactly what I'm looking for.
I have stood up an ec2 with the following specs, apt isntalled ffmpeg and streamlink, as well as ran the make. I am able to manually process the wavs this script generates, but for whatever reason the live transcription is not happening.

Piping from streamlink url=https://www.twitch.tv/grandmastergauri model=base.en step=10 threads=4
Buffering stream... (this should take 10 seconds)
Starting...

It seems the video is being captured and written to tmp, but I am not receiving the live transcription in the output.

ec2 details:

PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"

[ggerganov]

And are you sure the stream that you are using is currently live? 😄
Edit: Ah nevermind, I saw you can process the wavs manually. Not sure then

[nickswiss]

Yup! Have checked with multiple at this point. No worries, can use the bits of the streamlink > ffmpeg to write something potentially, or just figure out what is going on here and report back. Thanks for the package!