v1.1.0 slower than v1.0.3 on CPU

[pablopla]

I've compared the transcription speed on AMD Ryzen 5950X CPU with and without batching. faster-whisper is running inside a single docker container on a 5 minutes mp4 file. The turbo model is cached locally.
v1.1.0 is few seconds slower without batching and batching doesn't improve the speed.

Results:

Version	Precision	Beam size	Time
v1.0.3 (cpu_threads=4)	int8	1	55s
v1.1.0 (cpu_threads=4)	int8	1	1m2s
v1.1.0 (cpu_threads=4, batch_size=4)	int8	1	55s

Without batching:

from faster_whisper import WhisperModel

model_size = "turbo"
model = WhisperModel(model_size, device="cpu", compute_type="int8", cpu_threads=4)
segments, info = model.transcribe("test.mp4", beam_size=1, vad_filter=True, task="transcribe")
print("Detected language '%s' with probability %f" % (info.language, info.language_probability))
for segment in segments:
    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))

With batching

from faster_whisper import WhisperModel, BatchedInferencePipeline

model_size = "turbo"
model = WhisperModel(model_size, device="cpu", compute_type="int8", cpu_threads=4)
batched_model = BatchedInferencePipeline(model=model)
segments, info = batched_model.transcribe("test.mp4", beam_size=1, vad_filter=True, task="transcribe", batch_size=4)
print("Detected language '%s' with probability %f" % (info.language, info.language_probability))
for segment in segments:
    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))