Slower than openai whisper with my gpu

[tuotuoshao]

Env
device: A30
model version: large-v2
cuda version: cuda11

Measurement
for 57min audio
openai costs 450s
whisper-jax costs 400s

for 90s audio
openai costs 20s
whisper-jax costs 38s

for 2s audio
openai costs 1s
whisper-jax costs 5s

All my measurement are the second infefence.
Only if the audio’s duration is large enough, jax is faster but the increase is less then 10%. When processing short audio, jax is several times slower than openai.

Code config:
openai config:
model = whisper.load_model("large-v2")
whisper-jax config:
pipeline = FlaxWhisperPipline("openai/whisper-large-v2", dtype=jnp.float16, batch_size=16)

[ewwink]

try batch_size=1 or whatever number of GPU you have

[flexchar]

This is normal if your batch is small then there isn't room for a win. I wouldn't call it an issue. :)