Hello @sangeet2020,

Thanks for the PR.

Is it backwards compatible with torch 1.13?

Hello @Adel-Moumen,

So after quite a lot of experiments, it seems that the issue is not with the torch, but transformers.

Here are my observations on

• Python 3.10.6
• torch==2.0.1+cu117

I would request to you to try these setting once, so to be assured.

Let me know if you need log files for each.

Thank You

I think we need to proceed with that and merge it after doing some final tests. My understanding is that the problem is not pytorch 2.0 but the version of the transformer used. In that case, I think we should also change the extra-requirement file in the recipes that use whisper (e.g, https://github.com/speechbrain/speechbrain/blob/develop/recipes/CommonVoice/ASR/transformer/extra_requirements.txt).

@poonehmousavi could you please proceed with it and do the last tests? When you think everything is fine we can merge it (we need it for CL-MASR as well).

@mravanelli Yes the problem is transformers, with the proposed solution, it works with the new version of transformers. I have tested it on a small dataset. I will test that more thoroughly while running whisper experiments fro common-voice.

Also, it looks like there are many places were transformer is added as an additional dependency (e.g., https://github.com/speechbrain/speechbrain/blob/develop/recipes/LibriSpeech/ASR/CTC/extra_requirements.txt). Do you think all these recipes might be affected somehow by the change done in transformers >= 0.30?

These transformer models are affected by this change:
CLAP
M-CTC-T
SpeechT5
TVLT
Whisper
apart from Whisper is there any other model that we used from the above list?
Also in general, it shouldn't cause any problem, in a whisper we don't use the transformer function for extracting features and use the OpenAI function because of the efficiency as far as I remembered but not super sure and that is the result of the inconsistencies.
@Adel-Moumen what was the reason that we don't directly use transformer whisperfeature_extractor?

Hello @poonehmousavi, the reason was that it was extremely slow. This is why we decided to use the OpenAI's implementation, because it was way waster.

Hello @poonehmousavi, the reason was that it was extremely slow. This is why we decided to use the OpenAI's implementation because it was way waster.

They change their function and basically use the touch audio function, I guess it might be faster now. One other solution is to use their own function instead of transposing.

Hello @poonehmousavi, the reason was that it was extremely slow. This is why we decided to use the OpenAI's implementation because it was way waster.

They change their function and basically use the touch audio function, I guess it might be faster now. One other solution is to use their own function instead of transposing.

Yes, I agree. Could you please try and compare?

Sure. I will try that.

I think the proposed solution is not the best solution. When trying to infer a Whisper model trained on SpeechBrain using transformers4.30.2, I get a size mismatch while loading the state dictionary for the "HuggingFaceWhisper" module of the SB pre-trained model. And this error comes when doing parameter transfer in.

The error is

    hparams["pretrainer"].load_collected()
  File "/netscratch/sagar/thesis/speechbrain/speechbrain/utils/parameter_transfer.py", line 312, in load_collected
    self._call_load_hooks(paramfiles, device)
  File "/netscratch/sagar/thesis/speechbrain/speechbrain/utils/parameter_transfer.py", line 329, in _call_load_hooks
    default_hook(obj, loadpath, device=device)
  File "/netscratch/sagar/thesis/speechbrain/speechbrain/utils/checkpoints.py", line 144, in torch_parameter_transfe
r
    incompatible_keys = obj.load_state_dict(
  File "/netscratch/sagar/thesis/sb_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load
_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for HuggingFaceWhisper:
        size mismatch for _mel_filters: copying a param with shape torch.Size([80, 201]) from checkpoint, the shape 
in current model is torch.Size([201, 80]).

I have tested it on a small dataset.

@poonehmousavi
Could you try loading the intermediate checkpoint (only the whisper.ckpt) and do inference, while maintaining the same transformer version?

@poonehmousavi
Could you try loading the intermediate checkpoint (only the whisper.ckpt) and do inference, while maintaining the same transformer version?

For sure it won't work, because it tries to load the whisper-checkpoint trained with an older version of transformers which saves the _mel_filters and it caused the mismatch. I am going to update the checkpoints soon using new versions.

No even whisper checkpoint trained on latest transformer version doesn't work for me.

in
I will take a look to check what is the problem.

I did a push the solution implemented by @lucadellalib in the CL_MASR benchmark. This solution is more backward compatible.
However, the models that we have on HF only work with the old version of transformers as they have been trained in this way:

https://huggingface.co/speechbrain/whisper_rescuespeech
https://huggingface.co/speechbrain/asr-whisper-large-v2-commonvoice-fr
https://huggingface.co/speechbrain/asr-whisper-large-v2-commonvoice-fa
https://huggingface.co/speechbrain/asr-whisper-large-v2-commonvoice-sr
https://huggingface.co/speechbrain/asr-whisper-large-v2-commonvoice-mn
https://huggingface.co/speechbrain/asr-whisper-large-v2-commonvoice-ar
https://huggingface.co/speechbrain/asr-whisper-large-v2-commonvoice-hi

• @poonehmousavi, do you see a way to make these models compatible with the latest version of transformers? Since we are going to release a new version, we might want our model to work with the latest version. If there is not an easy fix for that, I guess to only option is retraining, right?

I would like to take this opportunity to bring attention to the poor performance I observed while using our Whisper interface.

For example, when using the French language (https://huggingface.co/speechbrain/asr-whisper-large-v2-commonvoice-fr), the following code snippet demonstrates the issue:

>>> from speechbrain.pretrained import WhisperASR
torchvision is not available - cannot save figures
>>> import speechbrain
>>> speechbrain.__file__
'/scratch/ravanelm/speechbrain_fix_whisper/speechbrain/__init__.py'
>>> asr_model = WhisperASR.from_hparams(source="speechbrain/asr-whisper-large-v2-commonvoice-fr", savedir="pretrained_models/asr-whisper-large-v2-commonvoice-fr")
Downloading (…)rocessor_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 185k/185k [00:00<00:00, 2.98MB/s]
Downloading (…)lve/main/config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.99k/1.99k [00:00<00:00, 11.5MB/s]
Downloading pytorch_model.bin: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6.17G/6.17G [00:55<00:00, 110MB/s]
speechbrain.lobes.models.huggingface_whisper - whisper encoder-decoder is frozen.
>>> asr_model.transcribe_file("/scratch/ravanelm/speechbrain_devjuly2/asr-whisper-large-v2-commonvoice-fr/example-fr.wav")
[['selon', 'se', 's', 'dw']]

The transcription output, [['selon', 'se', 's', 'dw']], does not accurately represent what the speaker is saying. It appears that the transcription is truncated.

The same issue occurs with the German language model:

>>> asr_model = WhisperASR.from_hparams(source="speechbrain/rescuespeech_whisper", savedir="pretrained_models/rescuespeech_whisper")
speechbrain.lobes.models.huggingface_whisper - whisper encoder-decoder is frozen.
>>> asr_model.transcribe_file("/scratch/ravanelm/speechbrain_devjuly2/whisper_rescuespeech/example_de.wav")
[['wenn', 'd', 'doff']]

Once again, the transcription [['wenn', 'd', 'doff']] is truncated.

These observations lead me to suspect that we may have a consistent issue across all the Whisper HF models we release. I believe there might be a problem with our interface that prevents it from performing the same computations during training and validation as in the fine-tuning recipes.

@poonehmousavi , any idea? This is pretty crucial as our Whisper interfaces appear to be functioning incorrectly.

@mravanelli @poonehmousavi I just pushed a better fix that should solve the remaining backward compatibility issues even with previously trained models

I have checked the proposed solution. It works with the new version and it is backward compatible with prev versions. The previously trained models can be used by this change as well, Therefore, I think we could merge this pull request.

I did some final tests and I confirm that everything is working and backward compatible. Thank you all for contributing to this fix!

I have tried the whisper interface for French and Arabic and I didn't observe the problem you mentioned. it is the result for example file on HF.

Thanks a lot @lucadellalib for making the fix and @poonehmousavi for doing the needed tests.
✌️