Whisper finetunng common voice #1809

poonehmousavi · 2023-01-20T18:01:57Z

Add whisper finetuning recepies for Common-voice data for following languages

Hindi
Arabic
Persian
Serbian
Mongolian
French
Note: When using whisper large model, to improve memory usage during model recovery . you could use (see Avoid loading checkpoint parameters on the target device #1743)

…ssue

Adel-Moumen

Hello,

Many thanks for this PR! You did a really great job. :-)

Could you please remove the file environment.yaml ? I don't see any reasons to keep it.

Please see all the issues mentioned in the review.

Concerning the normalized_transcripts=True have you trained your models with it ? If so, it might be worth checking if it did impact your results... also, are you going to release the models on the Gdrive/HF?

Please fix the pre-commit.

Thanks again for your impressive works! :-)
Adel

recipes/CommonVoice/ASR/transformer/hparams/train_ar_hf_whisper.yaml

recipes/CommonVoice/ASR/transformer/hparams/train_fa_hf_whisper.yaml

recipes/CommonVoice/ASR/transformer/train_with_whisper.py

recipes/CommonVoice/common_voice_prepare.py

recipes/CommonVoice/ASR/transformer/hparams/train_fa_hf_whisper.yaml

recipes/CommonVoice/ASR/transformer/hparams/train_ar_hf_whisper.yaml

recipes/CommonVoice/ASR/transformer/README.md

recipes/CommonVoice/common_voice_prepare.py

recipes/CommonVoice/ASR/transformer/hparams/train_sr_hf_whisper.yaml

poonehmousavi · 2023-02-05T20:08:40Z

All changed are applied and pre-commit is tested

Adel-Moumen

Really neats what you are doing! Thanks again!

Please look at my comments and could you please take a look at the pre-commit fails? Read the following tutorial https://speechbrain.readthedocs.io/en/latest/contributing.html so that you know how to solve the issues.

Thanks!

recipes/CommonVoice/common_voice_prepare.py

recipes/CommonVoice/ASR/transformer/hparams/train_fr_hf_whisper.yaml

recipes/CommonVoice/ASR/transformer/hparams/train_fa_hf_whisper.yaml

recipes/CommonVoice/ASR/transformer/hparams/train_ar_hf_whisper.yaml

anautsch · 2023-02-15T16:03:35Z

@Adel-Moumen pointed me to your current error log.

CSV:  recipes/CommonVoice/self-supervised-learning/wav2vec2/common_voice_prepare.py
path: recipes/CommonVoice/self-supervised-learning/wav2vec2/common_voice_prepare.py

they look identical, yet, the error is for

    if not (os.path.exists(file.strip())):
        print(
            "\tERROR: The file %s listed in %s does not exist!"
            % (file, recipe_csvfile)
        )

which suggests the file isn't existing—yet, paths etc seem to match.

You can try this offline when you made a change with

pytest tests/consistency

note: this will consider also files you have not versioned by git but which are in your repo folders.

I don't think my post is particularly helpful, other than you don't need to push that much here. I'm looking into it, if I can find sth more helpful...

anautsch · 2023-02-15T16:44:55Z

tests/recipes/CommonVoice.csv

+ASR,CommonVoice,recipes/CommonVoice/ASR/transformer/train_with_whisper.py,recipes/CommonVoice/ASR/transformer/hparams/train_sr_hf_whisper.yaml,recipes/CommonVoice/ASR/transformer/common_voice_prepare.py,recipes/CommonVoice/ASR/transformer/README.md,https://drive.google.com/drive/folders/11NMzY0zV-NqJmPMyZfC3RtT64bYe-G_O?usp=sharing,,--data_folder=tests/samples/ASR/ --train_csv=tests/samples/annotation/ASR_train.csv --valid_csv=tests/samples/annotation/ASR_train.csv --test_csv=tests/samples/annotation/ASR_train.csv --number_of_epochs=1 --skip_prep=True,
+ASR,CommonVoice,recipes/CommonVoice/ASR/transformer/train_with_whisper.py,recipes/CommonVoice/ASR/transformer/hparams/train_mn_hf_whisper.yaml,recipes/CommonVoice/ASR/transformer/common_voice_prepare.py,recipes/CommonVoice/ASR/transformer/README.md,https://drive.google.com/drive/folders/11NMzY0zV-NqJmPMyZfC3RtT64bYe-G_O?usp=sharing,,--data_folder=tests/samples/ASR/ --train_csv=tests/samples/annotation/ASR_train.csv --valid_csv=tests/samples/annotation/ASR_train.csv --test_csv=tests/samples/annotation/ASR_train.csv --number_of_epochs=1 --skip_prep=True,
+ASR,CommonVoice,recipes/CommonVoice/ASR/transformer/train_with_whisper.py,recipes/CommonVoice/ASR/transformer/hparams/train_hi_hf_whisper.yaml,recipes/CommonVoice/ASR/transformer/common_voice_prepare.py,recipes/CommonVoice/ASR/transformer/README.md,https://drive.google.com/drive/folders/11NMzY0zV-NqJmPMyZfC3RtT64bYe-G_O?usp=sharing,,--data_folder=tests/samples/ASR/ --train_csv=tests/samples/annotation/ASR_train.csv --valid_csv=tests/samples/annotation/ASR_train.csv --test_csv=tests/samples/annotation/ASR_train.csv --number_of_epochs=1 --skip_prep=True,
 SSL,CommonVoice,recipes/CommonVoice/self-supervised-learning/wav2vec2/train_hf_wav2vec2.py,recipes/CommonVoice/self-supervised-learning/wav2vec2/hparams/wav2vec2_base.yaml,recipes/CommonVoice/self-supervised-learning/wav2vec2/common_voice_prepare.py,recipes/CommonVoice/self-supervised-learning/wav2vec2/README.md,,,--data_folder=tests/samples/ASR/ --train_csv=tests/samples/annotation/ASR_train.csv --valid_csv=tests/samples/annotation/ASR_train.csv --test_csv=tests/samples/annotation/ASR_train.csv --number_of_epochs=2 --skip_prep=True --d_model=128 --wav2vec2_folder=tests/tmp/wav2vec2_checkpoint,


@poonehmousavi this looks good btw, the line of concern did not change ... interesting

Yes, that is what makes it more confusing.

wondering if the github workflow got stuck in an odd state

Additionally, I did try pytest on my local repo and I didn't got that error and I didn't have any uncommitted change related to that files.

it's the workflow then...

could reproduce the error on my end

ll recipes/CommonVoice/self-supervised-learning/wav2vec2/common_voice_prepare.py recipes/CommonVoice/self-supervised-learning/wav2vec2/common_voice_prepare.py -> '../../common_voice_prepare.py'$'\n'

this is an invalid symlink

…oonehmousavi/speechbrain into whisper-finetunng-common-voice

Adel-Moumen · 2023-02-17T21:53:49Z

LGTM!

Many thanks for your great work. It has been a pleasure to review your PR.

poonehmousavi added 24 commits December 7, 2022 12:19

add recepie for whisper finetining on common-voice data

3ef35ae

add encoder-freeze optionto hparams +add extra dependecies

7125dc6

minor bug

536de68

set accented_letter to True for arabic and french

e6faeaf

minor fix

7f921bc

fix reading audio bug

ec1ecec

remove extra files

2ea8dc1

fix loss

581011a

fix loss in ar and fr hparams

6a73603

add enviroment

6fd540a

change test to greedu search instead of beam-serach to solve memory i…

7ebf8da

…ssue

add hparms for mnongolian, spanish, hindi, serbian, german

f9e5a9b

fix

f3158c8

fix memory issue+ add ja and fa

b67c8e3

add whisper-encoder_only for common_voice, fix minor bugs

5a3b864

fix bug for es

ab32fdd

add weighted sum version

2b5e79f

update readme file

cb90d19

modify en hparams -set accented letter to False

27edc2c

add test_only option

7253544

add final result table- final cleanig

6af3bdd

minor chage

e917485

minor change

523bd2e

fix type

add221d

Adel-Moumen self-assigned this Jan 23, 2023

Adel-Moumen requested changes Jan 23, 2023

View reviewed changes

poonehmousavi added 3 commits January 24, 2023 22:32

fix requested change in review

8e64623

remove enviroment file

d92896a

fix flag checking for test_olnly

0ca9ea6

Adel-Moumen requested changes Jan 25, 2023

View reviewed changes

poonehmousavi added 3 commits January 28, 2023 20:02

add comments

fcb7617

final refactoring

51b19c5

final refactoring

d1d3042

Adel-Moumen requested changes Feb 8, 2023

View reviewed changes

poonehmousavi added 10 commits February 11, 2023 20:42

minor refactoring(removing blank line,..)

9b0a4da

remove blank lines

a6ba1b8

minor refactor

f71c2b5

apply pre-commit changes

73e22c3

fix precommit bugs

2f4d83d

Merge branch 'speechbrain:develop' into whisper-finetunng-common-voice

b7815e4

add test, fix pre-commits error

46a41b6

fix CL test erros and precommit error for complicated method

4503074

fix link issue For CL workflow

0622477

test

bdca54f

anautsch reviewed Feb 15, 2023

View reviewed changes

poonehmousavi added 9 commits February 17, 2023 13:05

fix cl symlink bug

a136f63

Merge branch 'whisper-finetunng-common-voice' of https://github.com/p…

67008c7

…oonehmousavi/speechbrain into whisper-finetunng-common-voice

Merge branch 'speechbrain:develop' into whisper-finetunng-common-voice

b997e6f

remove whitespace

2034e26

fix for CL

fe0af9b

remove doc_str example for whisper interface

43ed4d1

fi readme file problem

a80c979

remove HF link from readme file

afbd5af

remove datasets from dpendencies

4198fff

Adel-Moumen approved these changes Feb 17, 2023

View reviewed changes

Adel-Moumen merged commit c723843 into speechbrain:develop Feb 17, 2023

poonehmousavi deleted the whisper-finetunng-common-voice branch July 29, 2024 16:59

Whisper finetunng common voice #1809

Whisper finetunng common voice #1809

Uh oh!

Conversation

poonehmousavi commented Jan 20, 2023

Uh oh!

Adel-Moumen left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

poonehmousavi commented Feb 5, 2023

Uh oh!

Adel-Moumen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anautsch commented Feb 15, 2023

Uh oh!

anautsch Feb 15, 2023

Choose a reason for hiding this comment

Uh oh!

poonehmousavi Feb 15, 2023

Choose a reason for hiding this comment

Uh oh!

anautsch Feb 15, 2023

Choose a reason for hiding this comment

Uh oh!

poonehmousavi Feb 15, 2023

Choose a reason for hiding this comment

Uh oh!

anautsch Feb 15, 2023

Choose a reason for hiding this comment

Uh oh!

anautsch Feb 15, 2023

Choose a reason for hiding this comment

Uh oh!

anautsch Feb 15, 2023

Choose a reason for hiding this comment

Uh oh!

Adel-Moumen commented Feb 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Adel-Moumen left a comment •

edited

Loading