Thank you @salah-zaiem for this great work!

• I think the main question to discuss is how we would like to support and structure benchmarks in SpeechBrain.
  One idea is to put everything under speechbrain/recipes/benchmarks/SSL_Benchmark. Then we need to add a full section in our main speechbrain README.md and website that properly advertises the benchmarks we have.

• Currently, we are working on CL-MARS (the continual learning one) and MOABB (the EEG one) and we need some consistency. For CL-MASR (see CL-MASR #2033) things are a bit easier as here we have a single dataset and many baselines. In the SSL benchmark, instead, we have different models and different datasets/tasks. This creates a significant overlap with standard recipes, which I don't think we can avoid easily.

@salah-zaiem and @TParcollet do you have suggestions?

Thank you @salah-zaiem for this great work!

• I think the main question to discuss is how we would like to support and structure benchmarks in SpeechBrain.
  One idea is to put everything under speechbrain/recipes/benchmarks/SSL_Benchmark. Then we need to add a full section in our main speechbrain README.md and website that properly advertises the benchmarks we have.
• Currently, we are working on CL-MARS (the continual learning one) and MOABB (the EEG one) and we need some consistency. For CL-MASR (see CL-MASR #2033) things are a bit easier as here we have a single dataset and many baselines. In the SSL benchmark, instead, we have different models and different datasets/tasks. This creates a significant overlap with standard recipes, which I don't think we can avoid easily.

@salah-zaiem and @TParcollet do you have suggestions?

Putting everything under a Benchmark folder seems like a good idea.
There's a lot of overlap with a certain number of recipes yes. One first idea, is that I can generally replace the preparation scripts ("{dataset}_prepare.py") with simlinks I guess. The rest is not straightforward to remove. It is more or less equivalent to the addition of a train_with_wav2vec in the folders of the different recipes ( but it"s not because of the weighting ).
Another option, would be the SSL weighted feature extraction, something you can call in a compute_features, by adding it to "features.py". But it would make the use of the benchmark even more complicated. (even though it may be useful for other recipes)

Thank you @salah-zaiem. We discussed the role of benchmarks in the past speechbrain general meeting and we agree that benchmarks should go into an independent repository (speechbrain/benchmarks). For now, it is private, but I gave you access with writing permissions. Would you mind closing this PR and uploading your code there?

Thank you @salah-zaiem. We discussed the role of benchmarks in the past speechbrain general meeting and we agree that benchmarks should go into an independent repository (speechbrain/benchmarks). For now, it is private, but I gave you access with writing permissions. Would you mind closing this PR and uploading your code there?

I can close it, but I still need to get the WeightedSSLModel in SB to run the benchmark code. I kept this PR here only for that for the moment. Maybe I should just open a new PR ?