Skip to content

[AutoParallel] Reconstruct sharding mesh dimension inference logic - Part2 add sharding_mesh_dimension param #9382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Nov 20, 2024
Merged

[AutoParallel] Reconstruct sharding mesh dimension inference logic - Part2 add sharding_mesh_dimension param #9382

merged 5 commits into from
Nov 20, 2024

Conversation

@AndSonder
Copy link
Contributor

@AndSonder AndSonder commented Nov 7, 2024 •
edited
Loading

PR types

Bug fixes

PR changes

Others

Description

现在 sharding mesh dim 的推导的逻辑过于 hack，sharding 的 tensor axis 要根据 mp 的标记才能推导出来，后续将给这里的 sharding 推导逻辑重写，强制让用户传入 shard_mesh_dim，参数切分的维度默认给 0

本 PR 为 PaddleNLP 添加 sharding_mesh_dimension 参数，并在初始化 sharding func 的时候传入

相关 PR:

@paddle-bot
Copy link

paddle-bot bot commented Nov 7, 2024

Thanks for your contribution!

@paddle-bot paddle-bot bot assigned ZHUI Nov 7, 2024
@AndSonder AndSonder mentioned this pull request Nov 7, 2024
Merged
@AndSonder AndSonder changed the title [AutoParallel] fix error when using pure sharding [AutoParallel] Reconstruct sharding mesh dimension inference logic - Part1 add sharding_mesh_dimension param Nov 14, 2024
@AndSonder AndSonder changed the title [AutoParallel] Reconstruct sharding mesh dimension inference logic - Part1 add sharding_mesh_dimension param [AutoParallel] Reconstruct sharding mesh dimension inference logic - Part2 add sharding_mesh_dimension param Nov 14, 2024
@AndSonder AndSonder mentioned this pull request Nov 14, 2024
Merged
@codecov
Copy link

codecov bot commented Nov 14, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 50.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 53.05%. Comparing base (79bba4f) to head (1b4a3cc).
Report is 228 commits behind head on develop.

Files with missing lines Patch % Lines
paddlenlp/trainer/auto_trainer.py 0.00% 1 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff             @@
##           develop    #9382      +/-   ##
===========================================
+ Coverage    52.70%   53.05%   +0.34%     
===========================================
  Files          677      679       +2     
  Lines       109391   109303      -88     
===========================================
+ Hits         57658    57993     +335     
+ Misses       51733    51310     -423

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@AndSonder AndSonder mentioned this pull request Nov 14, 2024
Merged
wawltor
wawltor approved these changes Nov 20, 2024
View reviewed changes
Copy link
Contributor

@wawltor wawltor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wawltor wawltor merged commit bd1fd92 into PaddlePaddle:develop Nov 20, 2024
7 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wawltor wawltor wawltor approved these changes

Assignees

@ZHUI ZHUI

Labels

contributor

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@AndSonder @wawltor @ZHUI