New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow overriding per-dim group options via _MeshEnv.set_dim_group_options #126599

Open

yifuwang wants to merge 16 commits into gh/yifuwang/85/base from gh/yifuwang/85/head

+35 −1

Contributor

yifuwang commented May 18, 2024 •

edited

Stack from ghstack (oldest at bottom):


          [WIP] _set_mesh_dim_group_options

d9b4fbf

[ghstack-poisoned]

This was referenced May 18, 2024

Introduce ProcessGroupCudaP2P #122163

Closed

Introduce cuda_p2p based fused_all_gather_matmul and fused_matmul_reduce_scatter #126597

Draft

pytorch-bot bot commented May 18, 2024 •

edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/126599

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (3 Unrelated Failures)

As of commit a96de27 with merge base 4afc5c7 ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

pull / linux-focal-py3.8-clang10 / test (dynamo, 1, 3, linux.2xlarge) (gh) (detected as infra flaky with no log or failing log classifier)

UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:

pull / linux-focal-cuda12.4-py3.10-gcc9-sm86 / test (default, 3, 5, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) ()
test_foreach.py::TestForeachCUDA::test_parity__foreach_neg_fastpath_inplace_cuda_bool
pull / linux-focal-cuda12.4-py3.10-gcc9-sm86 / test (default, 4, 5, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) ()
inductor/test_efficient_conv_bn_eval.py::EfficientConvBNEvalCudaTests::test_basic_cuda

This comment was automatically generated by Dr. CI and updates every 15 minutes.

yifuwang mentioned this pull request

Introduce Inductor passes to micro-pipeline all-gather-matmul and matmul-reduce-scatter in certain cases #126598

Open

pytorch-bot bot added the oncall: distributed label

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

c2db8f4

ghstack-source-id: 3affd43fbf799cc2888289225d36347409d5c3dc
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

98c7a7c

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

8a6b914

ghstack-source-id: b20313fee3f27ad5104290d6a5a44fa4e52d30aa
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

7a690f9

ghstack-source-id: 36f90cd502058de8b4278a5f6eb42c638502d9f0
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

841c6a8

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

d7d8d23

ghstack-source-id: ff49b96912ec614b91df08ec4478f5c4bd881cd6
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

1c708fe

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang mentioned this pull request

Introduce cuda_p2p based fused_all_gather_matmul and fused_matmul_reduce_scatter #126634

Closed

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

703dbf7

ghstack-source-id: ff49b96912ec614b91df08ec4478f5c4bd881cd6
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

7e27bfc

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

7a8aa94

ghstack-source-id: 5b2dae763f7d1810f08467f839c64a20e872623c
Pull Request resolved: #126599

wanchaol reviewed

View reviewed changes

torch/distributed/device_mesh.py Outdated Show resolved Hide resolved

torch/distributed/device_mesh.py Outdated Show resolved Hide resolved

torch/distributed/device_mesh.py Show resolved Hide resolved


          Update on "[WIP] _set_mesh_dim_group_options"

8f4ef90

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

842f99a

ghstack-source-id: 651c4bf12ced6104c72eea1c4719d8e16475fdae
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

95d7aaf

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

e93f4c5

ghstack-source-id: 651c4bf12ced6104c72eea1c4719d8e16475fdae
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

80a104d

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

37c5e91

ghstack-source-id: 8ab9ab99138eeb16a242fb63d33e06ed77344e9a
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

0b6a4c8

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

3f42b55

ghstack-source-id: dfb90df6f9eab666c9e65d673b74bafee2736dfd
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

5ed025f

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

6f9c45e

ghstack-source-id: 0324a1947296112e37469d64138cffe6285ee644
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

a4f3d34

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

This was referenced May 29, 2024

force_stride_order on fused_all_gather_matmul/fused_matmul_reduce_scatter's operands to avoid a copy due to layout transformation #127454

Open

Improve the scheduling for fused_matmul_reduce_scatter #127455

Open


          Update on "Allow overriding per-dim group options via _MeshEnv.set_di…

1a03d53

…m_group_options"

[ghstack-poisoned]

yifuwang changed the title ~~[WIP] _set_mesh_dim_group_options~~ Allow overriding per-dim group options via _MeshEnv.set_dim_group_options


          Update on "Allow overriding per-dim group options via _MeshEnv.set_di…

e2f94b3

…m_group_options"

[ghstack-poisoned]

yifuwang requested a review from wanchaol

May 29, 2024 23:02

yifuwang marked this pull request as ready for review

May 29, 2024 23:02

yifuwang mentioned this pull request

Add profiler annotation for fused_all_gather_matmul and fused_matmul_reduce_scatter #127556

Open

yifuwang added 2 commits

May 30, 2024 11:52


          Update on "Allow overriding per-dim group options via _MeshEnv.set_di…

f7a72dd

…m_group_options"

[ghstack-poisoned]


          Update on "Allow overriding per-dim group options via _MeshEnv.set_di…

a96de27

…m_group_options"

[ghstack-poisoned]

yifuwang mentioned this pull request

Draft

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment