[AMD][ROCm] Improve support of AMD

[k-artem]

The patch delivers several fixes for building issues for CUDA part of DeepSpeed library.
Percentage of passed unit tests improved(tested on RDNA hardware, gfx110x and gfx12x) Before:
collected 5298 items / 15 skipped
2773 failed, 862 passed, 1665 skipped, 13 errors
After:
collected 5851 items / 11 skipped
4187 failed, 1373 passed, 292 skipped, 10 errors

Regarding testing of fp_quantizer(DS_BUILD_FP_QUANTIZER) via tests/unit/ops/fp_quantizer/test_fp_quant.py, this test depends on QPyTorch which should be patched before run on AMD, please apply Tiiiger/QPyTorch#71

[k-artem]

@hwchen2017 kindly ask for review after fixed your comments.

[loadams]

@k-artem - is this ready for final review? @hwchen2017 - any remaining review requests?

[k-artem]

@loadams could you please help with continue review?

[jithunnair-amd]

@tjruwase @loadams Can you please help move this PR forward? I believe we have addressed all review comments. This PR significantly improves DeepSpeed functionality on AMD hardware.

Also, we discussed this a while ago, but I don't think we moved forward on it: how do we remove the DeepSpeed dependency on this inactive repo?:

Regarding testing of fp_quantizer(DS_BUILD_FP_QUANTIZER) via tests/unit/ops/fp_quantizer/test_fp_quant.py, this test depends on QPyTorch which should be patched before run on AMD, please apply Tiiiger/QPyTorch#71

Related issue: #7216

[sfc-gh-truwase]

@tjruwase @loadams Can you please help move this PR forward? I believe we have addressed all review comments. This PR significantly improves DeepSpeed functionality on AMD hardware.

@jithunnair-amd, yes I will focus on this PR.

[sfc-gh-truwase]

Also, we discussed this a while ago, but I don't think we moved forward on it: how do we remove the DeepSpeed dependency on this inactive repo?:

Apologies for this question hanging for so long. Since so much has changed over the past months, I think it might be worth having a chat on this.

[jithunnair-amd]

Also, we discussed this a while ago, but I don't think we moved forward on it: how do we remove the DeepSpeed dependency on this inactive repo?:

Apologies for this question hanging for so long. Since so much has changed over the past months, I think it might be worth having a chat on this.

Sure, would you like to discuss here, or on a different platform eg. email? The gist of it is that we aren't aware of any alternatives for QPyTorch, so creating a deepspeed fork is the next best option to make updates to it. Currently, this lib is only used in unit tests (test_quantized_linear_module.py and test_fp_quant.py).

[sfc-gh-truwase]

alternatives for QPyTorch, so creating a deepspeed fork is the next best option to make updates to it. Currently, this lib is only used in unit tests (test_quantized_linear_module.py and test_fp_quant.py).

Got it. Unfortunately, we lack bandwidth to maintain QPyTorch fork. Moreover, our roadmap is to streamline by deprecating features subject to bandwidth and community interests. Are you interested in maintaining such a fork?

[sfc-gh-truwase]

@k-artem can you please address the formatting issues?

[k-artem]

@k-artem can you please address the formatting issues?

hi @sfc-gh-truwase , I checked it, actually it looks like a CI issue

yapf.....................................................................Failed
- hook id: yapf
- exit code: 1

Traceback (most recent call last):
  File "/home/runner/.cache/pre-commit/repoi51ipx2f/py_env-python3.10/bin/yapf", line 3, in <module>
    from yapf import run_main
  File "/home/runner/.cache/pre-commit/repoi51ipx2f/py_env-python3.10/lib/python3.10/site-packages/yapf/__init__.py", line 41, in <module>
    from yapf.yapflib import yapf_api
  File "/home/runner/.cache/pre-commit/repoi51ipx2f/py_env-python3.10/lib/python3.10/site-packages/yapf/yapflib/yapf_api.py", line 39, in <module>
    from yapf.pyparser import pyparser
  File "/home/runner/.cache/pre-commit/repoi51ipx2f/py_env-python3.10/lib/python3.10/site-packages/yapf/pyparser/pyparser.py", line 44, in <module>
    from yapf.yapflib import format_token
  File "/home/runner/.cache/pre-commit/repoi51ipx2f/py_env-python3.10/lib/python3.10/site-packages/yapf/yapflib/format_token.py", line 23, in <module>
    from yapf.pytree import pytree_utils
  File "/home/runner/.cache/pre-commit/repoi51ipx2f/py_env-python3.10/lib/python3.10/site-packages/yapf/pytree/pytree_utils.py", line 30, in <module>
    from yapf_third_party._ylib2to3 import pygram
  File "/home/runner/.cache/pre-commit/repoi51ipx2f/py_env-python3.10/lib/python3.10/site-packages/yapf_third_party/_ylib2to3/pygram.py", line 39, in <module>
    pattern_grammar = driver.load_grammar(_PATTERN_GRAMMAR_FILE)
  File "/home/runner/.cache/pre-commit/repoi51ipx2f/py_env-python3.10/lib/python3.10/site-packages/yapf_third_party/_ylib2to3/pgen2/driver.py", line 252, in load_grammar
    g.load(gp)
  File "/home/runner/.cache/pre-commit/repoi51ipx2f/py_env-python3.10/lib/python3.10/site-packages/yapf_third_party/_ylib2to3/pgen2/grammar.py", line 95, in load
    d = pickle.load(f)
EOFError: Ran out of input