Data type error when building SwiftTransformer for DistServe

[HalberdOfPineapple]

Hi all. I encountered some problems when building SwiftTransforemer as the dependency for DistServer (https://github.com/LLMServe/DistServe).

The versions of dependencies in my machine are like following:

gcc (Spack GCC) 12.3.0

(NVCC)
Cuda compilation tools, release 12.2, V12.2.128
Build cuda_12.2.r12.2/compiler.33053471_0

cmake version 3.27.7

And my machine is Ubuntu 20.04.1 LTS.

The installation instruction for DistServe includes the following commands:

git clone https://github.com/LLMServe/SwiftTransformer.git && cd SwiftTransformer && git submodule update --init --recursive
cmake -B build
cmake --build build -j$(nproc)

But when I execute cmake --build build -j$(nproc), it reports the following two errors caused by incorrect data types:
1):

[ 71%] Built target gmock_main
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc: In instantiation of 'void naiveGemmStridedBatched(cublasOperation_t, cublasOperation_t, int, int, int, T, const T*, long long int, const T*, long long int, T, T*, long long int, int) [with T = __half]':
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:120:27:   required from 'void CublasWrapperTestSuite_gemmStridedBatched_Test<gtest_TypeParam_>::TestBody() [with gtest_TypeParam_ = __half]'
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:70:1:   required from here
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:50:64: error: ambiguous overload for 'operator*' (operand types are 'const __half' and 'float')
   50 |                 Carray[batch * stride_c + i * ldc + j] = alpha * sum + beta * Carray[batch * stride_c + i * ldc + j];
      |                                                          ~~~~~~^~~~~
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:50:64: note: candidate: 'operator*(int, float)' (built-in)
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:50:64: note: candidate: 'operator*(long long unsigned int, float)' (built-in)
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:50:64: note: candidate: 'operator*(long long int, float)' (built-in)
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:50:64: note: candidate: 'operator*(long unsigned int, float)' (built-in)
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:50:64: note: candidate: 'operator*(long int, float)' (built-in)
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:50:64: note: candidate: 'operator*(unsigned int, float)' (built-in)
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:50:64: note: candidate: 'operator*(float, float)' (built-in)

This is caused by that the two operands alpha and sum cannot serve as compatible ones for operator *

2):

/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/../unittest_utils.h:93:45: error: call of overloaded 'fabs(__half)' is ambiguous
   93 |                                         fabs(answer[i]-reference[i]), fabs(answer[i]-reference[i])/fabs(reference[i]));
      |                                         ~~~~^~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/features.h:490,
                 from /appl/scibuilder-spack/aalto-rhel9-dev/2024-01-compilers/software/linux-rhel9-haswell/gcc-11.4.1/gcc-12.3.0-xh5vv5d/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../include/c++/12.3.0/x86_64-pc-linux-gnu/bits/os_defines.h:39,
                 from /appl/scibuilder-spack/aalto-rhel9-dev/2024-01-compilers/software/linux-rhel9-haswell/gcc-11.4.1/gcc-12.3.0-xh5vv5d/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../include/c++/12.3.0/x86_64-pc-linux-gnu/bits/c++config.h:655,
                 from /appl/scibuilder-spack/aalto-rhel9-dev/2024-01-compilers/software/linux-rhel9-haswell/gcc-11.4.1/gcc-12.3.0-xh5vv5d/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../include/c++/12.3.0/functional:48,
                 from /scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:1:
/usr/include/bits/mathcalls.h:162:1: note: candidate: 'double fabs(double)'
  162 | __MATHCALLX (fabs,, (_Mdouble_ __x), (__const__));
      | ^~~~~~~~~~~
In file included from /appl/scibuilder-spack/aalto-rhel9-dev/2024-01-compilers/software/linux-rhel9-haswell/gcc-11.4.1/gcc-12.3.0-xh5vv5d/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../include/c++/12.3.0/random:38,
                 from /scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:2:
/appl/scibuilder-spack/aalto-rhel9-dev/2024-01-compilers/software/linux-rhel9-haswell/gcc-11.4.1/gcc-12.3.0-xh5vv5d/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../include/c++/12.3.0/cmath:241:3: note: candidate: 'constexpr float std::fabs(float)'
  241 |   fabs(float __x)
      |   ^~~~
/appl/scibuilder-spack/aalto-rhel9-dev/2024-01-compilers/software/linux-rhel9-haswell/gcc-11.4.1/gcc-12.3.0-xh5vv5d/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../include/c++/12.3.0/cmath:245:3: note: candidate: 'constexpr long double std::fabs(long double)'
  245 |   fabs(long double __x)
      |   ^~~~
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/../unittest_utils.h:93:75: error: call of overloaded 'fabs(__half)' is ambiguous
   93 |                                         fabs(answer[i]-reference[i]), fabs(answer[i]-reference[i])/fabs(reference[i]));
      |                                                                       ~~~~^~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bits/mathcalls.h:162:1: note: candidate: 'double fabs(double)'
  162 | __MATHCALLX (fabs,, (_Mdouble_ __x), (__const__));
      | ^~~~~~~~~~~
/appl/scibuilder-spack/aalto-rhel9-dev/2024-01-compilers/software/linux-rhel9-haswell/gcc-11.4.1/gcc-12.3.0-xh5vv5d/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../include/c++/12.3.0/cmath:241:3: note: candidate: 'constexpr float std::fabs(float)'
  241 |   fabs(float __x)
      |   ^~~~
/appl/scibuilder-spack/aalto-rhel9-dev/2024-01-compilers/software/linux-rhel9-haswell/gcc-11.4.1/gcc-12.3.0-xh5vv5d/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../include/c++/12.3.0/cmath:245:3: note: candidate: 'constexpr long double std::fabs(long double)'
  245 |   fabs(long double __x)
      |   ^~~~

which is caused by that the value of answer[i]-reference[i] has type __half (more accurately template T as declared).

For these two problems, I made the following changes within my machine:

1. Change line 49 in .../SwiftTransformer/src/unittest/util/cublas_wrapper.cc into:

Carray[batch * stride_c + i * ldc + j] = alpha * static_cast<T>(sum) + beta * Carray[batch * stride_c + i * ldc + j];

2. Change line 93 in .../SwiftTransformer/src/unittest/util/../unittest_utils.h to be:

fabs(static_cast<float>(answer[i]-reference[i])), 
fabs(static_cast<float>(answer[i]-reference[i])) / fabs(static_cast<float>(reference[i])));

Now the build command can successfully be completed and the later pip install for DistServe can be done. But I am not sure whether such changes will result in extra overheads as changing __half to be float increases the bits required for the variable. (Sorry for my naiveness in this field).
If my modifications are reasonable, I can submit a new push request for that.

[lylcyl]

Hello, I also encountered some problems when building SwiftTransforemer as the dependency for DistServer (https://github.com/LLMServe/DistServe).
I would like to ask you if you have encountered the following error:

/home/infer/yilin/DistServe/SwiftTransformer/src/csrc/kernel/fused_decoding_stage_attention.cu(213): error: more than one conversion function from "half" to a built-in type applies:
function "__half::operator float() const" (declared at line 217 of /home/infer/anaconda/anaconda3/envs/Distserve/include/cuda_fp16.hpp)
function "__half::operator short() const" (declared at line 235 of /home/infer/anaconda/anaconda3/envs/Distserve/include/cuda_fp16.hpp)
function "__half::operator unsigned short() const" (declared at line 238 of /home/infer/anaconda/anaconda3/envs/Distserve/include/cuda_fp16.hpp)
function "__half::operator int() const" (declared at line 241 of /home/infer/anaconda/anaconda3/envs/Distserve/include/cuda_fp16.hpp)
function "__half::operator unsigned int() const" (declared at line 244 of /home/infer/anaconda/anaconda3/envs/Distserve/include/cuda_fp16.hpp)
function "__half::operator long long() const" (declared at line 247 of /home/infer/anaconda/anaconda3/envs/Distserve/include/cuda_fp16.hpp)
function "__half::operator unsigned long long() const" (declared at line 250 of /home/infer/anaconda/anaconda3/envs/Distserve/include/cuda_fp16.hpp)
function "__half::operator __nv_bool() const" (declared at line 254 of /home/infer/anaconda/anaconda3/envs/Distserve/include/cuda_fp16.hpp)
cur_result += (float)((T)attn_score[q_head_index][block_size_index] * v_block[block_size_index][hd_index]);
^
detected during:
instantiation of "void st::kernel::fusedDecodingStageAttentionKernel<T,Q_HEADS_PER_THREAD_BLOCK,HEAD_DIM,BLOCK_SIZE,THREAD_BLOCK_SIZE>(T *, const T *, T *, T *, float, const int64_t *, const int64_t *, const int64_t *, const int64_t *, int64_t, int64_t, int64_t) [with T=half, Q_HEADS_PER_THREAD_BLOCK=2L, HEAD_DIM=80L, BLOCK_SIZE=1L, THREAD_BLOCK_SIZE=256L]" at line 322
instantiation of "void st::kernel::fusedDecodingStageAttention(T *, const T *, T *, T *, float, const int64_t *, const int64_t *, int64_t, const int64_t *, const int64_t *, int64_t, int64_t, int64_t, int64_t, int64_t, int64_t, int64_t, int64_t) [with T=half]" at line 348

Error limit reached.
100 errors detected in the compilation of "/home/infer/yilin/DistServe/SwiftTransformer/src/csrc/kernel/fused_decoding_stage_attention.cu".
Compilation terminated.
make[2]: *** [src/csrc/kernel/CMakeFiles/kernel.dir/build.make:174: src/csrc/kernel/CMakeFiles/kernel.dir/fused_decoding_stage_attention.cu.o] Error 1
make[2]: *** [src/csrc/kernel/CMakeFiles/kernel.dir/build.make:160: src/csrc/kernel/CMakeFiles/kernel.dir/fused_context_stage_attention.cu.o] Error 1
[ 56%] Linking CXX static library ../../../lib/libgtest.a
[ 56%] Built target gtest
[ 57%] Building CXX object _deps/googletest-build/googletest/CMakeFiles/gtest_main.dir/src/gtest_main.cc.o
[ 58%] Building CXX object _deps/googletest-build/googlemock/CMakeFiles/gmock.dir/src/gmock-all.cc.o
[ 59%] Linking CXX static library ../../../lib/libgtest_main.a
[ 59%] Built target gtest_main
[ 60%] Building CXX object src/unittest/util/CMakeFiles/unittest_util.dir/cublas_wrapper.cc.o
[ 60%] Linking CXX static library ../../../lib/libgmock.a
[ 60%] Built target gmock
[ 61%] Building CXX object _deps/googletest-build/googlemock/CMakeFiles/gmock_main.dir/src/gmock_main.cc.o
[ 62%] Linking CXX executable ../../../bin/unittest_util
[ 62%] Built target unittest_util
[ 63%] Linking CXX static library ../../../lib/libgmock_main.a
[ 63%] Built target gmock_main
[ 64%] Linking CUDA device code CMakeFiles/py_nccl_utils.dir/cmake_device_link.o
[ 65%] Linking CXX static library libpy_nccl_utils.a
[ 65%] Built target py_nccl_utils
[ 66%] Linking CXX static library libpy_swapping.a
[ 66%] Built target py_swapping
[ 66%] Linking CXX static library libpy_block_migration.a
[ 66%] Built target py_block_migration
make[1]: *** [CMakeFiles/Makefile2:610: src/csrc/kernel/CMakeFiles/kernel.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
^Cnvcc error : 'cicc' died due to signal 2
make[2]: *** [src/csrc/kernel/CMakeFiles/xformers_autogen_impl.dir/build.make:202: src/csrc/kernel/CMakeFiles/xformers_autogen_impl.dir/xformers/xformers/csrc/attention/cuda/fmha/autogen/impl/cutlassB_f16_aligned_k128.cu.o] Error 2

[Liaukx]

Hi all. I encountered some problems when building SwiftTransforemer as the dependency for DistServer (https://github.com/LLMServe/DistServe).

The versions of dependencies in my machine are like following:

gcc (Spack GCC) 12.3.0

(NVCC)
Cuda compilation tools, release 12.2, V12.2.128
Build cuda_12.2.r12.2/compiler.33053471_0

cmake version 3.27.7

And my machine is Ubuntu 20.04.1 LTS.

The installation instruction for DistServe includes the following commands:

git clone https://github.com/LLMServe/SwiftTransformer.git && cd SwiftTransformer && git submodule update --init --recursive
cmake -B build
cmake --build build -j$(nproc)
But when I execute cmake --build build -j$(nproc), it reports the following two errors caused by incorrect data types: 1):

[ 71%] Built target gmock_main
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc: In instantiation of 'void naiveGemmStridedBatched(cublasOperation_t, cublasOperation_t, int, int, int, T, const T*, long long int, const T*, long long int, T, T*, long long int, int) [with T = __half]':
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:120:27:   required from 'void CublasWrapperTestSuite_gemmStridedBatched_Test<gtest_TypeParam_>::TestBody() [with gtest_TypeParam_ = __half]'
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:70:1:   required from here
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:50:64: error: ambiguous overload for 'operator*' (operand types are 'const __half' and 'float')
   50 |                 Carray[batch * stride_c + i * ldc + j] = alpha * sum + beta * Carray[batch * stride_c + i * ldc + j];
      |                                                          ~~~~~~^~~~~
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:50:64: note: candidate: 'operator*(int, float)' (built-in)
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:50:64: note: candidate: 'operator*(long long unsigned int, float)' (built-in)
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:50:64: note: candidate: 'operator*(long long int, float)' (built-in)
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:50:64: note: candidate: 'operator*(long unsigned int, float)' (built-in)
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:50:64: note: candidate: 'operator*(long int, float)' (built-in)
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:50:64: note: candidate: 'operator*(unsigned int, float)' (built-in)
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:50:64: note: candidate: 'operator*(float, float)' (built-in)

This is caused by that the two operands alpha and sum cannot serve as compatible ones for operator *

2):

/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/../unittest_utils.h:93:45: error: call of overloaded 'fabs(__half)' is ambiguous
   93 |                                         fabs(answer[i]-reference[i]), fabs(answer[i]-reference[i])/fabs(reference[i]));
      |                                         ~~~~^~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/features.h:490,
                 from /appl/scibuilder-spack/aalto-rhel9-dev/2024-01-compilers/software/linux-rhel9-haswell/gcc-11.4.1/gcc-12.3.0-xh5vv5d/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../include/c++/12.3.0/x86_64-pc-linux-gnu/bits/os_defines.h:39,
                 from /appl/scibuilder-spack/aalto-rhel9-dev/2024-01-compilers/software/linux-rhel9-haswell/gcc-11.4.1/gcc-12.3.0-xh5vv5d/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../include/c++/12.3.0/x86_64-pc-linux-gnu/bits/c++config.h:655,
                 from /appl/scibuilder-spack/aalto-rhel9-dev/2024-01-compilers/software/linux-rhel9-haswell/gcc-11.4.1/gcc-12.3.0-xh5vv5d/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../include/c++/12.3.0/functional:48,
                 from /scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:1:
/usr/include/bits/mathcalls.h:162:1: note: candidate: 'double fabs(double)'
  162 | __MATHCALLX (fabs,, (_Mdouble_ __x), (__const__));
      | ^~~~~~~~~~~
In file included from /appl/scibuilder-spack/aalto-rhel9-dev/2024-01-compilers/software/linux-rhel9-haswell/gcc-11.4.1/gcc-12.3.0-xh5vv5d/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../include/c++/12.3.0/random:38,
                 from /scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/cublas_wrapper.cc:2:
/appl/scibuilder-spack/aalto-rhel9-dev/2024-01-compilers/software/linux-rhel9-haswell/gcc-11.4.1/gcc-12.3.0-xh5vv5d/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../include/c++/12.3.0/cmath:241:3: note: candidate: 'constexpr float std::fabs(float)'
  241 |   fabs(float __x)
      |   ^~~~
/appl/scibuilder-spack/aalto-rhel9-dev/2024-01-compilers/software/linux-rhel9-haswell/gcc-11.4.1/gcc-12.3.0-xh5vv5d/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../include/c++/12.3.0/cmath:245:3: note: candidate: 'constexpr long double std::fabs(long double)'
  245 |   fabs(long double __x)
      |   ^~~~
/scratch/work/liw10/DistServe/SwiftTransformer/src/unittest/util/../unittest_utils.h:93:75: error: call of overloaded 'fabs(__half)' is ambiguous
   93 |                                         fabs(answer[i]-reference[i]), fabs(answer[i]-reference[i])/fabs(reference[i]));
      |                                                                       ~~~~^~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bits/mathcalls.h:162:1: note: candidate: 'double fabs(double)'
  162 | __MATHCALLX (fabs,, (_Mdouble_ __x), (__const__));
      | ^~~~~~~~~~~
/appl/scibuilder-spack/aalto-rhel9-dev/2024-01-compilers/software/linux-rhel9-haswell/gcc-11.4.1/gcc-12.3.0-xh5vv5d/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../include/c++/12.3.0/cmath:241:3: note: candidate: 'constexpr float std::fabs(float)'
  241 |   fabs(float __x)
      |   ^~~~
/appl/scibuilder-spack/aalto-rhel9-dev/2024-01-compilers/software/linux-rhel9-haswell/gcc-11.4.1/gcc-12.3.0-xh5vv5d/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../include/c++/12.3.0/cmath:245:3: note: candidate: 'constexpr long double std::fabs(long double)'
  245 |   fabs(long double __x)
      |   ^~~~

which is caused by that the value of answer[i]-reference[i] has type __half (more accurately template T as declared).

For these two problems, I made the following changes within my machine:

1. Change line 49 in .../SwiftTransformer/src/unittest/util/cublas_wrapper.cc into:

Carray[batch * stride_c + i * ldc + j] = alpha * static_cast(sum) + beta * Carray[batch * stride_c + i * ldc + j];
2. Change line 93 in .../SwiftTransformer/src/unittest/util/../unittest_utils.h to be:

fabs(static_cast(answer[i]-reference[i])),
fabs(static_cast(answer[i]-reference[i])) / fabs(static_cast(reference[i])));
Now the build command can successfully be completed and the later pip install for DistServe can be done. But I am not sure whether such changes will result in extra overheads as changing __half to be float increases the bits required for the variable. (Sorry for my naiveness in this field). If my modifications are reasonable, I can submit a new push request for that.

Thanks!! It works for me!