This repository was archived by the owner on Jan 1, 2025. It is now read-only.
This repository was archived by the owner on Jan 1, 2025. It is now read-only.
RuntimeError: d.is_cuda() INTERNAL ASSERT FAILED #241
Description
Every time I train Mask2Former with Detectron2, I encounter the same error. The specific error is as follows:
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: d.is_cuda() INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1678402421473/work/c10/cuda/impl/CUDAGuardImpl.h":31, please report a bug to PyTorch.
This error does not appear at the beginning but usually occurs after running for a long time, such as 5 hours. This makes debugging very difficult. I would be very grateful if someone could help me look into this issue.