offline/online serving is stuck at fetching files in paraworkers

[hyuenmin-choi]

Hello guys I have some problems with deploying distserve on docker.

This is command what i used to launch docker env.

docker run --gpus all -it --name dist_bench --network=host --shm-size=10g -v /home/hyunmin/dataset:/workspace/dataset nvcr.io/nvidia/pytorch:24.05-py3 bash

I succefully build DistServe & SwiftTransformer inside my docker too.

However, when I tried to launch distserve online api_server or offline llm inference.
It just stucks in initializing the engine.

Is this problem related to shm size options?

Can anyone share the env or commands which successfully launch the serving.

Thank you in advance.

[Moxixis]

Hello, I have also encountered this problem. Have you solved it now? Perhaps we can communicate

[hyuenmin-choi]

In my case, just waiting solved the problem.. The main cause was just because of slow model loading. :)
Once you load it, it will run faster using cached data.