How to handle preemption cases

[Mrxiangli]

Thank you for the contribution.

In the paper, you mentioned that DistServe does not consider preemption. During experiments/benchmarking, how do you control the request rates and the number of token generated for each request to make sure the decode GPU doesn't hit the memory limit? Thanks.

[Dreamer-HIT]

I have the same question. Have you solved this problem?

[Mrxiangli]

@Dreamer-HIT I think based on the code, when preemption occurs, the KV cache is transferred to host memory, and reload to the GPU when memory is available.