Qwen3-Next-80B-A3B-Instruct

[RealCaptainReaper]

Hey, thanks for maintaining! I recently tried Next and Nemotron-3-Nano-30B-A3B-Q4_K_M. Both failed after the second completion (the first one always works), with sth. like:

init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:

• the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 16
• the tokens for sequence 0 in the input batch have a starting position of Y = 2
  it is required that the sequence positions remain consecutive: Y = X + 1
  decode: failed to initialize batch
  llama_decode: failed to decode, ret = -1

Is it just something on my setup? Can you run these newer models successfully?

[JamePeng]

It seems to be related to the batch decode processing logic. I'll take a look at these new models later.

Could you describe your runtime environment, your hardware configuration (processor, memory, VRAM), and the software parameters you've set in your code?

[JamePeng]

It has been confirmed that the memory_seq_rm operation in the second round of the eval phase of the two new models, Qwen-Next and Nemotron-3-Nano, returned false, indicating that the cleanup was unsuccessful and caused the sequence misalignment, triggering the assert check.

[RealCaptainReaper]

ah ok, yes should have provided my test, the calls weren't consecutive, so that is probably the reason why it tried to clean it up:

response = main_model.create_completion(
    prompt ="""Hello""",
)

response = main_model.create_completion(
    prompt ="""Hello, again!"""
)

Based on your finding I temporarily added

if not is_success:
    self._ctx.memory_seq_rm(0, 0, -1)

so that I can do some more model tests. But when is_success = self._ctx.memory_seq_rm(0, current_pos, -1) is the issue in that case, what is the correct way to handle it?