Tests stuck in "inQueue" state indefinitely during concurrent execution

[zjmletang]

commit 9faca5e (HEAD, tag: v0.17.4)
Description:
When running multiple tests concurrently in AutoHCK, we observed that a subset of tests occasionally gets stuck in the inQueue state and never transitions to Running or Completed. This seems to happen more frequently as the number of concurrent tests increases.

We are trying to determine if this is a resource bottleneck, a logic issue in the AutoHCK scheduler, or a known limitation regarding the maximum number of concurrent test sessions.

[zjmletang]

from @kostyanf14
AutoHCK can wait for multiple tests concurrently, but I know several places where "NullReferenceException" can occur if you manually run tests via VNC.

[zjmletang]

Tracking this for now. I've lost the previous logs, but I'll provide updates and logs once I reproduce the issue

[zjmletang]

@kostyanf14 I’d like to add some thoughts regarding the concurrency behavior. I suspect the instability (specifically the ConnectTimeoutError and tests hanging in inQueue) might be directly related to resource exhaustion.

For example, if the host has 128 CPU cores, but we trigger multiple tests (e.g., testing 'm' drivers across 'n' operating systems) that collectively demand more than 128 cores, how is AutoHCK supposed to react?

I’m curious about the design philosophy here. Is the resource management (such as limiting the total number of parallel VMs) intended to be handled by the user before triggering the tests?

I haven't found any global logic in AutoHCK to track host-wide CPU/RAM usage. If the expectation is that the user should manually throttle the number of concurrent tasks to fit the host’s capacity, I’d be happy to adjust my workflow. However, if this is something AutoHCK aims to manage internally, I'm happy to provide more logs to help debug why the current scheduling might be failing under load.

[zjmletang]

This is the result of my testing with the master branch of the netkvm driver. The following log illustrates one of the "stuck" cases. Note that this might be independent of concurrency, as the test was performed with low parallelism.

master_netkvm_2025.txt

results (93).html