Flush test for vioscsi/viostor fails sometimes

[kostyanf14]

Discussion started in PR #486 (comment)
Continue there

[benyamin-codez]

Thanks @kostyanf14 and apologies for the noise...!
I appreciate you making the time to discuss the problem.

I had said:

Yeah, that's what I see in the few I peeked into.
The actual test passes and then an error occurs during test cleanup.
Apparently it occurs whilst "Initializing and formatting the disk"...
So it doesn't appear to be related to this SNMP kit or reboots at all - at least in these cases.

Is it possible the disk is OFFLINE and so it cannot be initialised or formatted?
... If so, this is something we might need to address in the driver...
Is it perhaps something to do with the type of disk backing, i.e. COW or RAW?
This appears to be running (and failing) on non-bootable disks (as expected - except the failing part)...
I presume there's no AV or anything else interfering with anything in the boot block...

Quite annoying...! 8^(

---

System reset was requested but system not rebooted.

Do you perhaps see that in QEMU logs..?

The 0x8007007E error (missing module), i.e.:
Caught std::exception: onecore\drivers\storage\storutil\disk.cpp(478)\Storage.Tests.FlushTest.dll!...
... would seem to indicate part of the HLK is missing or misconfigured.

Possibly also the test is wiping the disk where the module is expected...?!?!
As in format is called which wipes the disk where Storage.Tests.FlushTest.dll is...?
e.g. C:\HLK\JobsWorkingDir\Tasks\WTTJobRunDCAC4D33-DFF4-EF11-BD88-56000200DDDD\Storage.Tests.FlushTest.dll

The 0X80070002 errors are usually related to the trace being missing (that could also be a hint):
e.g. C:\HLK\JobsWorkingDir\Tasks\WTTJobRunDCAC4D33-DFF4-EF11-BD88-56000200DDDD\Te.wtl

Is the storage::FlushFUATest::TestCleanup task misconfigured with or hard-coded to C:\ or \\?\ or \Device.....?!?

---

...and you replied with:

You can configure only the following options:

• IP IP Address of Remote PDU
• OID OID of Remote PDU outlet
• Outlet Port of Remote PDU outlet
• Community Community of Remote PDU (e.g. private)

All other configurations are done automatically by HLK (not AutoHCK)

---

And so now my latest thoughts are:

I suppose it could be a more generic, e.g. Device.Storage, configuration item...

There seems to be a functional collision related to C:\.
Is that also the boot volume (containing %systemroot%)..?

The Flush Test doco states:

This test runs both on boot and non-boot controller compared to the older test which could only run on non-boot controller. This enables testing of both boot disks and data disks. For boot disk, the multithreaded scenarios are skipped, whereas for a data disk, all the scenarios are executed.

How does the configuration accommodate testing of C:\ if the test files live there...?

Maybe the storage::FlushFUATest::TestCleanup is performing steps out of order or not in some way preserving the location / contents of C:\HLK\JobsWorkingDir\Tasks\...
...before attempting to initialise and format the disk...!?!?!?! 8^O
... ... which fails - but where did we put the contents of C:\HLK\JobsWorkingDir\Tasks\...
... ... ... or did we just try to close all the handles and lost access to the files...?
... ... ... ... and now we cannot access them for the rest of the test, and so we get the two errors:

0x8007007E : C:\HLK\JobsWorkingDir\Tasks\WTTJobRun{GUID}\Storage.Tests.FlushTest.dll
0x80070002 : C:\HLK\JobsWorkingDir\Tasks\WTTJobRun{GUID}\Te.wtl

The first one might mean no more reboots after the first test...
Is there a guest based reboot counter that actually increments?

8^{o>

[kostyanf14]

During WHQL we have 2 disks: boot (sata) and test (scsi/stor).
HLK puts evening related to test to C:\HLK (boot disk).

[benyamin-codez]

I've had some reliability issues with SATA in the past, esp. in combination with scsi/stor.
I exclusively use IDE now for my system and boot volumes (if I'm not boot testing) when cycling driver revisions during test.
IDE seems more reliable imho than SATA, at least on QEMU/KVM.

Are the HLK files getting clobbered during the Flush Test...?

Is there a guest based reboot counter that actually increments?

This would perhaps be a revealing check methinks.

Do you have any theories yourself, as to what might be going on here?

[kostyanf14]

I've had some reliability issues with SATA in the past, esp. in combination with scsi/stor.
I exclusively use IDE now for my system and boot volumes (if I'm not boot testing) when cycling driver revisions during test.
IDE seems more reliable imho than SATA, at least on QEMU/KVM.

If I remember correctly, on Q35 machine type you don't have IDE only SATA. Even if you specify IDE, Q35 platform emulates SATA connection. In AutoHCK we use -device ide-hd for boot device, but in Windows this is Standard SATA AHCI Controller

Are the HLK files getting clobbered during the Flush Test...?

Shouldn't

Do you have any theories yourself, as to what might be going on here?

Yes, and not only theories but some investigations.

1. I see different behavior on the Server 2022 CORE and the Server 2022 DESKTOP. On the Core test fails always (but with different error depending on installed updates). Looks like MS forgot to add some DLL. Not a first time, similar problem on Server 2025 with Static Tools Logo Test where RoMetadata.dll is missing. On Desktop test sometimes pass. So, to pass it on CORE we need to find the missing DLL/Component and install it.

2. During development of the fake-snmp-reset tool we saw different results with Socket+QMP and Telnet+HMP. This is a reason why we have sleep 2 in the fake-snmp-reset. Looks like MS expects some timing of reset based on real hardware behavior. In this case, rerunning tests several times helps.

Currently, we don't have the time to investigate what is missing for the test, and for real certification, we can run the test several times on the DESKTOP and merge the results.

[benyamin-codez]

Thanks for sharing, Kostiantyn. Some interesting info.

It does look like a collection of 0x8007007E (missing module) errors and then I think you are correct, the fake-snmp-reset tool might need some temporal tweaking and maybe use a different technique to do the power-cycling. Real hardware takes time to boot, notably when processing HBA firmware at POST, so we might need to emulate those delays.

Also, do you know if we receive SNMP traps or are they perhaps informs hoping for a PDU response?

Currently, we don't have the time to investigate...

That's fair enough. Time permitting, I might have a look at it in a couple/few weeks...