CI is flaky after test modernization

[amotl]

Problem

After modernizing the test suite, we are seeing some flukes. That's normal. It will probably need a few adjustment cycles or other proceedings to get it stable again.

• https://github.com/crate/crate-python/actions/runs/20072361183/job/57577715457?pr=745#step:6:250
• https://github.com/crate/crate-python/actions/runs/20072361183/job/57577715440?pr=745#step:6:201

References

• Modernize project #739

[surister]

I don't think it's related, I have seen in the past these flaky errors where the test layer errors because the array result might have a different order.

See flaky errors before test modernization in nightly - main:

• https://github.com/crate/crate-python/actions/runs/18862248255/job/53822730843
• https://github.com/crate/crate-python/actions/runs/18394732908/job/52411734583
• https://github.com/crate/crate-python/actions/runs/18394732908/job/52411734583
• https://github.com/crate/crate-python/actions/runs/18026176366/job/51293519885

Additionally, the test_layer is was not modified in the PR, the only difference is that tests are now being run with pytest instead of zope test runner, my hypothesis is that either:

• The tests are not correctly written to take into account the driver's multi-threading behavior.
• There is a subtle bug in the multi-threading part of the code.

Having the right result but different order in multi-threading python code is common imo.

[amotl]

Yes I agree. I hadn't seen the test_additional_settings fluke yet, but the invalid_host vs. even_more_invalid_host error is a known fluke:

• Tests: Flakyness with "active servers" logic within test suite #404

In this case, the test case modernizations made the error more apparent, probably by introducing different timings. Do you see a chance to fix it positively?

[amotl]

I've re-opened GH-404, and transferred the gist of your evaluation, thanks! In this spirit, I think it's fine to close this issue again.