Coverage collection breaks when reorganizing tests into nested subdirectories

[EwoutH]

Over at Mesa we encountered a strange issue:

pytest-cov stopped collecting coverage data after reorganizing tests from a flat package structure into nested subdirectories even though the test discovery and execution worked perfectly. See mesa/mesa#2994 for the exact changes (only test file renames).

I tried about everything, but couldn't get it fixed with pytest-cov. Switching to coverage fixed it immediately (so we did that for now mesa/mesa#3005). I still would like to get to the root cause. Below some details.

What happens:

• All 379 tests are discovered ✅
• All 379 tests run successfully ✅
• pytest-cov reports: "No data was collected. (no-data-collected)" ❌
• coverage.py itself shows the warning, suggesting it ran but didn't trace anything

Environment:

• pytest-cov 7.0.0
• coverage 7.13.0
• Python 3.14.2
• pytest 9.0.2

Workaround:
Using coverage run -m pytest instead of pytest --cov works correctly
and reports 82% coverage as expected.

Configuration:

[tool.coverage.run]
source = ["mesa"]
branch = true
omit = ["tests/*"]
relative_files = true

Questions:

1. Is this a known limitation of pytest-cov with nested test packages?
2. Should pytest-cov detect this and provide a clearer error message?
3. Is there a configuration option we're missing that would make this work?

The cryptic "no data collected" error gave no hints about what changed or
how to fix it. Given that test reorganization into subdirectories is a
common refactoring, better diagnostics would be valuable.

[webknjaz]

One problem I see is that you stopped collecting coverage from your test folder entirely. Previously, it was being collected (and saved into the .coverage sqlite db file) but was excluded from the reports (by default; you could still request it to show up through CLI/pytest-cov wrapper). Moving omit from report to run made it more global and so coveragepy would pretend those files don't even exist.

Another problem I often see is that the old source setting is ambiguous — it's either an importable or a directory.

I recommend having source = . and source_pkgs = your_importable — this is what works best with platforms like codecov in particular.
I end up having to fix this often: python-trio/trio#3367

You should really measure everything, not just the runtime code. In fact, if you don't enforce 100% code coverage on your test folders, you my end up with some tests not actually running anywhere and nobody ever realizing: https://nedbatchelder.com/blog/202008/you_should_include_your_tests_in_coverage.html / https://nedbatchelder.com/blog/201908/dont_omit_tests_from_coverage.html / https://nedbatchelder.com/blog/201106/running_coverage_on_your_tests.html

The easiest way to achieve this is setting up enforcement on Codecov when individual CI jobs might have platform-dependent skips: https://github.com/ansible/awx-plugins/blob/274b32a/.codecov.yml#L33-L38 — you can define separate checks for folders like this.
But there's also plugins that may let you mark omits right in Python modules, granularly: jazzband/pip-tools#2264 / python-trio/trio#3163

Here's probably the most well-thought config example I've ever composed: https://github.com/ansible/awx-plugins/blob/274b32a/.coveragerc
With that, I also pre-load the plugin early, keeping shared CLI args in the pytest config: https://github.com/ansible/awx-plugins/blob/274b32a/pytest.ini#L21-L28
And the env-specific ones (local dev env vs. CI) go into the configs of those tools overridable defaults: https://github.com/ansible/awx-plugins/blob/274b32a/tox.ini#L130 / https://github.com/ansible/awx-plugins/blob/274b32a/.github/workflows/ci-cd.yml#L527-L537

Additionally, if you also use pytest-xdist, it is usually helpful to disable some warnings as well since they can be confusing due to internal order of operations being less deterministic: https://github.com/cherrypy/cheroot/blob/662cd9d/.coveragerc#L26-L42

Hope this helps!

[EwoutH]

Thanks for your quick and extensive reply. We're diving in, really appreciated!