speed up pytest collection #36938
Open
speed up pytest collection #36938
+136
−5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Thanks for contributing to openpilot! In order for us to review your PR as quickly as possible, check the following:
- Convert your PR to a draft unless it's ready to review
- Read the contributing docs
- Before marking as "ready for review", ensure:
- the goal is clearly stated in the description
- all the tests are passing
- the change is something we merge
- include a route or your device' dongle ID if relevant
Contributor
There was a problem hiding this comment.
Pull request overview
This PR optimizes pytest collection time by implementing a two-pronged approach: pre-warming expensive imports at conftest load time and lazy-loading fixture-specific modules only when needed. The changes achieve a 10-12% improvement in collection time.
Key Changes:
- Pre-warmed 13 heavy imports (numpy, hypothesis, cereal modules, casadi, etc.) in conftest.py to frontload import costs
- Implemented lazy-loading pattern for fixture-specific modules (OpenpilotPrefix, manager, HARDWARE) via getter functions
- Replaced TICI import with direct file existence check to avoid unnecessary module loading
- Migrated from --ignore flags to norecursedirs configuration in pyproject.toml for cleaner pytest configuration
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 27 comments.
| File | Description |
|---|---|
| conftest.py | Added pre-warming imports for 13 heavy modules, implemented lazy-loading getters for fixture modules, replaced TICI import with direct file check |
| pyproject.toml | Replaced --ignore command-line flags with norecursedirs configuration listing excluded directories |
Comments suppressed due to low confidence (1)
conftest.py:17
- Import of 'cereal' is not used.
import cereal
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
conftest.py
Outdated
Comment on lines
74
to
100
| _OpenpilotPrefix = None | ||
| _manager = None | ||
| _HARDWARE = None | ||
|
|
||
|
|
||
| def _get_openpilot_prefix(): | ||
| global _OpenpilotPrefix | ||
| if _OpenpilotPrefix is None: | ||
| from openpilot.common.prefix import OpenpilotPrefix | ||
| _OpenpilotPrefix = OpenpilotPrefix | ||
| return _OpenpilotPrefix | ||
|
|
||
|
|
||
| def _get_manager(): | ||
| global _manager | ||
| if _manager is None: | ||
| from openpilot.system.manager import manager | ||
| _manager = manager | ||
| return _manager | ||
|
|
||
|
|
||
| def _get_hardware(): | ||
| global _HARDWARE | ||
| if _HARDWARE is None: | ||
| from openpilot.system.hardware import HARDWARE | ||
| _HARDWARE = HARDWARE | ||
| return _HARDWARE |
There was a problem hiding this comment.
Inconsistent naming convention for private module-level variables. The global variables use leading underscores (_OpenpilotPrefix, _manager, _HARDWARE) following a private naming convention, but they represent cached module objects rather than truly private implementation details. The _HARDWARE variable also inconsistently uses SCREAMING_SNAKE_CASE while _manager uses lowercase. Consider using a consistent naming pattern like _openpilot_prefix_cache, _manager_cache, and _hardware_cache to better convey their purpose as cached values.
Suggested change
| _OpenpilotPrefix = None | |
| _manager = None | |
| _HARDWARE = None | |
| def _get_openpilot_prefix(): | |
| global _OpenpilotPrefix | |
| if _OpenpilotPrefix is None: | |
| from openpilot.common.prefix import OpenpilotPrefix | |
| _OpenpilotPrefix = OpenpilotPrefix | |
| return _OpenpilotPrefix | |
| def _get_manager(): | |
| global _manager | |
| if _manager is None: | |
| from openpilot.system.manager import manager | |
| _manager = manager | |
| return _manager | |
| def _get_hardware(): | |
| global _HARDWARE | |
| if _HARDWARE is None: | |
| from openpilot.system.hardware import HARDWARE | |
| _HARDWARE = HARDWARE | |
| return _HARDWARE | |
| _openpilot_prefix_cache = None | |
| _manager_cache = None | |
| _hardware_cache = None | |
| def _get_openpilot_prefix(): | |
| global _openpilot_prefix_cache | |
| if _openpilot_prefix_cache is None: | |
| from openpilot.common.prefix import OpenpilotPrefix | |
| _openpilot_prefix_cache = OpenpilotPrefix | |
| return _openpilot_prefix_cache | |
| def _get_manager(): | |
| global _manager_cache | |
| if _manager_cache is None: | |
| from openpilot.system.manager import manager | |
| _manager_cache = manager | |
| return _manager_cache | |
| def _get_hardware(): | |
| global _hardware_cache | |
| if _hardware_cache is None: | |
| from openpilot.system.hardware import HARDWARE | |
| _hardware_cache = HARDWARE | |
| return _hardware_cache |
| pass | ||
|
|
||
| try: | ||
| import cereal |
There was a problem hiding this comment.
Missing noqa comment for unused import. The cereal import at line 17 is imported for pre-warming but not used, similar to the other pre-warming imports. It should have a # noqa: F401 comment to suppress the unused import warning, consistent with the other pre-warming imports in this section.
Suggested change
| import cereal | |
| import cereal # noqa: F401 |
Comment on lines
+17
to
+21
| import cereal | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: |
There was a problem hiding this comment.
Module 'cereal' is imported with both 'import' and 'import from'.
Suggested change
| import cereal | |
| except ImportError: | |
| pass | |
| try: |
|
|
||
| # Pre-warm heavy imports before pytest collection | ||
| try: | ||
| import numpy # noqa: F401 |
There was a problem hiding this comment.
Import of 'numpy' is not used.
Suggested change
| import numpy # noqa: F401 | |
| import numpy # noqa: F401 | |
| _ = getattr(numpy, "__version__", None) |
| pass | ||
|
|
||
| try: | ||
| import hypothesis # noqa: F401 |
There was a problem hiding this comment.
Import of 'hypothesis' is not used.
Comment on lines
+8
to
+63
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import hypothesis # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import cereal | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from opendbc import car # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.tools.lib.logreader import LogReader # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import casadi # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.common.params import Params # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.selfdrive.test import helpers # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import parameterized # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import cereal.messaging # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from cereal import log # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from panda import Panda # noqa: F401 | ||
| except ImportError: |
There was a problem hiding this comment.
'except' clause does nothing but pass and there is no explanatory comment.
Suggested change
| except ImportError: | |
| pass | |
| try: | |
| import hypothesis # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import cereal | |
| except ImportError: | |
| pass | |
| try: | |
| from opendbc import car # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.tools.lib.logreader import LogReader # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import casadi # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.common.params import Params # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.selfdrive.test import helpers # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import parameterized # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import cereal.messaging # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from cereal import log # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from panda import Panda # noqa: F401 | |
| except ImportError: | |
| except ImportError: | |
| # Optional dependency for tests; safe to ignore if not installed. | |
| pass | |
| try: | |
| import hypothesis # noqa: F401 | |
| except ImportError: | |
| # Optional dependency for tests; safe to ignore if not installed. | |
| pass | |
| try: | |
| import cereal | |
| except ImportError: | |
| # Optional dependency for tests; safe to ignore if not installed. | |
| pass | |
| try: | |
| from opendbc import car # noqa: F401 | |
| except ImportError: | |
| # Optional dependency for tests; safe to ignore if not installed. | |
| pass | |
| try: | |
| from openpilot.tools.lib.logreader import LogReader # noqa: F401 | |
| except ImportError: | |
| # Optional dependency for tests; safe to ignore if not installed. | |
| pass | |
| try: | |
| import casadi # noqa: F401 | |
| except ImportError: | |
| # Optional dependency for tests; safe to ignore if not installed. | |
| pass | |
| try: | |
| from openpilot.common.params import Params # noqa: F401 | |
| except ImportError: | |
| # Optional dependency for tests; safe to ignore if not installed. | |
| pass | |
| try: | |
| from openpilot.selfdrive.test import helpers # noqa: F401 | |
| except ImportError: | |
| # Optional dependency for tests; safe to ignore if not installed. | |
| pass | |
| try: | |
| import parameterized # noqa: F401 | |
| except ImportError: | |
| # Optional dependency for tests; safe to ignore if not installed. | |
| pass | |
| try: | |
| import cereal.messaging # noqa: F401 | |
| except ImportError: | |
| # Optional dependency for tests; safe to ignore if not installed. | |
| pass | |
| try: | |
| from cereal import log # noqa: F401 | |
| except ImportError: | |
| # Optional dependency for tests; safe to ignore if not installed. | |
| pass | |
| try: | |
| from panda import Panda # noqa: F401 | |
| except ImportError: | |
| # Optional dependency for tests; safe to ignore if not installed. |
Comment on lines
+8
to
+68
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import hypothesis # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import cereal | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from opendbc import car # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.tools.lib.logreader import LogReader # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import casadi # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.common.params import Params # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.selfdrive.test import helpers # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import parameterized # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import cereal.messaging # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from cereal import log # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from panda import Panda # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.system.manager.process_config import managed_processes # noqa: F401 | ||
| except ImportError: |
There was a problem hiding this comment.
'except' clause does nothing but pass and there is no explanatory comment.
Suggested change
| except ImportError: | |
| pass | |
| try: | |
| import hypothesis # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import cereal | |
| except ImportError: | |
| pass | |
| try: | |
| from opendbc import car # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.tools.lib.logreader import LogReader # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import casadi # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.common.params import Params # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.selfdrive.test import helpers # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import parameterized # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import cereal.messaging # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from cereal import log # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from panda import Panda # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.system.manager.process_config import managed_processes # noqa: F401 | |
| except ImportError: | |
| except ImportError: | |
| # Optional dependency; safe to skip pre-warming if numpy is not installed. | |
| pass | |
| try: | |
| import hypothesis # noqa: F401 | |
| except ImportError: | |
| # Optional dependency; safe to skip pre-warming if hypothesis is not installed. | |
| pass | |
| try: | |
| import cereal | |
| except ImportError: | |
| # Optional dependency; safe to skip pre-warming if cereal is not installed. | |
| pass | |
| try: | |
| from opendbc import car # noqa: F401 | |
| except ImportError: | |
| # Optional dependency; safe to skip pre-warming if opendbc is not installed. | |
| pass | |
| try: | |
| from openpilot.tools.lib.logreader import LogReader # noqa: F401 | |
| except ImportError: | |
| # Optional dependency; safe to skip pre-warming if openpilot tools are not installed. | |
| pass | |
| try: | |
| import casadi # noqa: F401 | |
| except ImportError: | |
| # Optional dependency; safe to skip pre-warming if casadi is not installed. | |
| pass | |
| try: | |
| from openpilot.common.params import Params # noqa: F401 | |
| except ImportError: | |
| # Optional dependency; safe to skip pre-warming if openpilot common is not installed. | |
| pass | |
| try: | |
| from openpilot.selfdrive.test import helpers # noqa: F401 | |
| except ImportError: | |
| # Optional dependency; safe to skip pre-warming if openpilot selfdrive tests are not installed. | |
| pass | |
| try: | |
| import parameterized # noqa: F401 | |
| except ImportError: | |
| # Optional dependency; safe to skip pre-warming if parameterized is not installed. | |
| pass | |
| try: | |
| import cereal.messaging # noqa: F401 | |
| except ImportError: | |
| # Optional dependency; safe to skip pre-warming if cereal.messaging is not installed. | |
| pass | |
| try: | |
| from cereal import log # noqa: F401 | |
| except ImportError: | |
| # Optional dependency; safe to skip pre-warming if cereal.log is not installed. | |
| pass | |
| try: | |
| from panda import Panda # noqa: F401 | |
| except ImportError: | |
| # Optional dependency; safe to skip pre-warming if panda is not installed. | |
| pass | |
| try: | |
| from openpilot.system.manager.process_config import managed_processes # noqa: F401 | |
| except ImportError: | |
| # Optional dependency; safe to skip pre-warming if openpilot system manager is not installed. |
Comment on lines
+8
to
+68
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import hypothesis # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import cereal | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from opendbc import car # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.tools.lib.logreader import LogReader # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import casadi # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.common.params import Params # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.selfdrive.test import helpers # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import parameterized # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import cereal.messaging # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from cereal import log # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from panda import Panda # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.system.manager.process_config import managed_processes # noqa: F401 | ||
| except ImportError: |
There was a problem hiding this comment.
'except' clause does nothing but pass and there is no explanatory comment.
Suggested change
| except ImportError: | |
| pass | |
| try: | |
| import hypothesis # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import cereal | |
| except ImportError: | |
| pass | |
| try: | |
| from opendbc import car # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.tools.lib.logreader import LogReader # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import casadi # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.common.params import Params # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.selfdrive.test import helpers # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import parameterized # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import cereal.messaging # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from cereal import log # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from panda import Panda # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.system.manager.process_config import managed_processes # noqa: F401 | |
| except ImportError: | |
| except ImportError: | |
| # Optional dependency for speeding up tests; ignore if not installed. | |
| pass | |
| try: | |
| import hypothesis # noqa: F401 | |
| except ImportError: | |
| # Optional dependency for property-based tests; tests will be skipped or reduced if absent. | |
| pass | |
| try: | |
| import cereal | |
| except ImportError: | |
| # Optional dependency; some tests may be skipped or have reduced functionality if absent. | |
| pass | |
| try: | |
| from opendbc import car # noqa: F401 | |
| except ImportError: | |
| # Optional dependency; ignore if not installed in this environment. | |
| pass | |
| try: | |
| from openpilot.tools.lib.logreader import LogReader # noqa: F401 | |
| except ImportError: | |
| # Optional dependency used by certain tests; ignore if unavailable. | |
| pass | |
| try: | |
| import casadi # noqa: F401 | |
| except ImportError: | |
| # Optional optimization/solver dependency; tests handle its absence. | |
| pass | |
| try: | |
| from openpilot.common.params import Params # noqa: F401 | |
| except ImportError: | |
| # Optional openpilot dependency; ignore if project components are not installed. | |
| pass | |
| try: | |
| from openpilot.selfdrive.test import helpers # noqa: F401 | |
| except ImportError: | |
| # Optional openpilot test helpers; tests may be limited without this module. | |
| pass | |
| try: | |
| import parameterized # noqa: F401 | |
| except ImportError: | |
| # Optional test utility; parameterized tests may not run if missing. | |
| pass | |
| try: | |
| import cereal.messaging # noqa: F401 | |
| except ImportError: | |
| # Optional messaging dependency; ignore if not installed. | |
| pass | |
| try: | |
| from cereal import log # noqa: F401 | |
| except ImportError: | |
| # Optional logging schema dependency; tests adapt if unavailable. | |
| pass | |
| try: | |
| from panda import Panda # noqa: F401 | |
| except ImportError: | |
| # Optional hardware interface dependency; tests that need Panda will handle its absence. | |
| pass | |
| try: | |
| from openpilot.system.manager.process_config import managed_processes # noqa: F401 | |
| except ImportError: | |
| # Optional openpilot process configuration; ignore if openpilot is not installed. |
Comment on lines
+6
to
+69
| try: | ||
| import numpy # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import hypothesis # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import cereal | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from opendbc import car # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.tools.lib.logreader import LogReader # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import casadi # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.common.params import Params # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.selfdrive.test import helpers # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import parameterized # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import cereal.messaging # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from cereal import log # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from panda import Panda # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.system.manager.process_config import managed_processes # noqa: F401 | ||
| except ImportError: | ||
| pass |
There was a problem hiding this comment.
'except' clause does nothing but pass and there is no explanatory comment.
Suggested change
| try: | |
| import numpy # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import hypothesis # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import cereal | |
| except ImportError: | |
| pass | |
| try: | |
| from opendbc import car # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.tools.lib.logreader import LogReader # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import casadi # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.common.params import Params # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.selfdrive.test import helpers # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import parameterized # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import cereal.messaging # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from cereal import log # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from panda import Panda # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.system.manager.process_config import managed_processes # noqa: F401 | |
| except ImportError: | |
| pass | |
| with contextlib.suppress(ImportError): | |
| import numpy # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| import hypothesis # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| import cereal | |
| with contextlib.suppress(ImportError): | |
| from opendbc import car # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| from openpilot.tools.lib.logreader import LogReader # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| import casadi # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| from openpilot.common.params import Params # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| from openpilot.selfdrive.test import helpers # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| import parameterized # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| import cereal.messaging # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| from cereal import log # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| from panda import Panda # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| from openpilot.system.manager.process_config import managed_processes # noqa: F401 |
Comment on lines
+6
to
+69
| try: | ||
| import numpy # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import hypothesis # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import cereal | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from opendbc import car # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.tools.lib.logreader import LogReader # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import casadi # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.common.params import Params # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.selfdrive.test import helpers # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import parameterized # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| import cereal.messaging # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from cereal import log # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from panda import Panda # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
||
| try: | ||
| from openpilot.system.manager.process_config import managed_processes # noqa: F401 | ||
| except ImportError: | ||
| pass |
There was a problem hiding this comment.
'except' clause does nothing but pass and there is no explanatory comment.
Suggested change
| try: | |
| import numpy # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import hypothesis # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import cereal | |
| except ImportError: | |
| pass | |
| try: | |
| from opendbc import car # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.tools.lib.logreader import LogReader # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import casadi # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.common.params import Params # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.selfdrive.test import helpers # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import parameterized # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| import cereal.messaging # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from cereal import log # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from panda import Panda # noqa: F401 | |
| except ImportError: | |
| pass | |
| try: | |
| from openpilot.system.manager.process_config import managed_processes # noqa: F401 | |
| except ImportError: | |
| pass | |
| with contextlib.suppress(ImportError): | |
| import numpy # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| import hypothesis # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| import cereal | |
| with contextlib.suppress(ImportError): | |
| from opendbc import car # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| from openpilot.tools.lib.logreader import LogReader # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| import casadi # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| from openpilot.common.params import Params # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| from openpilot.selfdrive.test import helpers # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| import parameterized # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| import cereal.messaging # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| from cereal import log # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| from panda import Panda # noqa: F401 | |
| with contextlib.suppress(ImportError): | |
| from openpilot.system.manager.process_config import managed_processes # noqa: F401 |
- pre-warm heavy imports (numpy, hypothesis, cereal, opendbc.car, etc) - lazy-load fixture modules (OpenpilotPrefix, manager, HARDWARE) - use norecursedirs instead of --ignore flags
rosacry
force-pushed
the
pytest-collection-speedup
branch
from
c9bc7c9 to
e255c71
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #32611
Summary
Speeds up pytest collection by pre-warming heavy imports and lazy-loading fixture modules.
Implementation Details
OpenpilotPrefix,manager,HARDWARE) via getter functionsTICIimport with directos.path.isfile('/TICI')check--ignoreflags tonorecursedirsin pyproject.tomlPre-warmed Imports
Testing
Ran
pytest --collect-only -qmultiple times on both upstream/master and this branch to verify collection times. All 2951 tests still collected correctly.Benchmarks
Tested on WSL2 Ubuntu (Ryzen 9800x3D):
The cold-start improvement is most relevant for CI runners where imports aren't pre-cached.
Files Changed
conftest.py- Pre-warm imports, lazy-load fixtures, simplify TICI checkpyproject.toml- Use norecursedirs instead of --ignore flags