Add per-interpreter storage for gil_safe_call_once_and_store
#5933
+189
−38
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
XuehaiPan
force-pushed
the
subinterp-call-once-and-store
branch
3 times, most recently
from
58834ac to
a46973b
Compare
XuehaiPan
force-pushed
the
subinterp-call-once-and-store
branch
from
a46973b to
e741760
Compare
XuehaiPan
force-pushed
the
subinterp-call-once-and-store
branch
from
8e5bb6e to
d5b8813
Compare
XuehaiPan
commented
Dec 16, 2025
Comment on lines
+599
to
608
| // FIXME: We cannot use `get_num_interpreters_seen() > 1` here to create a fast path for | ||
| // the multi-interpreter case. The singleton may be initialized by a subinterpreter not the | ||
| // main interpreter. | ||
| // | ||
| // For multi-interpreter support, the subinterpreters can be initialized concurrently, and | ||
| // the first time this function may not be called in the main interpreter. | ||
| // For example, a clean main interpreter that does not import any pybind11 module and then | ||
| // spawns multiple subinterpreters using `InterpreterPoolExecutor` that each imports a | ||
| // pybind11 module concurrently. | ||
| if (get_num_interpreters_seen() > 1) { |
Contributor
Author
There was a problem hiding this comment.
cc @b-pass @rwgk for thoughts about this, see the code comment referenced for more details.
I test the patch in this PR in:
Most things works fine except test_import_in_subinterpreter_before_main:
run_in_subprocess(
"""
with contextlib.closing(interpreters.create()) as subinterpreter:
subinterpreter.exec('import optree')
import optree
"""
)If I remove the get_num_interpreters_seen() > 1 condition, my import test works but the cpptest in pybind11 CI breaks because internals_singleton_pp_ is never initialized. For instance, the memory address get_internals() should be a shared constant for differenet pybind11 modules in a single program.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixes #5926
For
!defined(PYBIND11_HAS_SUBINTERPRETER_SUPPORT), the previous behavior is unchanged.For
defined(PYBIND11_HAS_SUBINTERPRETER_SUPPORT), the pybind11internalsis already a per-interpreter-dependent singleton stored in the interpreter's state dict. This PR adds a new map to the interpreter-dependentinternalsfrom thegil_safe_call_once_and_store's address to the result, where thegil_safe_call_once_and_storeis a static object that has a fixed address.Suggested changelog entry:
gil_safe_call_once_and_storeto make it safe under multi-interpreters.