Introduce explicit thread pool management #311
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
osmium uses thread pools to make reading and writing files faster through parallelization. When not configured explicitly, it uses a internally allocated thread pool saved in a static variable in the library. This is less than ideal for pyosmium because the library is spread over multiple shared library modules, each managing its own static variable. The setup also posed problems on Windows for some reason.
This PR changes the pool handling, so that every Reader and Writer create and use their own private thread pool. This means that the life time of the pool is bound to the life time of the Reader/Writer avoiding thread pools leaking at the end of the program.
For most users, this change will not make any difference because their scripts only use one, max two, readers or writers at the time. When using many readers in parallel however, there might be a noticeable increase in active threads and memory usage from the buffers. For those users, thread pools can now be managed from the Python side. To get back the previous behaviour of a single shared thread pool, create a thread pool at the beginning of the application and hand it in into the various processing functions.