Skip to content

Conversation

@lonvia
Copy link
Member

@lonvia lonvia commented Dec 22, 2025

osmium uses thread pools to make reading and writing files faster through parallelization. When not configured explicitly, it uses a internally allocated thread pool saved in a static variable in the library. This is less than ideal for pyosmium because the library is spread over multiple shared library modules, each managing its own static variable. The setup also posed problems on Windows for some reason.

This PR changes the pool handling, so that every Reader and Writer create and use their own private thread pool. This means that the life time of the pool is bound to the life time of the Reader/Writer avoiding thread pools leaking at the end of the program.

For most users, this change will not make any difference because their scripts only use one, max two, readers or writers at the time. When using many readers in parallel however, there might be a noticeable increase in active threads and memory usage from the buffers. For those users, thread pools can now be managed from the Python side. To get back the previous behaviour of a single shared thread pool, create a thread pool at the beginning of the application and hand it in into the various processing functions.

@lonvia lonvia merged commit 7c45537 into osmcode:master Dec 23, 2025
28 checks passed
@lonvia lonvia deleted the reorganise-threadpools branch December 23, 2025 08:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant