Multithreaded import scanning

[seddonym]

Prior to this PR, we used Python's joblib library to scan imports in parallel.

This PR, enabled by the refactor of #237, moves the parallelism to Rust-based multithreading instead of doing multiprocessing with Python.

The benchmarks say it'll make it much slower, but I'm not seeing that in practice: running this on a very large graph, uncached, seems to speed it up from ~11s to ~8s, locally (compared with Grimp's latest unyanked release, 3.9). With a fully populated cache it's about the same, which is what we'd expect since this change is limited to the uncached path.

This is good news because I had to yank the Grimp 3.10 release due to it performing unexpectedly poorly on an uncached graph - this should allow us to do another release.

[codspeed-hq]

CodSpeed Instrumentation Performance Report

Merging #236 will degrade performances by 80.31%

_{Comparing concurrent-scanning (f764d6c) with main (67d1ec5)}

Summary

❌ 2 (👁 2) regressions
✅ 21 untouched benchmarks

Benchmarks breakdown

	Benchmark	BASE	HEAD	Change
👁	test_build_django_from_cache_a_few_misses[350]	297.9 ms	740.3 ms	-59.76%
👁	test_build_django_uncached	143.4 ms	728.1 ms	-80.31%

[Peter554]

Temporarily releases the GIL, thus allowing other Python threads to run

Which other python threads are running?

Are you sure allow_threads is needed? In the context of this commit it makes sense, but after the next commit, once joblib is removed, I'm unsure if it's still needed.

🐼 I think allow_threads is deprecating in favor of detach.

[seddonym]

Noted re. detach - I'll update when I update pyo3.

This was the crucial line I had to add to get things working - without it, everything was much slower - it didn't seem to be operating in parallel.

From https://pyo3.rs/main/parallelism.html

You should always call detach in situations that spawn worker threads, but especially so in cases where worker threads need to acquire the GIL, to prevent deadlocks.
(emphasis mine)

FWIW the explanation I got from Gemini of why this is was this:

Rayon's parallel iterators release the Global Interpreter Lock (GIL) internally, but pyo3 needs to be aware of this to ensure the Rust code can safely run in a separate thread without holding the GIL. The allow_threads block handles this by temporarily releasing the GIL, allowing Rayon to create its thread pool and execute the parallel workload. Without allow_threads, Rayon's thread spawn attempts would be blocked by the GIL, leading to deadlocks or other unexpected behavior.

[Peter554]

🎉

[Peter554]

Looks great!

I'm a but unsure if allow_threads is needed #236 (comment), but that's only minor