Handling a submodule within your repository that has compiled code

[adam2392]

Hi,

I am loving the new feature of editable installs on v0.13. I have a question regarding submodules. I am using Meson-Python itself to build my entire Python package, which has Cython and Python code and prolly soon a little bit of C++.

I am currently trying to vendor a fork of a popular package (scikit-learn) within my codebase. The current strategy I have is to create a maintained fork of scikit-learn and release it on pypi under an alias (e.g. scikit-learn-tree). However, this is not an ideal strategy as this prevents 3rd party packages that depend on scikit-learn from depending on my package. As a work-around it has been suggested that I vendor the forked repo as a submodule within my package instead.

However, this now raises a few questions as to how to build and develop on the package:

1. Is meson-python capable of building a submodule that doesn't use meson, but setuptools? E.g. something like ./spin build -j 4 (which also calls python setup.py install ./<my_pacakage>/_lib/scikit_learn_fork/)
2. Can this be done automatically and cleanly?
3. If not, is there another way of handling this situation using Meson?

[eli-schwartz]

You probably need to bite the bullet and port that submodule to meson too.

Question: why do you need to vendor at all, though? Vendoring just means you:

• build a private copy every time that cannot be reused between reinstalls of your own package
• the private copy is probably imported as mypackage.scikit_learn_tree instead of scikit_learn. But you could simply publish a public fork as "scikit_learn_tree" that doesn't conflict at all?

[dnicolodi]

The current strategy I have is to create a maintained fork of scikit-learn

I don't understand why you need a fork of scikit-learn. Isn't scikit-learn maintained? I see a release on March 9th, thus I would call it maintained. What is the reason why you need to fork or vendor scikit-learn?

[adam2392]

We want a fork because we make changes to a specific submodule inside scikit-learn that isn't in main yet. We therefore want the explicit fork so we don't get into this business of copying/pasting the code and then not being able to sync in changes from main. This just improves our maintainence in terms of refactoring to "be up-to-date".

Question: why do you need to vendor at all, though? Vendoring just means you:

• build a private copy every time that cannot be reused between reinstalls of your own package
• the private copy is probably imported as mypackage.scikit_learn_tree instead of scikit_learn. But you could simply publish a public fork as "scikit_learn_tree" that doesn't conflict at all?

This is an interesting strategy... I guess the issue is that rn scikit-learn-tree (the repo) gets imported with the sklearn namespace, which causes the issue. I suppose, I could just rename the entire module as sklearn_fork and stick w/ releasing the fork in this way then... I know this is not the original question anymore, but does that sound like a good fix?

scikit-learn unfortunately hasn't swapped to meson build yet, so would prefer not doing the first soln :p