Add documentation on building the package

[marcverhagen]

New Feature Summary

There is no documentation besides the actual code in the makefile and the setup file.

I created some informal documentation on how to do this, with some explanations, and want to add this somewhere.

Related

No response

Alternatives

No response

Additional context

No response

[keighrim]

more documentation needed for details like

• what's "generated" when doing make package ?
• how to upload to pypi?
• how the documentation (gh-pages) is published?

[keighrim]

On top of the additional documentation needs, @marcverhagen and I just had a conversation on migrating setup.py+Makefile to pyproject.toml+python -m build workflow. I'll experiment with the idea within the open PR (#329)

[keighrim]

Here's some finding and proposal from my research into the migration problem

---

migration scenario: from setup.py to pyproject.toml

summary

Do we need setup.py?
With requires-python = ">=3.10", setup.py is technically unnecessary but pragmatically useful during migration.

All Python 3.10+ users have modern pip and setuptools that can build packages directly from pyproject.toml using PEP 517/518 standards. However, maintaining setup.py temporarily supports legacy workflows and developer habits while the ecosystem transitions to modern build tools.

general assessment of the current python packaging landscape

Python 3.10 was released in October 2021 (4yr ago), which means:

1. pip
   • Python 3.10+ distributions ship with pip >= 21.x (pip install . and pip install -e .)
   • pip 19.0 (January 2019) added PEP 517/518 support (python -m build support )
   • All Python 3.10+ users have pip that can build from pyproject.toml alone
2. setuptools
   • setuptools 61.0 (March 2022) added full pyproject.toml metadata support
   • Python 3.10 users can easily install modern setuptools
   • We can declare requires = ["setuptools>=61.0", ...] to make sure the build environment is correct

why we kept using setup.py

Despite Python 3.10+ making setup.py technically unnecessary, this project continues to use it for the following reasons:

1. GitHub Actions CI Pipeline Integration

   • Our CI/CD pipeline uses Makefile which internally calls python setup.py sdist and related commands
   • Migrating the entire build pipeline would require updating multiple workflow files and testing
   • The CI Pipeline is also shared with clams-python project, so when one migrates, both need to be updated simultaneously

2. Developer Habits and Familiarity

   • Simply, I'm accustomed to using python setup.py develop for development installations
   • Existing documentation and scripts reference setup.py when they do (although as the original body of this issue goes, documentation rarely says anything about actually running make or setup.py)

3. Build-Time Code Generation (Most Critical)

   • The vocabulary package and other python code must be generated at build time by fetching and processing external vocabulary definitions
   • This code generation logic is currently implemented in setup.py using custom command classes
   • Setup.py's prep_ext_files decorator hooks into setuptools' sdist and develop commands to execute generation logic before packaging
   • Migrating this would require implementing a custom build backend or setuptools plugin

So the main roadblock to removing setup.py is the build-time code generation logic. If we can migrate that logic to somehow work with pyproject.toml, we can safely remove setup.py, and replace setup.py + Makefile with pyproject.toml + python -m build commands.

migration plan (using setuptools's "entry points")

Here's the result of my research on the migration path.

1. Setuptools provides a finalize_distribution_options entry point specifically designed for pre-build hooks. Looks like it's the recommended approach for running code generation before packaging.

2. We add entry point to pyproject.toml:

   [project.entry-points."setuptools.finalize_distribution_options"]
   build_hooks = "build_tools.hooks:setup_hooks"

3. Separately implement the hook function in somewhere like build_tools/hooks.py:

   def setup_hooks(dist):
       """
       Called by setuptools during build preparation.
   
       This function runs before building wheels/sdists and can:
       - Generate Python code (vocabulary package)
       - Fetch external resources
       - Validate build environment
   
       :param dist: setuptools Distribution object
       """
       from build_tools.generation import generate_vocabulary
   
       generate_vocabulary()
       # Other build-time generation logic...

References:

• Setuptools Extension Points
• Entry Points Specification
• PEP 517 - Build System Interface

This approach eliminates the need for setup.py while maintaining the same build-time code generation functionality.