Release package 3.0.12 cannot be compiled with `-DWITH_PYTHON=1`

[cyrilbouvier]

I downloaded the .tar.gz package from https://github.com/SimonRohou/tubex-lib/releases/tag/v3.0.12
I wanted to compiled it with the python bindings. My cmake commands was:

cmake -DCMAKE_INSTALL_PREFIX=/nfs/work/contredo/software/tubex/3.0.12/ibex/2.8.9/gaol/none -DWITH_PYTHON=1 ..

There was several problems:

1. Cmake was unable to find the correct Tubex version (because the tubex directory is not a git repo, the cmake command version_from_git failed, but it did not prevent cmake to continue). See the start of the cmake output below. [ This error also appears without -DWITH_PYTHON=1 ]

2. The cmake command failed with the error

CMake Error at python/CMakeLists.txt:5 (add_subdirectory):
  The source directory

    /nfs/work/contredo/src/tubex-lib-3.0.13/python/pybind11

  does not contain a CMakeLists.txt file.

I was able to fix this by downloading the pybind11 source code into the directory (the problem is that the submodule is not added into the .tar.gz archive).
This error does not appears if I remove -DWITH_PYTHON=1.

3. With the fix from 2, the cmake command successfully finished (see the output below). But the make command failed with the error

[ 74%] Building CXX object python/CMakeFiles/tube.dir/src/core/arithmetic/tubex_py_arithmetic.cpp.o
[ 75%] Building CXX object python/CMakeFiles/tube.dir/src/core/cn/tubex_py_ContractorNetwork.cpp.o
/nfs/work/contredo/src/tubex-lib-3.0.12/python/src/core/cn/tubex_py_ContractorNetwork.cpp:21:10: fatal error: tubex_py_ContractorNetwork_docs.h: No such file or directory
 #include "tubex_py_ContractorNetwork_docs.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

This error does not appears if I remove -DWITH_PYTHON=1.

Cmake output:

-- Found Git: /usr/bin/git (found version "1.8.3.1") 
CMake Warning at scripts/CMakeModules/version_from_git.cmake:66 (message):
  [MunkeiVersionFromGit] Failed to execute Git: fatal: Not a git repository
  (or any parent up to mount point /nfs)

  Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Call Stack (most recent call first):
  CMakeLists.txt:9 (version_from_git)


-- The CXX compiler identification is GNU 7.5.0
-- Check for working CXX compiler: /trinity/shared/apps/cv-standard/gcc/7.5.0/bin/c++
-- Check for working CXX compiler: /trinity/shared/apps/cv-standard/gcc/7.5.0/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Full project version is 
-- Configuring build for tubex-lib 
-- Configuring tubex-lib in DEBUG mode as none was specified.
-- Running on system Linux-3.10.0-514.10.2.el7.x86_64 with processor x86_64
-- Using CMake 3.6.0
-- C++ compiler: GNU 7.5.0
-- Will build static libraries
-- Found IBEX version 2.8.9
-- Found Doxygen: /usr/bin/doxygen (found version "1.8.5") 
-- Failed to find sphinx-build executable (missing:  SPHINX_EXECUTABLE) 
-- Sphinx not found, not able to generate/install the manual.
-- Found PythonInterp: /trinity/shared/apps/local/Python/3.8.2/bin/python3.8 (found version "3.8.2") 
-- Found PythonLibs: /trinity/shared/apps/local/Python/3.8.2/lib/libpython3.8.a
-- pybind11 v2.5.dev1
-- /!\ Run "make api" for generating Doxygen doc, then "cmake .." again, before "make"
-- Generating docstrings from doxygen files...
-- /!\ Enable to build doc files (generate XML Doxygen files first)
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- LTO enabled
-- Configuring done
-- Generating done
-- Build files have been written to: /nfs/work/contredo/src/tubex-lib-3.0.12/build-ibex-2.8.9-gaol-none

[SimonRohou]

Hi Cyril!
Thanks for these feedbacks.

The problems of 1) and 2) are related to the conversion "Git repo" > "Zip source code" which does not contain the git module dependencies and the git tags, as you mention.
I don't know good solutions for this (maybe you have a hint?). Dealing with versions by using git tags is very convenient.

I could however remove the "source code" files of this Github page and invite the user to git clone the repo and git checkout the tag of the requested version.
For information related to the Python binding (for instance, configure the git submodule), more information can be found on http://simon-rohou.fr/research/tubex-lib/doc/dev/info_dev.html (section Installing local Python binding).

About 3), the reason is that we are using Doxygen to "transfer" the documentation of C++ functions to the help of Python functions. This was an attempt to have up-to-date documentation in the binding.
I think it did the job so far, however it requires a specific compilation procedure.

The steps are:

1. Write Doxygen doc for each functionnality in C++ header files
2. Configure CMake for preparing the output of documentation (command cmake)
3. Use Doxygen to generate some XML documentation of the functions (command make api)
4. Convert this XML files into .h header files of the Python binding (written in C++), and containing the strings of the documentation (automatic command called in make api). It will generate for instance the file tubex_py_ContractorNetwork_docs.h that is missing in your output.
5. Configure CMake again, for taking these files into account (command cmake)
6. Compile the projet + the binding simultaneously (command make)

This is documented in the following page:
http://simon-rohou.fr/research/tubex-lib/doc/dev/info_dev.html

If you have better ideas for all these points, do not hesitate!

Thanks,
Simon

[cyrilbouvier]

The problems of 1) and 2) are related to the conversion "Git repo" > "Zip source code" which does not contain the git module dependencies and the git tags, as you mention.
I don't know good solutions for this (maybe you have a hint?). Dealing with versions by using git tags is very convenient.

I do not know how to fix 1) and 2). For 2), it seems that including submodules in package generated by Github is a long-awaited request. For 1), you could write a fallback in version_from_git to read the version from a file, but it would mean generating the file and adding it in the archive when the archive is generated.

About 3), the reason is that we are using Doxygen to "transfer" the documentation of C++ functions to the help of Python functions. This was an attempt to have up-to-date documentation in the binding.
I think it did the job so far, however it requires a specific compilation procedure.

For this problem, I was able to rewrite the Cmake script to do this automatically. I was going to propose a PR but I do not have the right to push commits and branchs on this repo. So here is the diff of my commit:

diff --git a/python/CMakeLists.txt b/python/CMakeLists.txt
index a64dac51..805d8ef9 100644
--- a/python/CMakeLists.txt
+++ b/python/CMakeLists.txt
@@ -28,25 +28,15 @@
   # Generating docstring documentation from Doxygen
   ################################################################################
 
-  if(NOT EXISTS ${CMAKE_CURRENT_BINARY_DIR}/../doc/api/xml)
-    if(NOT WIN32)
-      string(ASCII 27 Esc)
-      set(ColourReset "${Esc}[m")
-      set(Red         "${Esc}[31m")
-      set(Blue        "${Esc}[34m")
-    endif()
-
-    message(STATUS "${Red}/!\\ Run \"make api\" for generating Doxygen doc, then \"cmake ..\" again, before \"make\"${ColourReset}")
-  endif()
-
-  message(STATUS "Generating docstrings from doxygen files...")
-  file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/docstring)
-  execute_process(COMMAND ${PYTHON_EXECUTABLE}
-    ${PROJECT_SOURCE_DIR}/scripts/pybind/doxygen2docstring.py
-    ${CMAKE_CURRENT_BINARY_DIR}/../doc/api/xml/
-    ${CMAKE_CURRENT_BINARY_DIR}/docstring)
   # The file generated_docstring.cpp contains all docstring of the library
-
+  add_custom_target(docs-headers
+                    COMMAND ${CMAKE_COMMAND} -E make_directory
+                                          ${CMAKE_CURRENT_BINARY_DIR}/docstring
+                    COMMAND ${PYTHON_EXECUTABLE}
+                      ${PROJECT_SOURCE_DIR}/scripts/pybind/doxygen2docstring.py
+                      ${CMAKE_CURRENT_BINARY_DIR}/../doc/api/xml/
+                      ${CMAKE_CURRENT_BINARY_DIR}/docstring
+                    DEPENDS api)
 
   ################################################################################
   # Library for Python binding
@@ -90,6 +80,8 @@
     src/robotics/tubex_py_DataLoader.cpp
   )
 
+  add_dependencies (tube docs-headers)
+
   target_include_directories(tube
     PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/../include 
     PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}

After that I can compile with make builddir && cd builddir && cmake -DWITH_PYTHON=1 .. && make and it works.