Reorganise dependency repositories

[chrispbradley]

One of the original requirements for the build system was to be able to specify the version of a dependency library which would be used for the build. This then allows for new releases of dependency libraries to be investigated for general use and bring in bug fixes or new features. In addition to this, the OpenCMISS-Dependencies organisation contains copies of the dependency library code so that any OpenCMISS specific modifications or bug fixes can be incorporated. When these dependency copies were originally made most of the libraries did not use git to store the code. This situation has evolved and now a number of dependency libraries are stored in a GitHub repo or they are mirrored there. In order to make the updating of dependency library versions easier, it would be beneficial to take advantage of these GitHub repos by forking them to the OpenCMISS-Dependencies rather than manually creating our own copy.

A quick examination of the dependency libraries shows that they can be grouped into three main categories: 1) GitHub repos that use git tags to label the different versions of the library. These repos tend to have a small number of branches, typically a master branch and perhaps a develop or bug fix branch. 2) GitHub repos that use git branches to label the different versions of the library. These repos create a new branch for each major version and possibly branches off this for each minor version. 3) Not under a GitHub repo (although they may use git and be hosted on other platforms e.g. GitLab). For these libraries we would need to manually add the source code to a repo under OpenCMISS-Dependencies as is done now. For the GitLab repos it may be possible to ask the developer to mirror the GitLab repo on GitHub.

To allow for version specification and to take advantage of forks the build/manage system would require some changes.

To handle the version specification by user there would be a file with CMake variables that specify the current global default versions of the dependency libraries (as in the current OCComponentVersions.cmake file). The local config file would then have a commented out variable list of the versions to use. To use a version of a dependency that is different from the default the developer would uncomment the version variable and set it to the version required. There would also need to be some modifications to the build system to checkout the version set if it is different to the requested version.

For the GitHub dependency libraries that use tags a fork of the library would be made to OpenCMISS-Dependencies. In our fork two additional branches would be made, opencmiss and opencmissdevelop or ocdevelop or something. Because the upstream repository of the fork may have branches called develop or master then we would need some sort of prefix or postfix for the OpenCMISS branches to distinguish them. A particular library version tag (say v1.2.3) would then be pulled into the opencmiss branch. Any OpenCMISS specific changes would then be made and a new tag (say opencmiss-v1.2.3) created for the opencmiss branch. Note that because the upstream library will have tags of various names and formats we would need to prefix or post fix opencmiss or oc or something to the tag name in order to distinguish it. When the dependency library releases a new version then the opencmiss fork would be updated by pulling from the upstream. The new version would then be pulled into the opencmiss branch, conflicts fixed and a new opencmiss version tag created. The build system would then checkout the appropriate opencmiss version tag as specified from the opencmiss branch.

For the GitHub dependency libraries that use branches would need to be handled in a slightly more complicated manner. First a fork of the library would be made to OpenCMISS-Dependencies. Following on from what Alan does with OpenCOR dependencies that use branches we would, for each version branch (e.g., v1.2.3), an opencmiss version branch would be created off the library version branch e.g., opencmiss-v1.2.3 or v1.2.3-opencmiss or something else. Again we would need some prefix or postfix or something to distinguish the opencmiss branch from any similarity named branch in the upstream repo. Any opencmiss specific changes would then be made on the opencmiss version branch and a tag would be created on the opencmiss specific branch (say opencmiss-v1.2,3 or something). An opencmiss version develop branch may also be useful. The build system would then just checkout the opencmiss version tag. When the dependency releases a new version then the OpenCMISS-Dependency fork would pull in the upstream repo. A new opencmiss version branch would then be created for the new version. Unfortunately any opencmiss specific changes would then have to be readded to the new version branch. This, however, may be easier than it sounds as you can do a diff between the library v1.2.3 branch and the opencmiss-v1.2.3 branch and pull in that diff to the new opencmiss version branch.

For the third category dependencies we would need to manually add the library to OpenCMISS-Dependencies and manually make any version updates as it is done at the moment. To keep these repos similar to the ones above then it may be good to rename the current master and develop branches to opencmiss and opencmiss-develop. OpenCMISS version tags would also need to be added to the branches so that the build system could just checkout the appropriate opencmiss version tag from the opencmiss branch and keep everything consistent with the first two repo setups.

Any comments or other suggestions to achieve the desired result and functionality?

[vraj004]

Can you explain why we need opencmiss and opencmiss-dev for these 3rd party repositories as well?

[chrispbradley]

Hi @vraj004, which 3rd party repos? Type 1, 2 or 3?

[vraj004]

Sorry, I meant dependencies. This task of updating and fixing dependency issues will mostly been done by developers who actively maintain the oc libraries. So a develop suffix makes sense. Would we then push the fixed version to the oc suffix repo for use with the main oc libraries?

[chrispbradley]

I'm not 100% sure of what you're asking? You ask for an explanation of the need for the opencmiss and opencmiss-dev branches and then say a develop suffix makes sense? The code that will be checked out from a dependency repo for OpenCMISS users will be from the opencmiss or opencmiss-version branch. This branch is currently called master in our current dependency repo setup. We can't use the name master as this may clash with a branch named master in the upstream repo so we need to call it something different. This is the "trunk" version of the dependency (appropriately modified for OpenCMISS if required) and should always work. Yes, sometimes changes and modifications to dependencies are required and so a developer would be wise to put these into an opencmiss-dev or opencmiss-version-dev branch in order to test changes and not break the main trunk branch. This branch is currently called develop in our current dependency repo setup. Again, we can't use develop because it may clash with a branch named develop in the upstream dependency repo. Once a developer is happy with any changes in the opencmiss-dev or opencmiss-version-dev branch they will move these in to the opencmiss or opencmiss-version branch. Note that this is the process is for OpenCMISS specific changes or bug fixes etc. Any changes by the developers of the dependency will be done in their branches (without the opencmiss prefix or suffix). Is this what you mean?

[PrasadBabarendaGamage]

Just following up to see if there were any comments from others? - we are hoping to begin work on it from next Thurs (14th October). Thanks

[hsorby]

I am not pro this change. I have a lot of infrastructure that relies on the status quo. I also have a number of different projects that make use of the dependencies here and documentation on how to use them. I cannot see how you are proposing anything other than changing the labels of the references that we currently use.

I only see a very minor benefit to using forked repos. I have not read anything here that shows how a forked repo provides a practical benefit over and above what we currently have. As the repos. already exist in OpenCMISS-dependencies there is no cost to creating them and in any case the cost of creating a new dependency is so minimal it can be discounted.

The current build system does allow you to set the reference for whichever repository that you require a specified reference for, that functionality already exists this change will not help that in any way.

As for updating to a newer version of a repo the difference between what needs to happen is again minimal. Any person competent with the command line can make a new version available with maybe six or seven simple commands, easy to put into a script to be run when required.