Multiple packages in one repo.

[matthewhammer]

We discussed wanting a new vessel feature to support multiple packages in one repo.

In particular, we'd like to continue to use vessel but have it support

1. moving the base library (back) into the main motoko repository.
2. splitting the base library into platform-independent parts (data structures) and IC-specific parts.

To support 1 and 2, we'd like to refine how vessel works so that a package is located not only by a repo, but instead by a pair of fields, repo and path, and the path specifies where within that repo one finds the vessel config files and the src directory for the package.

If the path field is absent, it is assumed to be the empty path, meaning that the repo has only one package, at its root (the current design).

Then, we'd have the motoko repo organized like this (simplified), assuming that the compiler and base and ic directories represent the code now in the motoko repo and base repos, but reorganized:

compiler
  src
  ...
lib
  base
    src
  ic
    src

cc @rossberg @crusso

[matthewhammer]

I did some initial work on a POC for this, in #38, but now I realize that assuming that every path ends with src seems somewhat artificial. An alternative design would have that path as None Text means that the code is in src, and otherwise, the path gives the complete path to the code, including any src directory at the end of that path (or not).

I am also wondering how these changes will affect the meta data in the vessel-package-set repo.

Some experimenting indicates that the Dhall type in file src/Package.dhall may need to change, from:

{ name : Text, version : Text, repo : Text, dependencies : List Text }

to this

{ name : Text, version : Text, repo : Text, path : Optional Text, dependencies : List Text }

and likewise, the dhall meta data may need to include path : None Text for the existing packages. My sense is that dhall is "too static" to permit any kind of field-is-absent test, and avoid this in the dhall code itself. However, maybe the serde support via Rust will automatically do this for its inputs? I'm still uncertain.

[matthewhammer]

I realize that assuming that every path ends with src seems somewhat artificial.

Updated #38 to relax this design decision.

I am also noticing that others are affected by the src/ constraint too, and perhaps, that fact justifies the extra field for specifying each package, as writing path : None Text is not very much extra work for the common case where it is src.

[q-uint]

I'd love to see support for this! 💯

---

Q: What would be the difference between the following package structures:

1. Importing one package from src/ and using subpaths to create 'sub-packages'.

// e.g.
// src/SubPkg1/Module1.mo
// src/SubPkg1/Class1.mo
// src/SubPkg2/ClassX/lib.mo

import Module1 "mo:pkg/SubPkg1/Module1"
import Class1 "mo:pkg/SubPkg1/Class1"
// ...
import ClassX "mo:pkg/SubPkg2/ClassX

2. Using the (new) proposed structure to seperate them in package-set.dhall.

// e.g.
// pkg1/src/Module1.mo
// pkg1/src/Class1.mo
// pkg2/src/ClassX/lib.mo

import Module1 "mo:pkg1/Module1"
import Class1 "mo:pkg1/Class1"
// ...
import ClassX "mo:pkg2/ClassX

Are there any advantages? Is one better than another?

---

I think the 2nd one is much cleaner than the first, but it just introduces another way to 'do the same'?

[rossberg]

@di-wu, I'd think the difference is that 2 makes them two distinct packages, which means clients can install them independently, they can be versioned independently, and so on. The ability to decouple is valuable.

[q-uint]

Great! I've got over 30 Motoko packages piled up, this'd help for sure!
Thanks for the response.

[q-uint]

They can be versioned independently...

Vessel uses git tags for versioning. How would this work with multiple packages in one repository?

[matthewhammer]

Vessel uses git tags for versioning. How would this work with multiple packages in one repository?

As an example, suppose one repo has two packages, A and B, and 50 git revisions. Suppose that C wants to use A and B in its codebase, as imported packages via vessel.

Then, C could chose each of A and B among 50 versions (each), where any pair of versions can represent the versions for A and B . To do so, C uses vessel to check out each of those two versions in the pair, one for A (ignoring B's path in that checkout) and one for B (ignoring A's path in that checkout). There is no issue, just some wasted space on disk for the paths in versions not being used.

I agree that this is non-intuitive since each package somehow "deserves" an independent timeline, but must instead share one with the other packages in its repo. But OTOH, that's what it means to share a repo.