flatten-tool and BODS 0.4

[kathryn-ods]

Previously we've used flatten tool to create BODS templates.

Now that BODS has been upgraded to JSON 2020-12 I think this doesn't work anymore.

Following the instructions for https://flatten-tool.readthedocs.io/en/latest/usage-bods/#create-template

I am now getting the error

Error while resolving urn:components#/$defs/UnspecifiedRecord: URLError: <urlopen error unknown url type: urn>

[kd-ods]

@rhiaro will you have a chance to look at this over the next week or two?

[radix0000]

Currently flatten-tool does not support handling urn: links between schema files, where "the schema" is a directory rather than a single file. For updating BODS Cove to 0.4 I have got around this, with a branch (https://github.com/OpenDataServices/flatten-tool/tree/handle_bods_0.4) that allows BODS Cove to call flatten tool with the 0.4 schema. This allows the (Python) caller to pass a dict containing the schema (with minimal, and generic, changes to flatten-tool), and the loading and resolving of links can be done in cove_bods.

The issue is that there are actually multiple schema changes here, the actual change to DRAFT202012 of jsonschema, but also the use of urn: references to link the various schema files. Currently I am using the same approach as in the data standard tests themselves, of using the Register construct provided by jsonschema, within lib-cove-bods. This works well and has allowed the adding BODS 0.4, without the cascading changes propagating out into repositories we don't control. However this does mean there is no single schema file, which the command line tool expects.

There is a question about whether in the long term we want to support passing a schema to flatten-tool as a directory. This would require some changes to the CLI, as it would be necessary to pass not only the directory name, but also the root file (e.g. statement.json in the case of BODS 0.4), and adding code for building a Register construct out of the files in a robust manner (not just for BODS 0.4).

Updating compile-to-json-schema to build a single schema file where there are urn: links, would be an alternative way to go. I have currently avoided this for a number of reasons. It might need to involve either forking jsonref to add urn: support, or re-writing compile-to-json-schema to not rely on jsonref. Neither of these seems particularly palatable alternatives.

[kd-ods]

I've updated the title of this ticket so that it's a bit more general. I'd like to work out pretty quickly how to get flatten-tool to handle the BODS 0.4 schema.

Once we've done that we'll need to update the Flatten tool for BODS documentation. Much of what's there will only apply to BODS <= 0.3.

In fact, I think an interim update to that docs page would be a good call. I'll put in a separate ticket.

[radix0000]

Some cursory testing I have done suggests that there is more than one issue involved here, not just the resolving of urn: links between the schema files, but apparently a lack of support for the more complex use of conditionals in the BODS 0.4 schema. There are are a number of options to resolve this, but it seems like it should looked at once we have Data Review tool up and running.