Allow for client to provide records with newer COMPATIBLE or older (upgradable) version of schema to the server

[yarikoptic]

Summary over compatibility is in the docstring for the function in the diff. Citing current one here for shortcut:

    Compatibility here means that the server's schema version can be either

    - lower than client has, but within the same MAJOR.MINOR component of the version
      number for 0.x series, and same MAJOR version for/after 1.x series;
    - the same;
    - higher than the client has, but only if the client's schema version is listed
      among the server's `allowed_schema_versions` (as returned by the `/info` API endpoint),
      or if not there -- `dandischema.consts.ALLOWED_INPUT_SCHEMAS` is consulted.

    If neither of above, a `SchemaVersionError` is raised.

notes:

• - the same; is the current behavior which causes us troubles.
• with this change, if client uses more recent version, dandiset validation (I think) would fail since server would not know how to validate it. So it would need to wait for server to upgrade dandischema to match or exceed that one
  • I think that could be addressed in the future with client downgrading (when there is no data loss) the schema to match the server. Ref: Provide "downgrade" functionality to migrate and make dandi-cli use migrate! dandi-schema#343
  • meanwhile, I think, I should at least add a warning to users that they are using schema newer than what server expects and they might need to wait for it to upgrade to pass validation or need to downgrade dandischema package

And yes , it is looking into the future as for us to establish sensible behavior(s), and we might need to adjust depending on how we address

• expose "allowed_schema_versions" to /info (/server-info) dandi-archive#2624

but overall point is that dandi-cli was too protective of dandi-archive which does not really care what version of metadata you feed it with. Since there is other clients etc, if we want to restrict anyhow more -- we need to do it on server side! Meanwhile -- I am relaxing very sensibly requirements on client side! This way we very much decouple back our client from server and ease migrations etc.

Extra feature could be downgrades of schema records (dandi/dandi-schema#343) but that is yet to be decided on.

Additional refs:

• Add ability to downgrade schema to 0.6.10 dandi-schema#342 (comment)

Here I would introduce notion of "compatible" schema and forbid upload from "incompatible".

TODO

• check how server behaves now and if it cares at all... may be we should allow even for upload of older versions and just rely on minimal client version to guide... on the other hand -- it is easy to upgrade client/dandischema locally so cost is small but potentially with benefits. But may be we should just issue a warning if we detect some "incompatible" version and otherwise let server decide since it would be the one capable of upgrading and judging. Notes on this exploration are below in comments.

[codecov]

Codecov Report

❌ Patch coverage is 88.88889% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.08%. Comparing base (a79fffc) to head (0a0c59c).
⚠️ Report is 7 commits behind head on master.

Files with missing lines	Patch %	Lines
dandi/dandiapi.py	80.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1732      +/-   ##
==========================================
- Coverage   75.11%   75.08%   -0.04%     
==========================================
  Files          84       84              
  Lines       11856    11862       +6     
==========================================
  Hits         8906     8906              
- Misses       2950     2956       +6     

Flag	Coverage Δ
unittests	75.08% <88.88%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[yarikoptic]

with such dirty script

#!/bin/bash

export PS4='> '
set -x
set -eu
cd "$(mktemp -d ${TMPDIR:-/tmp}/dl-XXXXXXX)"

dandiset=000029

echo "Working at $PWD"

echo -ne "Current server info: "
curl --silent -X 'GET' \
  'https://api.dandiarchive.org/api/info/' \
  -H 'accept: application/json'  | jq .

# exploring either dandi-archive cares about versions of metadata at all!

asset=356b20b7-ae80-4d42-9715-075492eb025d
name=orig

# get info since that is where blob_id is more readily accessible as well as metadata
curl --silent -X 'GET' \
  "https://api.dandiarchive.org/api/dandisets/$dandiset/versions/draft/assets/$asset/info/" \
  -H 'accept: application/json'  > $name-info.json

blob_id=$(jq .blob -r  $name-info.json)

echo -ne "Asset metadata schema: "
jq .metadata.schemaVersion $name-info.json

echo -ne "Validation status: "
curl --silent -X 'GET' \
  "https://api.dandiarchive.org/api/dandisets/$dandiset/versions/draft/assets/$asset/validation/" \
  -H 'accept: application/json'

# Let's reupload into another file with some antiquated version of schema under a new path:
jq .metadata $name-info.json \
    | jq '.schemaVersion = "0.1.0"'  \
    | sed -e 's,\(path".*\)\.nwb,\1_dup-1.nwb,g' \
> $name-newass-meta.json

# we need to specify path as key?!
new_path=$(jq -r .path $name-newass-meta.json)

echo -e "New path arg: $new_path"

cat $name-newass-meta.json \
    | jq --arg blob_id "$blob_id" --arg path "$new_path" --slurpfile metadata $name-newass-meta.json \
        '{ blob_id: $blob_id, path: $path, metadata: $metadata[0] }' \
> $name-newass-put-data.json

curl -X 'PUT' \
  "https://api.dandiarchive.org/api/dandisets/$dandiset/versions/draft/assets/$asset/"\
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H "Authorization: token $DANDI_API_KEY" \
  -d "@$name-newass-put-data.json" > $name-newass-put-result.json

new_asset=$(jq .asset_id $name-newass-put-result.json)
# is it valid???
curl --silent -X 'GET' \
  "https://api.dandiarchive.org/api/dandisets/$dandiset/versions/draft/assets/$new_asset/validation/" \
  -H 'accept: application/json'

I discovered

• server does not care which schemaVersion upon upload. I did upload just fine stating it is "0.1.0" and we have now such asset https://api.dandiarchive.org/api/dandisets/000029/versions/draft/assets/87d291ef-bb3d-491b-8691-7ae9cbd659e3/
• validation of it did fail with desired

"Metadata version 0.1.0 is not allowed. Allowed are: 0.4.4, 0.5.1, 0.5.2, 0.6.0, 0.6.1, 0.6.10, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.6.7, 0.6.8, 0.6.9."

so, I think , Ideally it should be a server which exposes that information since it is concerned about its behavior and it in this case where server operates with newer version of the library and client tries to upload with older!

edit: I verified that if I specify version within that set, I did get new asset validate just fine -- see this version of asset with schema 0.6.1 which is reported valid

[candleindark]

Let a few comments

[yarikoptic]

Ok, let's begin this week with this release ;-)

[github-actions]

🚀 PR was released in 0.73.1 🚀