AWS S3 Catalog

[ndp-opendap]

Create a BESCatalog implementation for an s3 bucket that utilizes the "/" character in object names to delimit the nodes in the catalog graph.

The s3-catalog should support the dmr++ system inherently;

• Their should be configuration settings for both the data bucket and the dmr++ bucket (these might be the same but then, maybe not)
• When the catalog identifies a "leaf" object that it recognizes as data (or maybe just a leaf regardless of identity) it should look in the dmr++ bucket to see if a dmr++ exists for the object.
• If a dmr++ is available then retrieve it (caching) and use it to build the various service responses.
• If no dmr++ is available then the catalog should attempt to make one?
• Get the source object (pull it from S3 - threading??) and cache it.
• If we aren't going to match the object name to a data reader (i.e. hdf5_handler) using a regex then we may need to inspect the object's content (range GET) to see if it's a file type for which we can create a dmr++.
• If we can make a dmr++ then do so a write it to the dmr++ bucket as <object_name>.dmrpp
• If we can't make a dmr++ then try to match the downloaded object with a data handler and serve the data that way.

[jgallagher59701]

I think there are several features here.

We could write a simple S3 Catalog that shows pages and provides a way to 'download files.' That could be implemented using the MVC design - so we have code that navigates the S3 bucket contents (model) and other code that builds (controler) web pages (view) for that. This model could then be used to make solutions for the following cases.

1. We could have a collection of DMR++ files in S3 and not have the template 'feature' but instead have the href point toward the data object. This would address a number of issues with DMR++ documents that reference two or more 'data blobs.'

2. We could also have a group of data files - data blobs from the DMR++ perspective - which already have DMR++ files. In that case, how do we find them? Of course, there are many ways, but which of those many ways do we want to support?

We could have a group of data files that have no DMR++. We could build a DMR++ for them 'on the fly,' store it somewhere, and then goto no. 2 (or a smart version of 2, given that the code would likely have the DMR++ file/object in hand).

I think we should move away from the href template in NASA and not use that in any other system we build.

[ndp-opendap]

I think we should move away from the href template in NASA and not use that in any other system we build.

I strongly disagree. Dropping the dmrpp:href attribute means that the dmr++ file is basically useless outside of the ecosystem in which it was created. No client will ever be able to get data using a dmr++ if there is not a dmrpp:href attribute, or something similar. This is especially true for dmr++ files that reference multiple source data granules.

Or are you meaning you want the dmr++ files to be complete and not use a templated value for the dmrpp:href attribute:

dmrpp:href="OPeNDAP_DMRpp_DATA_ACCESS_URL"

Getting rid of that would be a good thing.

[jgallagher59701]

(laughing) "Dropping the dmrpp:href"
I don't mean we drop the href.

I mean we stop using the template thing and put the real URL in there.