diffrn_data_frame should include a pointer to detector_id

[jamesrhester]

As a detector element can only be uniquely identified by both the detector id and the detector element id taken together (they are both keys of the diffrn_detector_element category), any reference to a detector element needs to include both. In particular, _diffrn_data_frame has a pointer only to _diffrn_detector_element.id so needs an additional data name that is a child of _diffrn_detector.id.

[yayahjb]

I'll add it as an implicit. Extensive use of implicit is an important distinction between imgCIF and other CIF dictionaries. See your 2005 message: https://www.iucr.org/__data/iucr/lists/cif-developers/msg00218.html We are going to need this in DDLm imgCIF (perhaps by using dREL) or we will invalidate a lot of imgCIF data files.

…

On Wed, Apr 4, 2018 at 10:48 PM, jamesrhester ***@***.***> wrote: As a detector element can only be uniquely identified by both the detector id and the detector element id taken together (they are both keys of the diffrn_detector_element category), any reference to a detector element needs to include both. In particular, _diffrn_data_frame has a pointer only to _diffrn_detector_element.id so needs an additional data name that is a child of _diffrn_detector.id. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEPiAQ28PIN-jARqddGX9Aw6ZhtW6Uh_ks5tlYYPgaJpZM4THx-2> .

[jamesrhester]

Just to recap on the behaviour of 'implicit': if _item.mandatory_code is implicit, the presence of an item is mandatory, but can be determined from context (Vol G p 66, section 2.6.6.1.1). For example, Vol G notes that parsing the save frame name is sufficient to determine _item.name for a definition so it is not necessary to include it in many of the DDL2 attribute definitions, even though it is, in a formal sense, part of the key for most DDL2 categories, so it has an _item.mandatory_code of implicit.

My understanding is that @yayahjb is proposing to add a child of _diffrn_detector_id to diffrn_data_frame with _item.mandatory_code set to implicit, with the meaning is that a value is required, but you can figure it out anyway if it is missing. Unfortunately, if data are measured from multiple detectors (potentially with identically-named elements), the particular detector producing a given data frame will not be determinable. Therefore, the implicit approach from DDL2 is inappropriate.

Of course, if an identifier data name takes only a single arbitrary value in a data block, then it is fine in a practical sense to drop it, as any value can just be created on the spot as needed. This behaviour is captured in DDLm using _audit.schema as follows:

1. Define a new schema, for which diffrn_detector is redefined as a Set category.
2. As a consequence, _diffrn_detector.id and all child data names are dropped, including the one missing from diffrn_data_frame.

The usual approach in core_CIF so far has been in the reverse direction: the default schema is the simplest one (the most Set categories) and then these categories are redefined as Loops for more complex situations. So in this approach the "base" DDLm imgCIF would correspond to the (most common) single-experiment, single-detector (anything else usually single?) scenario, and extension dictionaries would Loop diffrn_detector (and/or Loop variant). When translating to DDL2 imgCIF this latter dictionary would be the basis for translation.

If we wish to render historic imgCIF files more conformant to the DDLm imgCIF, we should choose the default situation (i.e. the one that does not require an explicit _audit.schema to be present in the data block) to be that which covers the most historic files. This suggests that the "base" described in the previous paragraph would be the appropriate choice. I note also that this default _audit.schema for DDLm core CIF corresponds to a single _diffrn.id, whereas mmCIF allows multiple _diffrn.ids in a single data block. Thus we are in any case forced to have a single-experiment imgCIF (which would be all of the historic imgCIFs anyway due to #2 causing difficulties otherwise). Note that this would also solve the problem raised in #2 , at least as far as DDLm imgCIF is concerned.

[yayahjb]

I believe our objective is to make a DDLm translation of the DDL2 imgCIF dictionary that will support _all_ existing CBF data. Inasmuch as any and all CBF and mmCIF categories can be looped, and implicit is heavily used, and we are increasingly dealing with working with very large heterogeneous multi-image data sets from multiple sources which may bring in multiple detectors, it would be counter-productive to make a set of different_detector or to ignore the variety of keys explicitly and implicitly used for managing images by thenselves, in single experiment single species clusters, in single experiment multi species clusters , in multi experiment single species clusters, and in multi experiment multi species clusters, etc. DDL2 gives us the power to represent all of these cleanly, and more. Indeed DDL2 gives us the power to represent just about anything that fits the relational model. If such capabilities are not yet in DDLm, we need to add support for implicit keys, for looping any and all categories, and for representing one-row category instances in data sets as looped in one data set and unlooped in another. We can make recommendations on better ways to organize data, but with so many existing data sets it would be impractical to mandate major changes in any significant fraction of them.

…

-- Herbert

On Thu, Apr 5, 2018 at 23:22 jamesrhester ***@***.***> wrote: Just to recap on the behaviour of 'implicit': if _item.mandatory_code is implicit, the presence of an item is mandatory, but can be determined from context (Vol G p 66, section 2.6.6.1.1). For example, Vol G notes that parsing the save frame name is sufficient to determine _item.name for a definition so it is not necessary to include it in many of the DDL2 attribute definitions, even though it is, in a formal sense, part of the key for most DDL2 categories, so it has an _item.mandatory_code of implicit. My understanding is that @yayahjb <https://github.com/yayahjb> is proposing to add a child of _diffrn_detector_id to diffrn_data_frame with _item.mandatory_code set to implicit, with the meaning is that a value is required, but you can figure it out anyway if it is missing. Unfortunately, if data are measured from multiple detectors (potentially with identically-named elements), the particular detector producing a given data frame will *not* be determinable. Therefore, the implicit approach from DDL2 is inappropriate. Of course, if an identifier data name takes only a single arbitrary value in a data block, then it is fine in a practical sense to drop it, as any value can just be created on the spot as needed. This behaviour is captured in DDLm using _audit.schema as follows: 1. Define a new schema, for which diffrn_detector is redefined as a Set category. 2. As a consequence, _diffrn_detector.id and all child data names are dropped, including the one missing from diffrn_data_frame. The usual approach in core_CIF so far has been in the reverse direction: the default schema is the simplest one (the most Set categories) and then these categories are redefined as Loops for more complex situations. So in this approach the "base" DDLm imgCIF would correspond to the (most common) single-experiment, single-detector (anything else usually single?) scenario, and extension dictionaries would Loop diffrn_detector (and/or Loop variant). When translating to DDL2 imgCIF this latter dictionary would be the basis for translation. If we wish to render historic imgCIF files more conformant to the DDLm imgCIF, we should choose the default situation (i.e. the one that does not require an explicit _audit.schema to be present in the data block) to be that which covers the most historic files. This suggests that the "base" described in the previous paragraph would be the appropriate choice. I note also that this default _audit.schema for DDLm core CIF corresponds to a single _diffrn.id, whereas mmCIF allows multiple _diffrn.ids in a single data block. Thus we are in any case forced to have a single-experiment imgCIF (which would be all of the historic imgCIFs anyway due to #2 <#2> causing difficulties otherwise). Note that this would also solve the problem raised in #2 <#2> , at least as far as DDLm imgCIF is concerned. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEPiAaSNWrf2rEGn78ngGsRXA0BJF9NMks5tltGBgaJpZM4THx-2> .

[jamesrhester]

@yayahjb , As pointed out above, the use of _item.mandatory_code of implicit in imgCIF appears to be inconsistent with the DDL2 definition. Instead, in imgCIF it appears to mean "if only one value is possible, dataname can be left out/might be missing".

If such capabilities are not yet in DDLm, we need to add support for implicit keys, for looping any and all categories, and for representing one-row category instances in data sets as looped in one data set and unlooped in another.

Allowing any category to be looped is just a matter of making sure the parent and child keys are all idenfied in the dictionary, which has already been done in imgCIF (in theory) - so no problems there, it is not a DDL issue. One row data sets can be presented syntactically unlooped, that has also long been agreed. You would need to define what implicit actually means in order to see if it makes any sense in a DDLm context. What I suspect we are really looking for instead of implicit is a blanket (and eminently sensible) rule stating that if a key data name can only take a single, arbitrary value then it can be missing from the data file.

[jamesrhester]

We went on a bit of a tangent there. I believe @yayahjb agreed at the top to add a child of _diffrn_detector.id to diffrn_data_frame as implicit. I assume this will form part of the key of diffrn_data_frame ?