How to record data about refinements?

[rowlesmr]

From #248

@jamesrhester :
However, we haven't yet untangled REFINE_LS to work out what Set categories are keyed to it, which will influence our choice here. If it is keyed to Structure, then that's not suitable as our overall refinement might include many structures. If it's keyed to exptl_crystal that's bad as well as we have many different crystallites.

What I think is clear is that there would be a "refinement id" to capture the results of multiple different refinements, and so perhaps we can do the same as "structure" and use the e.g. REFINEMENT category to assemble links to structures derived in that refinement and measurements used by that refinement. That REFINEMENT category would then hold the overall refinement statistics defined in a general way (not talk about reflections or points). But then it's not clear what the relationship to REFINE_LS is in the single crystal case. Need to think about this.

[rowlesmr]

What is clear is that we need to be able to handle all the combinations of multiple phases and multiple diffractograms

Many sequential refinements, simultaneous refinements, single refinements...

[jamesrhester]

I guess the first question is: does a refinement coincide with a complete dataset? Would we ever want to include more than one refinement in a dataset? I'm going to go with "yes", considering sequential refinements where many values are common but the individual measurements are often treated as separate refinements.

A refinement consists of observations and matching model calculations. Calculations are performed based on parameters which are either fixed or varying. Constraints and restraints may be applied. A refinement produces results in the form of the final values of the parameters, model calculations, and refinement statistics.

The current refine_ls category posits one of {F_calc, F^2_calc, I_calc} as the model values with corresponding *_meas values as the observations. I think it's implied that it captures refinement for a set of reflections that may arise from a number of crystals that have been merged, with the results of the modelling tabulated in refln and the refined model in the atom_site category, which currently therefore implies a single 'structure.id' as well.

Design

I think we can just about make refine_ls work for us. If not, anything below could just refer to some new refinement category instead.

Possible steps:

1. Define a _refine_ls.id to identify a refinement
2. Define the overall refinement statistics datanames required by PD in refine_ls, potentially superceding current data names by being more generic.
3. Define loop category refine_pd listing all pd_diffractogram.ids used as observations, with weights and any other per-diffractogram numbers. _refine_pd.refine_id is a key data name.
4. For every data name that may be a parameter, define a <dataname>_param_index that both indicates use as a parameter, and gives the index of that parameter in the correlation matrix. See below.
5. _structure.refine_id references which refinement a structure comes from
6. The remaining issues are constraints and restraints. I propose leaving those for now, important as they are.

<dataname>_param_index

I don't see a way around this. We could tabulate data names in some refine_param category, but a data name by itself doesn't identify a parameter - you need both column and row in a table, and data name gives you only column. We'd put
these definitions in a separate dictionary to avoid clutter. On the bright side, doing it this way is the proper relational way to do it.

Looking ahead to constraints and restraints, we'll need a way to write out equations, so I'm thinking we'll need a new DDLm type that holds dREL but is not used for evaluation of the data name itself, just as a way to express an equation.

I'd be happy to delay adding all these data names until after Vol G is published.

Application to PD

Single diffractogram, one phase

_refine_pd.pd_diffractogram points to the diffractogram, from which the observations pd_proc and calculated points values pd_calc can be found. Parameters identified as above.

Multiple diffractogram, multiple phases

Diffractograms listed in refine_pd. _structure.refine_id used to identify phases that participate in refinement

Sequential refinements

Each refinement gets an id, with the individual refinements described as above. Not sure how to express constraints across refinements at this stage, but as each refinement can be referred to and each parameter is (in principle) also findable the constraint can be expressed in the future.

[rowlesmr]

I'd be happy to delay adding all these data names until after Vol G is published.

I agree.

.

Just want to flag that I think that it would be better to define a new category, rather than co-opt and existing one. Mainly because REFINE_LS is already quite full of data names that would be used for single-crystal things, and if combined with powder data in a refinement, could be used for reporting single-crystal results.

[jamesrhester]

Just want to flag that I think that it would be better to define a new category, rather than co-opt and existing one. Mainly because REFINE_LS is already quite full of data names that would be used for single-crystal things, and if combined with powder data in a refinement, could be used for reporting single-crystal results.

Thought about this over the weekend, indeed, each distinct way of setting up a LS refinement should have its own category. Otherwise we find outselves in the situation where the meaning of the reference to the list of observations depends on the value of some other data name, which is a situation we want to avoid (Originally I had a vague idea of an uber-category that generalised and replaced refine_ls - so that's not going to work).

On the other hand, a GOF from LS refinement should have the same interpretation, even if calculated differently, just like F_meas - F_calc can be used to calculate a difference-density map regardless of whether those F values come from powder or single-crystal. If the F_meas/F_calc values from a powder refinement were to be expressed using different data names, that would unnecessarily complicate software. Similarly, the downstream use of a GOF from powder should be the same as from single crystal (averting our eyes from our powder habit of adjusting esds to make the GOF look good).

If single-crystal and powder data participate in a single refinement, then there will still be a single GOF at the end.

So it looks like we need a refinement category in core for the non-technique-specific statistics (GOF, number of parameters, smallest shift/esd), then a pd_refine_* category for the powder-specific statistics and setup.

In general, it seems like there's a bit of work to do which I think we could leave until after Vol G if we are generally agreed on the overall scheme.

[rowlesmr]

Yes. and I'll stop doing refinement statistics addition.