Concept Relationships

[kosack]

Here I have started to try to link the various concepts we need to incorporate in VODF, to help to define the core structure (e.g. see #1) (here's a quick UML guide to help follow what the different lines mean)

---
title: VODF Concept Relationships
config:
  theme: neutral
  class:
    hideEmptyMembersBox: true
---

classDiagram
    Observation "1" *-- "1" TimeInterval 
    Observation "1" *-- "1" SpatialRegion
    Observation "1" -- "0..*" StableObservationInterval
    StableObservationInterval --|> TimeInterval
    StableObservationInterval "1" -- "0..*" IRF   
    IRF *-- EffectiveArea
    IRF *-- "0..1" PointSpread
    IRF *-- EnergyMigration
    IRF *--"0..1" RadMax

    IRF "1" -- "1" EventType
    EventList -- "0..*" EventType

    EventType <|--  EventTypeGroup
    EventTypeGroup "1" *-- "1..*" EventType

    Observation "1" -- "1..*" EventList
    StableObservationInterval "1" -- "1..*" EventList
    note for EventList "linked to SOI or Observation?"

Loading

EDIT: this is now out of date, see the discussion below

[bkhelifi]

Very nice plot, thank you.

About your question: most of the event are within SOI, but not all! Some belongs, in general, to time intervals during which the detector is not stable.
So, I think that eventlist are linket to a physical observation.

As a reminder, one of the mandatory field of an event list is the "timestamp", which allows to link to everything (SOI, and then IRFs)

[kosack]

That's right, I forgot that there are also events outside of an SOI, so then the linking has to be to observation.

[kosack]

Notes from meeting on 2025-03-19:

---
title: VODF Concept Relationships (for level-1)
config:
  theme: neutral
  class:
    hideEmptyMembersBox: true
---

classDiagram

   class DataRelease  {
         + data_release_id
         + time_interval
         + facility
         + processing_configuration
         + processing_pipeline_version
   }

  class Observation {
       + obs_id
       + time_interval
       + spatial_region : SkyRegion
       + instrument_name
       + ...
  }

  Observation "1" -- "0..1" Pointing
  Pointing <|-- NominalPointing
  Pointing <|-- TimeSeriesPointing  

  DataRelease "1..*" -- "1" ObservationCollection
  ObservationCollection "1" -- "1..*" Observation

  ObsCoreEntry -- Observation
  style ObsCoreEntry  color:skyblue

  Observation  "1" -- "0..1" EventList
  Observation  "0..*" *--  "0..1" IRF
  Observation "1" --  "1..*" OnTime
  OnTime <|-- SingleOnTime
  OnTime <|-- TimeSeriesOnTime  
  note for TimeSeriesOnTime "aka. 'GTI'"

   EventList "1" -- "0..*" Event

   Event "properties" -- "dimensions" IRFComponent

   IRF "1"  *-- "1..*" IRFComponent

 


   

Loading

• SOIs are now "gone", or rather renamed to Observation, which is the minimal unit.
  • the creation and assignment of Observations (obs_ids) is up to the producer.
  • Note that for CTAO, e.g. that means there can be a difference between ObservationBlock and Observation (previously we just used these interchangibly)
• all events must be in a OnTime interval (the producer shall ensure that), in other words, for places where there are holes in the OnTime, the instrument is off or not recording or the data were lost, and there should be a gap in event times that needs to be taken into account. Therefore no filtering is needed.
  • Bad data quality events != OnTime!, those could be in an Observation with no IRF, or withy a "degraded" IRF.
• an observation can have no IRF, and still have OnTime

[kosack]

We can then break the many-to-many relationship by introducing back the ObservationGrouping table.

classDiagram


  DataRelease "1" --  "1" ObservationGrouping
  ObservationGrouping "1" -- "1..*" Observation
  Observation "1"  -- "0..*" DataRelease 

Loading

[kosack]

Another discussion about IRFs: @LauraOlivera noted that we should just define "DL4 IRFs", which are just things generated from the DL3 IRFs by projection/binning, and potentially transformation into different coordinates. That includes the PSF Map, Exposure Map, etc. We have to be careful to also keep "provenance" info in these DL4 products, like what projections and transformations were used.

[kosack]

Minimum serialized info

Next we have to think about how to map this to implementation. First let's forget for a moment about discoverability, and think only about fetching 1 observation:

The minimum contents of a "Observation HDU" is the:

• Headers: Observation, DataRelease, and ObservationCollection
• Bintable: a grouping table containing the links (URI) to the HDUs of that observation to its associated EventList, Pointing, OnTime , IRF HDUs ...
  The sub-HDUs in the group may or may not be included, and could be fetched separately (e.g. if the URI points to a new file)

This is what is "data-linked" to a row in the results of a cone search or similar...

Now discoverability:

The "Observation Catalog" (e.g. ObsCore table, GADF ObsTable) for discoverability, is essentially the Headers of this Observation HDU turned into columns of a table . Therefore it shall be possible to regenerate the observation catalog from a set of Observation HDUs.

Concequently, this is FAIR, and IVOA compatible

[kosack]

We should specify that the v1.0 of VODF we will not support any IRF decomposition, but only the one we use now. But in principle, this could change in future versions.

classDiagram
   IRF "1"  *-- "1..*" IRFComponent

   StandardIRFDecomposition --|> IRF
   StandardIRFDecomposition *-- Aeff
   StandardIRFDecomposition *-- PSF
   StandardIRFDecomposition *-- Emig
   StandardIRFDecomposition *-- Bkg

Loading

[kosack]

Note that the top part of this down to Observation works also for DL4+, so the things that change are:

• events → optionally binned events , i.e. count maps, etc (optionally due to unbinned analysis, etc)
• IRFs → projected into exposure
• pointing → dropped

[kosack]

Another note: this is compatible with an Observation with no event list, i.e. just an IRF (or even without one), which can be used for simulations, etc.

[kosack]

Also to think about is to better define IRFComponents :

IRFComponents have:

• axes: could be internal (part of a "Map") or external (separate HDUs)
• values
• validity ranges/areas