Similar to what is in gammapy for the hdu_indextable.
You don't need to link all the hirarchy, just the level you want, like down to the IRF component (e.g. A_EFF). Could even link directly to the GROUPING table, but then you don't have as much control.
This HDU index table should have the exact structure of the IRF GROUPING Table, and the only difference is that MEMBER_LOCATION must be replaced with the filename of each observation file.
Other columns, like obs_id can be added optionally for efficiency (e.g. to avoid having to do a string compare with the filename).