BES experiences libdap segmentation faults when reading SimpleGroup.nc4.h5

[ndp-opendap]

Setup:

• The data file and the BES command files are included in the attached tar ball.
• The data file should be located here: /dap4/SimpleGroup.nc4.h5 relative to the BES Data Root directory.

Either of the included BES command files will cause the error, note that the trick to getting the problem to surface is to have besstandalone run the command two times:

besstandalone -c $prefix/etc/bes/bes.conf -i SimpleGroup.dmr.bescmd -f SimpleGroup.dmr -r 2
besstandalone -c $prefix/etc/bes/bes.conf -i SimpleGroup.dap.bescmd -f SimpleGroup.dap -r 2

simple_group_test.tgz

[ndp-opendap]

Here is the dataset DMR:

<?xml version="1.0" encoding="ISO-8859-1"?>
<Dataset xmlns="http://xml.opendap.org/ns/DAP/4.0#" xml:base="http://ngap-west.opendap.org:8080/opendap/dap4/SimpleGroup.nc4.h5" dapVersion="4.0" dmrVersion="1.0" name="SimpleGroup.nc4.h5">
    <Dimension name="time" size="1"/>
    <Dimension name="nv" size="2"/>
    <Dimension name="Z" size="1000"/>
    <Float32 name="time">
        <Dim name="/time"/>
    </Float32>
    <Float32 name="Z">
        <Dim name="/Z"/>
    </Float32>
    <Float32 name="Pressure">
        <Dim name="/Z"/>
        <Map name="/Z"/>
    </Float32>
    <Float32 name="time_bnds">
        <Dim name="/time"/>
        <Dim name="/nv"/>
        <Map name="/time"/>
    </Float32>
    <Attribute name="description" type="String">
        <Value>A simple group for testing.</Value>
    </Attribute>
    <Group name="SimpleGroup">
        <Dimension name="Y" size="40"/>
        <Dimension name="X" size="40"/>
        <Int16 name="Y">
            <Dim name="/SimpleGroup/Y"/>
        </Int16>
        <Int16 name="X">
            <Dim name="/SimpleGroup/X"/>
        </Int16>
        <Float32 name="Temperature">
            <Dim name="/time"/>
            <Dim name="/SimpleGroup/Y"/>
            <Dim name="/SimpleGroup/X"/>
            <Map name="/time"/>
            <Map name="/SimpleGroup/Y"/>
            <Map name="/SimpleGroup/X"/>
        </Float32>
        <Float32 name="Salinity">
            <Dim name="/time"/>
            <Dim name="/SimpleGroup/Y"/>
            <Dim name="/SimpleGroup/X"/>
            <Map name="/time"/>
            <Map name="/SimpleGroup/Y"/>
            <Map name="/SimpleGroup/X"/>
        </Float32>
    </Group>
</Dataset>

[ndp-opendap]

Here's the stack trace from my CLion debugger:

libdap::D4Dimensions::parent() const D4Dimensions.h:147
BESDapResponseBuilder::send_dmr(std::ostream &, libdap::DMR &, bool) BESDapResponseBuilder.cc:1342
SendDMR::send_internal(BESResponseObject *, BESDataHandlerInterface &) BESDapTransmit.cc:276
Sender::send(BESResponseObject *, BESDataHandlerInterface &) BESDapTransmit.cc:82
BESDapTransmit::send_basic_dmr(BESResponseObject *, BESDataHandlerInterface &) BESDapTransmit.cc:370
BESTransmitter::send_response(const std::string &, BESResponseObject *, BESDataHandlerInterface &) BESTransmitter.cc:93
BESDMRResponseHandler::transmit(BESTransmitter *, BESDataHandlerInterface &) BESDMRResponseHandler.cc:185
BESXMLInterface::transmit_data() BESXMLInterface.cc:423
BESXMLInterface::execute_data_request_plan() BESXMLInterface.cc:379
BESInterface::execute_request(const std::string &) BESInterface.cc:414
StandAloneClient::executeCommand(const std::string &, int) StandAloneClient.cc:238
StandAloneClient::executeCommands(std::ifstream &, int) StandAloneClient.cc:353
StandAloneApp::run() StandAloneApp.cc:240
BESApp::main(int, char **) BESApp.cc:57
main besstandalone.cc:34

Also of note: When the CLion debugger is at D4Dimensions::parent() (D4Dimensions.h:147) it shows that the this pointer is NULL.

[ndp-opendap]

Valgrind shows a slightly different failure point, D4Dimensions.cc:55. But that was collected from the docker container opendap/hyrax:1.17.1-572 and running valgrind (with the besstandalone commands shown above) inside the container .

valgrind said:

==2909== Invalid read of size 8
==2909==    at 0x52919B5: UnknownInlinedFun (basic_string.h:444)
==2909==    by 0x52919B5: libdap::D4Dimension::fully_qualified_name[abi:cxx11]() const (D4Dimensions.cc:55)
==2909==    by 0x5231EB2: libdap::Array::print_dim_element(libdap::XMLWriter const&, libdap::Array::dimension const&, bool) ==2909==    by 0x5232898: libdap::Array::print_dap4(libdap::XMLWriter&, bool) (Array.cc:927)
==2909==    by 0x5294316: libdap::D4Group::print_dap4(libdap::XMLWriter&, bool) (D4Group.cc:600)
==2909==    by 0x5294366: libdap::D4Group::print_dap4(libdap::XMLWriter&, bool) (D4Group.cc:608)
==2909==    by 0x529B32E: libdap::DMR::print_dap4(libdap::XMLWriter&, bool) (DMR.cc:337)
==2909==    by 0x9FD90F6: BESDapResponseBuilder::send_dmr(std::ostream&, libdap::DMR&, bool) (BESDapResponseBuilder.cc:1342)
==2909==    by 0x9FCE54F: SendDMR::send_internal(BESResponseObject*, BESDataHandlerInterface&) (BESDapTransmit.cc:276)
==2909==    by 0x9FCA6E1: Sender::send(BESResponseObject*, BESDataHandlerInterface&) (BESDapTransmit.cc:82)
==2909==    by 0x9FC92E7: BESDapTransmit::send_basic_dmr(BESResponseObject*, BESDataHandlerInterface&) (BESDapTransmit.cc:370)
(Array.cc:869)

I am attaching the complete valgrind output.

SimpleGroup.dmr.2.vg.txt

[Mikejmnez]

Here is the Python Code I used to create this file:

import numpy as np
from netCDF4 import Dataset

temp_data=np.arange(0, 1600, 1, dtype="<f4").reshape(1, 40, 40)
sal_data = 30 * np.ones(1600, dtype="<f4").reshape(1, 40, 40)
x = np.arange(0, 40, dtype="<i2")
y = np.arange(0, 40, dtype="<i2")
Z = -np.arange(0, 1000, 1, dtype=">f4")
pressure_data = -1025*8.8*Z

with Dataset('SimpleGroup.nc4', 'w') as output:
    output.description='A simple group for testing.'
    output.createDimension("time", 1)  # unlimited dimension
    output.createDimension("nv", 2)  # unlimited dimension
    output.createDimension("Z", len(Z))
    output.createVariable("time", "<f4", ("time"))
    output.createVariable("Z", "<f4", ("Z"))
    output.createVariable("time_bnds", "<f4", ("time",'nv'))
    output.createVariable("Pressure",  "<f4", ("Z"))

    output['time'][:] = np.array(0.5, dtype="<f4")
    output['time_bnds'][:] = np.arange(2, dtype="<f4")
    output['Z'][:] = Z
    output['Pressure'][:] = pressure_data
    
    group1 = output.createGroup('SimpleGroup')
    group1.createDimension("Y", 40)
    group1.createVariable("Y", "<i2", ("Y", ))
    group1.createDimension("X", 40)
    group1.createVariable("X", "<i2", ("X", ))
    group1.createVariable("Temperature", "<f4", ('time', "Y", "X"))
    group1.createVariable("Salinity","<f4", ('time', "Y", "X"))

    group1['Temperature'][:] = temp_data
    group1['Salinity'][:] = sal_data
    group1['X'][:] = x
    group1['Y'][:] = y



# now inspect the tree of file
from pydap.handlers.netcdf import NetCDFHandler

dataset = NetCDFHandler('SimpleGroup.nc4').dataset
dataset.tree()
.SimpleGroup.nc4
├──time_bnds
├──Pressure
├──SimpleGroup
│  ├──Y
│  ├──X
│  ├──Temperature
│  └──Salinity
├──time
└──Z

packages versions:

numpy=2.3.1
netCDF4=1.7.2
netCDF 4.9.3 # this is what I get when I run on the terminal nc-config --version

[ndp-opendap]

@Mikejmnez made a new file where the Dimension /time is moved from the root Group to the Group /SimpleGroup

I am unable to make this new file cause SIGSEGV.

File attached.

SimpleGroup3.tgz

[Mikejmnez]

@Mikejmnez made a new file where the Dimension /time is moved from the root Group to the Group /SimpleGroup

I am unable to make this new file cause SIGSEGV.

Here is that file's hierarchy: time, time_bnds, named dimension nv were moved inside SimpleGroup

.SimpleGroup3.nc4
├──Pressure
├──SimpleGroup
│  ├──time
│  ├──time_bnds
│  ├──Y
│  ├──X
│  ├──Temperature
│  └──Salinity
└──Z

[jgallagher59701]

Oh, There is a known issue with D4Maps and the parent 'link.' See HYRAX-1166 (yes, bizarre coincidence regarding the numbers) libdap4 bug: D4Maps copy constructor sets the d_array pointer to the source Array and not the parent Array

Can you look at what happens with the original (errant) file when there are no Maps for /SimpleGroup/Temperature and .../Salinity?

[Mikejmnez]

@jgallagher59701 hmm I am not sure how to make it so there are NO maps defined for those variables, while also defining the dimensions. I can certainly produce a file with the same variable, but without defining dimensions for Temperature and Salinity (just named dimensions). Would that help? So same file, make is to that Temperature and Salinity have their 3 dimensions be only named.

[jgallagher59701]

@jgallagher59701 hmm I am not sure how to make it so there are NO maps defined for those variables, while also defining the dimensions. I can certainly produce a file with the same variable, but without defining dimensions for Temperature and Salinity (just named dimensions). Would that help? So same file, make is to that Temperature and Salinity have their 3 dimensions be only named.

Yes, that would likely result in a file without Maps.

[ndp-opendap]

I spent some time in the debugger and I wonder it's the creation of the DMR object that is source of the issue. Consider this stack trace:

add_dap4_coverage_default(libdap::D4Group *, const std::vector<…> &) h5dmr.cc:2074
HDF5RequestHandler::hdf5_build_dmr_from_file(BESDataHandlerInterface &, BESDMRResponse &, libdap::DMR *, const std::string &, long long &, long long &) HDF5RequestHandler.cc:1511
HDF5RequestHandler::hdf5_build_dmr(BESDataHandlerInterface &) HDF5RequestHandler.cc:1320
BESRequestHandlerList::execute_current(BESDataHandlerInterface &) BESRequestHandlerList.cc:271
BESRequestHandlerList::execute_each(BESDataHandlerInterface &) BESRequestHandlerList.cc:183
BESDMRResponseHandler::execute(BESDataHandlerInterface &) BESDMRResponseHandler.cc:129
BESXMLInterface::execute_data_request_plan() BESXMLInterface.cc:373
BESInterface::execute_request(const std::string &) BESInterface.cc:414
StandAloneClient::executeCommand(const std::string &, int) StandAloneClient.cc:238
StandAloneClient::executeCommands(std::ifstream &, int) StandAloneClient.cc:353
StandAloneApp::run() StandAloneApp.cc:240
BESApp::main(int, char **) BESApp.cc:57
main besstandalone.cc:34

The hdf5_handler builds the DMR object but using hdf5 library to interrogate the file and then it inflates the DMR using hand written code (Not a parser) which could, for this SimpleGroup case, have an issue when associating the Dims Maps and Dimensions when the references cross Group boundaries. This activity seems like a place where something like this could go wrong.

// The main routine to handle the coverage support. The handled_all_cv_names should be detected before
// calling this routine. It includes all valid dimension scale variables. that is: the pure netCDF4-like
// dimensions are not included.
void add_dap4_coverage_default(D4Group* d4_root, const vector<string>& handled_all_cv_names) {

[ndp-opendap]

This one causes problems as well:

SimpleGroup4.nc4.h5.tgz

[Mikejmnez]

Here is a table to summarize files, their status, and their characteristics. These all have the same underlyind "simple" data:

Name	Characteristics	Status
SimpleGroup.nc4.h5	original file, time dim is at root level, Temp & Salin. in /SimpleGroup have time as Dim	❌
SimpleGroup3.nc4.h5	All dimensions of Temp and Salin are Variables in SimpleGroup	✅
SimpleGroup4.nc4.h5	Temp and Salin only have named dimensions (but Pressure and time_bnds have Maps)	❌
SimpleGroup5.nc4.h5	There are only named dimensions in file.	?

The interesting part is that SimpleGroup3.nc4.h5 works (see DMR above), even though it also has Maps

[ndp-opendap]

I also see a similar problem in /dap4/unaligned_simple_datatree.nc.h5 Using besstandalone and asking for the DMR two times reliably causes a SIGSEGV.

I am attaching the BES command.

UnalignedSimpleDataTree.dmr.bescmd.xml

[ndp-opendap]

Also I think SimpleGroup3 may also create issues in Hyrax...

[ndp-opendap]

I renamed the /dap4/unaligned_simple_datatree.nc.h5 file to /dap4/unaligned_simple_datatree.nc.h5.off to stop the bad behaviour for now.

But I added it here:
unaligned_simple_datatree.nc.h5.tgz