Bug Report: Creating large tifs causes gdal.translate to fail

[2320sharon]

Seg2map downloaded a bunch of multispectral tifs, and the goal is to combine them into an orthomosaic. The code bein used is from the merge_tifs function in src/seg2map/downloads.py

A vrt can be created, but when the code that converts it to a tif is run, i.e.

virtual_dataset = gdal.Translate(
           vrt_path.replace(".vrt", ".tif"),
           creationOptions=["COMPRESS=LZW", "TILED=YES"],
           srcDS=vrt_path,
       )

it creates the tif, then deletes it, and virtual_dataset is returned as None. It doesnt seem to matter if I change directories, or change output file name. Any idea why it creates then destroys the tif?

A Partial Solution

gdal.SetCacheMax(2**30)
When the cache is set to 2**30 bytes the code was able to mosaic 500 files (1.3 GB), which creates a 3.8GB mosaic tif file, but not all the files were able to be mosaiced together. The total number of files was (973, or 2.6GB)

[dbuscombe-usgs]

A tif may be created, but it is incomplete. I have tried various alternative options using gdal.Translate and gdal.Warp but haven't yet created a complete tiff.

I was able to convert a folder of tifs to a mosaic jpeg directly from the command line, using gdal_translate -of GTiff -co "COMPRESS=JPEG" -co "TILED=YES" -co "PHOTOMETRIC=RGB" merged_multispectral_2010.vrt mosaic.jpg, however I have so far been unable to reproduce this using the python API

[dbuscombe-usgs]

I also trialed approaches to mosaic creation that first involves creating jpegs (with worldfiles) from the individual tif files. However, while I can get the desired result from the command line gdal_translate, I cannot seem to get anything to work with the python API