Large memory usage by panel_postprocessing_annotation.py

[scottrk]

Performing Duplexome level analysis. The generated captured_panel.tmp.gz file is ~18GB in size. The process eventually uses up all the available memory (>500GB) and is killed by the OOM killer (or Slurm, depending). I do get the following warning, which may be relevant to the issue:

/mnt/home/scottrk/.nextflow/assets/bbglab/deepCSA/bin/panel_postprocessing_annotation.py:143: DtypeWarning: Columns (7) have mixed types. Specify dtype option on import or set low_memory=False.

[FerriolCalvet]

Hi Scott!
thanks for reporting it, we are currently working in an updated version of the pipeline where the first steps are parallelized and then the duplexome level analysis are faster and more robust, but we are still working on it and it is maybe not fully functional.
I hope that in the next couple of weeks we can get this to work properly

(we will also do a deeper code review to address all these warnings)