crashes after "checkpoint extract_barcodes"

[IlinaMargo]

Good afternoon! I previously ran the tests and everything worked fine.

Now, I'm trying to run BLR on my own data — a flax genome (Linum usitatissimum, ASM22429v2) with stLFR reads. However, I'm running into issues, and I'm not sure how to debug them since there’s no clear error message.

I install and run it as follows:
git clone https://github.com/AfshinLab/BLR.git
conda install -c conda-forge conda-lock
conda activate blr
pip install .
conda create --name blr --file environment.linux-64.lock
blr init --reads1=/media/eternus1/projects/milina/genotrophs/raw_data/20DNA/20DNA.clean_R1.fastq.gz -l stlfr /media/eternus1/projects/milina/genotrophs/stLFR/blr_output/
blr config --set genome_reference /media/eternus1/projects/milina/reference/GCA_000224295.2_ASM22429v2_genomic.fa

I also updated blr.yaml to include the chromosome names from the reference:
phasing_contigs: CP027619.1,CP027626.1,CP027627.1,CP027628.1,CP027629.1,CP027630.1,CP027631.1,CP027632.1,CP027633.1,CP027620.1,CP027621.1,CP027622.1,CP027623.1,CP027624.1,CP027625.1

The pipeline runs up to about 40% and then just seems to stall without an actual error. Here’s the end of the Snakemake log:(/media/eternus1/projects/milina/genotrophs/stLFR/blr_output/.snakemake/log/2025-04-16T162136.645977.snakemake.log)

samtools index final.barcodes.bam final.barcodes.bam.bai
[Thu Apr 17 10:36:26 2025]
Finished job 615.
621 of 628 steps (99%) done
Select jobs to execute...

[Thu Apr 17 10:36:26 2025]
checkpoint extract_barcodes:
input: mtglink.gfa, final.barcodes.bam, final.barcodes.bam.bai
output: mtglink_tmp/read_subsampling_pre
log: mtglink_tmp/read_subsampling_pre.log
jobid: 617
Downstream jobs will be updated after completion.

python -c "import sys; print('.'.join(map(str, sys.version_info[:2])))"
Activating conda environment: /home/milina/common_conda_envs/39858c45c61b8084a22c2330556deb97
python /media/eternus1/projects/milina/genotrophs/stLFR/blr_output/.snakemake/scripts/tmpenjhiz6x.extract_barcodes.py
Activating conda environment: /home/milina/common_conda_envs/39858c45c61b8084a22c2330556deb97
Updating job aggregate_extracts.
Updating job mtglink.
[Thu Apr 17 10:37:23 2025]
Finished job 617.
622 of 1557 steps (40%) done
Complete log: /media/eternus1/projects/milina/genotrophs/stLFR/blr_output/.snakemake/log/2025-04-16T162136.645977.snakemake.log

There's no error message in the logs, which makes it hard for me to understand what’s going wrong or whether it’s just slow.
Any ideas on what might be happening, or how I could better debug this?
Also in my output directory I have some log files too, but there is no information about mistake, only three of this logs have zero information: ./final.samtools_stats.txt.log ./initialmapping_nobc.bam.sorting.log and ./manta.log

Some system Information:
OS: Ubuntu 22.04.4 LTS
Kernel: 5.15.0-78-generic
CPU: Intel(R) Xeon(R) Platinum 8268 @ 2.90GHz — 192 threads (4 sockets × 24 cores × 2 threads)
RAM: 2.9 TiB total, ~2.6 TiB available
Python version: 3.6.13
Snakemake version: 6.4.1
Conda version: 24.11.3
pip version: 21.2.4

Thanks so much for your help!

Best,
Margo Ilina

[IlinaMargo]

Just to add — I launched the pipeline using:
blr run --cores 50

[pontushojer]

Strange that the log outputs 622 of 1557 steps (40%) done and then finishes without errors. I am thinking if this has something to do with a lack of large insertions. Is the file final.manta_large_insertions.vcf empty?

Also, could you provide a list of the files generated in the workdir and the chunks e.g. the output of ls -l /path/to/workdir and ls -l /path/to/workdir/chunks. Just so I have a better idea what has been run and not.

[IlinaMargo]

Thanks for the quick reply!
The file final.manta_large_insertions.vcf is not empty, it contains 995 lines.

As requested, here is the output of directories in txt file (because there are too much lines)

workdir_and_chunks_files.txt

[pontushojer]

Thanks, seem everything but the MTG-Link large insertion calling finished ok.

Could you list the contents of the mtglink_tmp directory as well.

Also, what are contents of the log file mtglink_tmp/read_subsampling_pre.log.

[IlinaMargo]

That is the content of mtglink_tmp directory:
drwxrwxr-x 2 milina milina 237568 Apr 17 09:40 insertion_fastas
drwxrwxr-x 2 milina milina 151552 Apr 17 10:37 read_subsampling_pre
-rw-rw-r-- 1 milina milina 103047 Apr 17 10:37 read_subsampling_pre.log

mtglink_tmp/read_subsampling_pre.log in the file below

read_subsampling_pre.log

[pontushojer]

Hi again, sorry for the slow reply.

I cannot find any obvious reason for the error yet. Could you list me the contents of mtglink_tmp/read_subsampling_pre directory? Any *.bxu files were generated for the next step.

Also did you try to resume the run? If not could you try just to see what happens.

[IlinaMargo]

Hi,
thank you for your help — you're actually one of the very few stLFR tool developers who replies at all, and I really appreciate it!

Here is the content of the mtglink_tmp/read_subsampling_pre directory you asked for:

read_subsampling_pre_content.txt

Regarding resuming the run — yes, I did try it before, but I don’t remember if it was with the exact same settings. I will try again now to see if anything changes.

It's interesting... I’m also wondering if the issue might be related to my data somehow, but it still seems a bit strange.

[IlinaMargo]

Good afternoon again,
yes, I remembered that I had actually tried resuming the run, and the pipeline completed about 99% of the steps before stopping.
Later, I reinstalled everything, which is why I didn’t see the same behavior again.

Here are the last messages from the log file:

[Fri Apr 25 16:25:08 2025]
Finished job 925.
922 of 935 steps (99%) done
[Fri Apr 25 16:25:09 2025]
Finished job 1472.
923 of 935 steps (99%) done
[Fri Apr 25 16:25:09 2025]
Finished job 1350.
924 of 935 steps (99%) done
[Fri Apr 25 16:25:18 2025]
Finished job 885.
925 of 935 steps (99%) done
[Fri Apr 25 16:26:19 2025]
Finished job 930.
926 of 935 steps (99%) done
[Fri Apr 25 16:26:25 2025]
Finished job 675.
927 of 935 steps (99%) done
[Fri Apr 25 16:26:25 2025]
Finished job 990.
928 of 935 steps (99%) done
[Fri Apr 25 16:26:27 2025]
Finished job 1005.
929 of 935 steps (99%) done
Select jobs to execute...

[Fri Apr 25 16:26:27 2025]
rule aggregate_extracts:
input: mtglink_tmp/read_subsampling_pre/mtglink.gfa.CP027623.1_12298942-13286736-L+_CP027623.1_13286836-13603271-R+.g100.flank10000.occ2.bxu.fastq, mtglink_tmp/read_subsampling_pre/mtglink.gfa.CP027623.1_182480-433383-L+_CP027623.1_433486-474845-R+.g103.flank10000.occ2.bxu.fastq
output: aggregate_extracts.done
jobid: 618

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /media/eternus1/projects/milina/genotrophs/stLFR/blr_output/.snakemake/log/2025-04-25T151340.000691.snakemake.log

Next, I tried running the tool again and got the same error message in the console that I had after the second restart:

output: aggregate_extracts.done
jobid: 618

Traceback (most recent call last):
File "/home/milina/miniconda3/envs/blr/lib/python3.6/site-packages/snakemake/init.py", line 785, in snakemake
keepmetadata=keep_metadata,
File "/home/milina/miniconda3/envs/blr/lib/python3.6/site-packages/snakemake/workflow.py", line 731, in execute
dag.update_checkpoint_dependencies()
File "/home/milina/miniconda3/envs/blr/lib/python3.6/site-packages/snakemake/dag.py", line 1312, in update_checkpoint_dependencies
self.postprocess()
File "/home/milina/miniconda3/envs/blr/lib/python3.6/site-packages/snakemake/dag.py", line 1193, in postprocess
self.cleanup()
File "/home/milina/miniconda3/envs/blr/lib/python3.6/site-packages/snakemake/dag.py", line 258, in cleanup
del self.depending[dep][job]
KeyError: index_bed
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /media/eternus1/projects/milina/genotrophs/stLFR/blr_output/.snakemake/log/2025-04-28T095457.227551.snakemake.log
Error in snakemake invocation: Command '['snakemake', '-s', '/home/milina/miniconda3/envs/blr/lib/python3.6/site-packages/blr/Snakefile', '--cores', '50', '--printshellcmds', '--use-conda', '--conda-prefix', '/home/milina/common_conda_envs/']' returned non-zero exit status 1.

Finally! there is a message of mistake...

[pontushojer]

Hi, thank you for your help — you're actually one of the very few stLFR tool developers who replies at all, and I really appreciate it!

Thank you for that!

Here is the content of the mtglink_tmp/read_subsampling_pre directory you asked for:

read_subsampling_pre_content.txt

So the *.bxu files are generated then I still not sure where the issue is.

On the rerun you get this error

KeyError: index_bed

This is strange as this rule is only used for generating the primary.bed.gz.tbi file atm, but this file allready exists based on you previous correspondance.

Based on everything it seems that there is some issue with the MTG-Link insertion calling but it is still unclear where exacltly. Could you try and run again but with the command:

blr run --cores 50 final.mtglink_insertions.vcf

Please send me the full output log if its not too large.

[IlinaMargo]

Hi again!
After re-running with your command, I encountered the following error in the CLI:
Traceback (most recent call last):
File "/home/milina/miniconda3/envs/blr/lib/python3.6/site-packages/snakemake/init.py", line 785, in snakemake
keepmetadata=keep_metadata,
File "/home/milina/miniconda3/envs/blr/lib/python3.6/site-packages/snakemake/workflow.py", line 731, in execute
dag.update_checkpoint_dependencies()
File "/home/milina/miniconda3/envs/blr/lib/python3.6/site-packages/snakemake/dag.py", line 1312, in update_checkpoint_dependencies
self.postprocess()
File "/home/milina/miniconda3/envs/blr/lib/python3.6/site-packages/snakemake/dag.py", line 1193, in postprocess
self.cleanup()
File "/home/milina/miniconda3/envs/blr/lib/python3.6/site-packages/snakemake/dag.py", line 258, in cleanup
del self.depending[dep][job]
KeyError: index_bed
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /media/eternus1/projects/milina/genotrophs/stLFR/blr_output/.snakemake/log/2025-05-05T172624.286077.snakemake.log
Error in snakemake invocation: Command '['snakemake', '-s', '/home/milina/miniconda3/envs/blr/lib/python3.6/site-packages/blr/Snakefile', '--cores', '50', '--printshellcmds', '--use-conda', '--conda-prefix', '/home/milina/common_conda_envs/', 'final.mtglink_insertions.vcf']' returned non-zero exit status 1.

And here is the related .log output:

Building DAG of jobs...
Updating job aggregate_extracts.
Updating job mtglink.
Using shell: /usr/bin/bash
Provided cores: 50
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 aggregate_extracts
1 create_mtglink_vcf
1 mtglink
3
Select jobs to execute...

[Mon May 5 17:21:16 2025]
rule aggregate_extracts:
input: mtglink_tmp/read_subsampling_pre/mtglink.gfa.CP027623.1_12298942-13286736-L+_CP027623.1_13286836-13603271-R+.g100.flank10000.occ2.bxu.fastq, mtglink_tmp/read_subsampling_pre/mtglink.gfa.CP027623.1_182480-433383-L+_CP027623.1_433486-474845-R+.g103.flank10000.occ2.bxu.fastq
output: aggregate_extracts.done
jobid: 555

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /media/eternus1/projects/milina/genotrophs/stLFR/blr_output/.snakemake/log/2025-05-05T172112.472578.snakemake.log

Let me know if there's anything else I should check or share.

Thanks again!

[pontushojer]

Seems this is a known snakemake issue for version 6.4.1 snakemake/snakemake#1024

In the linked issue it is stated that downgrading to snakemake 6.3.0 might work.

One other thing to try before then is to force the rule to rerun e.g. deleting the primary.bed.gz.tbi file and then trying to rerun the pipeline. Maybe you could test that first before we start to mess with dependencies.

[IlinaMargo]

Hi,

Unfortunately, none of the suggestions worked on my end, and I wasn't able to fully resolve the dependency issues despite several attempts.
Would it be possible to ask if you could provide a Docker container with BLR and all its dependencies pre-installed? That might help avoid these environment-related problems altogether.

Thanks so much again for your time and support!