Skip to content

DRAM2 launches way to many SLURM jobs per input fasta #448

@madeline-scyphers

Description

@madeline-scyphers

Description of the bug

DRAM2 launches 1 job per input file for many steps in the pipeline (1 job per prodigal, 1 job per kegg search, 1 job per camper search, etc.). This does not scale at all with large numbers of mags or assemblies. We need to combine inputs, either by concatenating them together (after rename to make unique headers) or run things in for loops inside jobs (potentationly certain processes might be better one way or another). We obviously want to still partition things out to take advantage of the queue. And if concatenating, we probably want to try and make sure the concatenated file stays inside the provisioned memory, or we risk major slow downs.

System information

All versions of DRAM2 currently

Command used and terminal output

Relevant files

No response

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

Status

Todo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions