Text/workflows dec19 #26
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Document all 11 steps of the scRNA-seq analysis pipeline with: - Parameters and default values - Algorithms used (DESeq2, Harmony, GSEA, etc.) - Input/output specifications - Workflow diagram - Output directory structure 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Create prefect_workflow/config/config.yaml with all workflow parameters - Add load_config() function to flows.py for YAML config loading - Update run_workflow() and analyze_single_cell_type() to accept config_path - Add --config CLI argument to run_workflow.py - Change Prefect output folder from "workflow" to "wf_prefect" - Change Snakemake output folder from "workflow" to "wf_snakemake" - Update WORKFLOW_OVERVIEW.md to reflect new output structure 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Create RUNNING_WORKFLOWS.md covering all four workflow implementations: - Monolithic: single script, no checkpointing - Modular: reusable functions, basic skip functionality - Stateless: BIDS checkpointing, execution logging - Prefect: config file, all cell types, orchestration - Snakemake: dynamic rules, parallel execution Includes prerequisites, CLI options, configuration examples, output locations, and comparison table. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add report/ directory with RST caption files for each figure type - Add global report declaration to Snakefile - Update preprocessing.smk rules to declare figures as outputs with report() wrapper - Update pseudobulk.smk checkpoint to include figure with report() wrapper - Update per_cell_type.smk rules to include figures with report() wrapper and subcategory - Update common.smk aggregate function to include figure files - Add report and report-zip targets to Makefile - Update WORKFLOW_OVERVIEW.md with report generation documentation Usage: snakemake --report report.html --config datadir=/path/to/data 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Create a simple pandas-based data analysis workflow to demonstrate workflow manager features. The workflow loads Self-Regulation Ontology data, filters to numerical columns, joins datasets, computes Spearman correlations, and generates a clustered heatmap. Includes: - Core modules: load_data, filter_data, join_data, correlation, visualization - Prefect workflow with tasks and flows - Snakemake workflow with rules and report generation - GNU Make workflow with Makefile and CLI scripts 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The doublet_detection_umap.png was declared as an output in the Snakemake workflow but never generated because: 1. plot_doublets() was never called in run_qc_pipeline() 2. No UMAP coordinates existed at the QC step Added compute_umap_for_qc() to compute a simple UMAP embedding specifically for doublet visualization. This runs on a temporary copy to preserve raw counts, then copies coordinates back. Updated run_qc_pipeline() to compute UMAP and plot doublets after detection but before filtering, so both doublets and singlets are visible in the visualization. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Caption paths were relative to the rules/ directory but pointed to report/ instead of ../report/. This caused "caption file not found" errors when generating Snakemake reports. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Scripts were using snakemake.output[0] but the rules have named outputs (checkpoint=, fig_*=). This caused checkpoint files to not be saved to the correct path. Changed to snakemake.output.checkpoint in filter.py, qc.py, dimred.py, and cluster.py. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This represents an incomplete version of the workflows directory. it's mostly done but still need work in a number of sections. I'm merging it now because I want to work on other sections before coming back to this, and I would like to have the rest of the code changes merged. Keeping this chapter out of the rendered book for now since it's still half-baked.