From 2ff9d953357d319e02b4dfe5fe3ff4863d011c2a Mon Sep 17 00:00:00 2001 From: John Bradley Date: Wed, 8 Mar 2023 09:16:03 -0500 Subject: [PATCH] Add questions/objectives to remaining episodes --- episodes/04_setup_project.md | 11 +++++++++++ episodes/05_setup_env.md | 13 +++++++++++++ episodes/10_reuse_workflow.md | 24 ++++++++++++++++++++---- episodes/11_summary_rule.md | 19 +++++++++++++------ episodes/12_run_on_slurm.md | 25 +++++++++++++++++++++++++ 5 files changed, 82 insertions(+), 10 deletions(-) diff --git a/episodes/04_setup_project.md b/episodes/04_setup_project.md index d24d277a..6274d418 100644 --- a/episodes/04_setup_project.md +++ b/episodes/04_setup_project.md @@ -3,6 +3,17 @@ title: "Project Setup" teaching: 1 exercises: 5 --- +:::::::::::::::::::::::::::::::::::::: questions + +- How do I create a directory with the necessary files for this class? + +:::::::::::::::::::::::::::::::::::::::::::::::: +::::::::::::::::::::::::::::::::::::: objectives + +- Create a "SnakemakeWorkflow" directory with scripts and cached container images + +:::::::::::::::::::::::::::::::::::::::::::::::: + ## Log into OSC - Visit https://ondemand.osc.edu/. diff --git a/episodes/05_setup_env.md b/episodes/05_setup_env.md index cb110c36..d3fef253 100644 --- a/episodes/05_setup_env.md +++ b/episodes/05_setup_env.md @@ -3,6 +3,19 @@ title: "Environment Setup" teaching: 1 exercises: 4 --- +:::::::::::::::::::::::::::::::::::::: questions + +- How do I activate the Snakemake for this class? +- How do I start an interactive job to run snakemake on a worker node? + +:::::::::::::::::::::::::::::::::::::::::::::::: +::::::::::::::::::::::::::::::::::::: objectives + +- Setup the shell environment +- Verify snakemake and R versions +- Start an interactive session on a worker node + +:::::::::::::::::::::::::::::::::::::::::::::::: ## Software Environment Setup diff --git a/episodes/10_reuse_workflow.md b/episodes/10_reuse_workflow.md index 45384f17..0455e482 100644 --- a/episodes/10_reuse_workflow.md +++ b/episodes/10_reuse_workflow.md @@ -3,14 +3,28 @@ title: "Reuse another workflow" teaching: 10 exercises: 2 --- +:::::::::::::::::::::::::::::::::::::: questions +- What do I need to do to re-use another Snakemake workflow? + +:::::::::::::::::::::::::::::::::::::::::::::::: +::::::::::::::::::::::::::::::::::::: objectives + +- Add a module to our Snakefile to re-use parts of the BGNN_Core_Workflow workflow +- Make changes so our workflow will be filename compatible + +:::::::::::::::::::::::::::::::::::::::::::::::: + +## Reusing Another Workflow To re-use a workflow you generally need: -- where to find workflow - github "hdr-bgnn/BGNN_Core_Workflow" -- what is the relative path to the Snakefile "workflow/Snakefile" -- what tag or version to use "1.0.0" -- what file naming convention is the workflow using +- Where to find workflow? - github "hdr-bgnn/BGNN_Core_Workflow" +- What is the relative path to the Snakefile? "workflow/Snakefile" +- What tag or version to use? "1.0.0" +- What file naming convention is the workflow using? +- What dependencies must be manually installed? +See source code for the workflow https://github.com/hdr-bgnn/BGNN_Core_Workflow/blob/main/workflow/Snakefile#L19 ``` @@ -70,3 +84,5 @@ use rule segment_image from bgnn_core ```bash snakemake -c1 --use-singularity Segmented/bj373514_segmented.png ``` + + diff --git a/episodes/11_summary_rule.md b/episodes/11_summary_rule.md index f1aecd09..aea64428 100644 --- a/episodes/11_summary_rule.md +++ b/episodes/11_summary_rule.md @@ -4,6 +4,19 @@ teaching: 10 exercises: 2 --- +:::::::::::::::::::::::::::::::::::::: questions + +- How can we specify a rule that has many dynamic input files? + +:::::::::::::::::::::::::::::::::::::::::::::::: +::::::::::::::::::::::::::::::::::::: objectives + +- Add a summary rule that requires Segmented images +- Use expand function to simplify creating filenames + +:::::::::::::::::::::::::::::::::::::::::::::::: + + Add summary_report to config.yaml: ```bash cat Scripts/SummaryReport.R @@ -43,10 +56,4 @@ rule summary: ``` -Where did my logs go? -```bash -ls logs -ls logs/ -``` - diff --git a/episodes/12_run_on_slurm.md b/episodes/12_run_on_slurm.md index 4a87122e..6c4310c8 100644 --- a/episodes/12_run_on_slurm.md +++ b/episodes/12_run_on_slurm.md @@ -4,6 +4,22 @@ teaching: 10 exercises: 2 --- +:::::::::::::::::::::::::::::::::::::: questions + +- How can can I add memory, cpu, and gpu requirements to a rule? +- How can can I scale up my workflow in a cluster? +- What logs are created when running my workflow with a cluster? + +:::::::::::::::::::::::::::::::::::::::::::::::: +::::::::::::::::::::::::::::::::::::: objectives + +- Add a memory requirement to a rule +- Create sbatch script to run a workflow at scale +- Delete all outputs +- Run the workflow at scale + +:::::::::::::::::::::::::::::::::::::::::::::::: + ## Problems with way we have been running snakemake - Only using a single node so limited scaling - Must keep our terminal window connected or the job might stop @@ -53,6 +69,15 @@ Run snakemake in the background scaling up sbatch run-workflow.sh -c10 ``` +## Monitor job ``` squeue -u $LOGNAME ``` + +## Notice new job logs + +Where did my logs go? +```bash +ls logs +ls logs/ +```