Skip to content

Commit

Permalink
Add questions/objectives to remaining episodes
Browse files Browse the repository at this point in the history
  • Loading branch information
johnbradley committed Mar 8, 2023
1 parent ac3f06e commit 2ff9d95
Show file tree
Hide file tree
Showing 5 changed files with 82 additions and 10 deletions.
11 changes: 11 additions & 0 deletions episodes/04_setup_project.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,17 @@ title: "Project Setup"
teaching: 1
exercises: 5
---
:::::::::::::::::::::::::::::::::::::: questions

- How do I create a directory with the necessary files for this class?

::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::: objectives

- Create a "SnakemakeWorkflow" directory with scripts and cached container images

::::::::::::::::::::::::::::::::::::::::::::::::


## Log into OSC
- Visit https://ondemand.osc.edu/.
Expand Down
13 changes: 13 additions & 0 deletions episodes/05_setup_env.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,19 @@ title: "Environment Setup"
teaching: 1
exercises: 4
---
:::::::::::::::::::::::::::::::::::::: questions

- How do I activate the Snakemake for this class?
- How do I start an interactive job to run snakemake on a worker node?

::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::: objectives

- Setup the shell environment
- Verify snakemake and R versions
- Start an interactive session on a worker node

::::::::::::::::::::::::::::::::::::::::::::::::

## Software Environment Setup

Expand Down
24 changes: 20 additions & 4 deletions episodes/10_reuse_workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,28 @@ title: "Reuse another workflow"
teaching: 10
exercises: 2
---
:::::::::::::::::::::::::::::::::::::: questions

- What do I need to do to re-use another Snakemake workflow?

::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::: objectives

- Add a module to our Snakefile to re-use parts of the BGNN_Core_Workflow workflow
- Make changes so our workflow will be filename compatible

::::::::::::::::::::::::::::::::::::::::::::::::

## Reusing Another Workflow
To re-use a workflow you generally need:

- where to find workflow - github "hdr-bgnn/BGNN_Core_Workflow"
- what is the relative path to the Snakefile "workflow/Snakefile"
- what tag or version to use "1.0.0"
- what file naming convention is the workflow using
- Where to find workflow? - github "hdr-bgnn/BGNN_Core_Workflow"
- What is the relative path to the Snakefile? "workflow/Snakefile"
- What tag or version to use? "1.0.0"
- What file naming convention is the workflow using?
- What dependencies must be manually installed?

See source code for the workflow
https://github.com/hdr-bgnn/BGNN_Core_Workflow/blob/main/workflow/Snakefile#L19

```
Expand Down Expand Up @@ -70,3 +84,5 @@ use rule segment_image from bgnn_core
```bash
snakemake -c1 --use-singularity Segmented/bj373514_segmented.png
```


19 changes: 13 additions & 6 deletions episodes/11_summary_rule.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,19 @@ teaching: 10
exercises: 2
---

:::::::::::::::::::::::::::::::::::::: questions

- How can we specify a rule that has many dynamic input files?

::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::: objectives

- Add a summary rule that requires Segmented images
- Use expand function to simplify creating filenames

::::::::::::::::::::::::::::::::::::::::::::::::


Add summary_report to config.yaml:
```bash
cat Scripts/SummaryReport.R
Expand Down Expand Up @@ -43,10 +56,4 @@ rule summary:
```


Where did my logs go?
```bash
ls logs
ls logs/
```


25 changes: 25 additions & 0 deletions episodes/12_run_on_slurm.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,22 @@ teaching: 10
exercises: 2
---

:::::::::::::::::::::::::::::::::::::: questions

- How can can I add memory, cpu, and gpu requirements to a rule?
- How can can I scale up my workflow in a cluster?
- What logs are created when running my workflow with a cluster?

::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::: objectives

- Add a memory requirement to a rule
- Create sbatch script to run a workflow at scale
- Delete all outputs
- Run the workflow at scale

::::::::::::::::::::::::::::::::::::::::::::::::

## Problems with way we have been running snakemake
- Only using a single node so limited scaling
- Must keep our terminal window connected or the job might stop
Expand Down Expand Up @@ -53,6 +69,15 @@ Run snakemake in the background scaling up
sbatch run-workflow.sh -c10
```

## Monitor job
```
squeue -u $LOGNAME
```

## Notice new job logs

Where did my logs go?
```bash
ls logs
ls logs/
```

0 comments on commit 2ff9d95

Please sign in to comment.