Feat/workflow updates #414

sateeshperi · 2024-09-06T16:39:54Z

module updates
sub-wf updates
remove modules.config -> individual nextflow.config
re-organization
#416
#361
#328
#417

… index

github-actions · 2024-09-06T16:41:27Z

`nf-core lint` overall result: Passed ✅

Posted for pipeline commit 0cedbce

+| ✅ 187 tests passed       |+
#| ❔   4 tests were ignored |#

❔ Tests ignored:

files_exist - File is ignored: lib/nfcore_external_java_deps.jar
files_exist - File is ignored: conf/modules.config
actions_ci - actions_ci
modules_config - modules_config

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-methylseq_logo_light.png
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-methylseq_logo_light.png
files_exist - File found: docs/images/nf-core-methylseq_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-methylseq_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowMethylseq.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 2.7.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
nextflow_config - Config default value correct: params.aligner= bismark
nextflow_config - Config default value correct: params.clip_r1= 0
nextflow_config - Config default value correct: params.clip_r2= 0
nextflow_config - Config default value correct: params.three_prime_clip_r1= 0
nextflow_config - Config default value correct: params.three_prime_clip_r2= 0
nextflow_config - Config default value correct: params.nextseq_trim= 0
nextflow_config - Config default value correct: params.num_mismatches= 0.6
nextflow_config - Config default value correct: params.no_overlap= true
nextflow_config - Config default value correct: params.ignore_r1= 0
nextflow_config - Config default value correct: params.ignore_r2= 2
nextflow_config - Config default value correct: params.ignore_3prime_r1= 0
nextflow_config - Config default value correct: params.ignore_3prime_r2= 2
nextflow_config - Config default value correct: params.min_depth= 0
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 16
nextflow_config - Config default value correct: params.max_memory= 128.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/methylseq
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-methylseq_logo_light.png matches the template
files_unchanged - docs/images/nf-core-methylseq_logo_light.png matches the template
files_unchanged - docs/images/nf-core-methylseq_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_todos - No TODO strings found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (318 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - BISMARK_ALIGN found in conf/base.config and Nextflow scripts.
base_config - BISMARK_DEDUPLICATE found in conf/base.config and Nextflow scripts.
base_config - BISMARK_METHYLATIONEXTRACTOR found in conf/base.config and Nextflow scripts.
base_config - BWAMETH_ALIGN found in conf/base.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 2.14.1

Run details

nf-core/tools version 2.14.1
Run at 2024-09-23 14:45:25

conda update all --yes

Using this to confirm that's where the bug is

Add missing type field to entry in nextflow_schema.json

…snapshots

edmundmiller

Awesome, those testing util functions save so much duplication and manual work!

workflows/methylseq/main.nf

Co-authored-by: Edmund Miller <[email protected]>

sateeshperi added 12 commits September 2, 2024 10:09

all module updates

d88f89a

update wf channels to match module updates

ddaf8b3

fix bwameth fasta value ch by remapping

34ddc79

update pipeline test snapshots

982e7b2

add bwameth-align methyldackel tests and add GRCh37,8 bwameth igenome…

d168cf3

… index

move prepare genome to main

2bffd87

fix ch_bwameth_index assign bug in prepare genome

3dbc090

rm fasta and index init from nextflow.config

77f2f75

update samtools/faidx inputs

29c4dfd

reorg modules.config to separate configs and module updates

7b276c9

reorg sub-wf folders

03967bd

rm modules.config and harshil align

33891bd

sateeshperi self-assigned this Sep 6, 2024

sateeshperi added this to the 2.7.0 milestone Sep 6, 2024

sateeshperi and others added 15 commits September 6, 2024 16:48

update .nf-core.yml, ignore modules.config

b0d03cb

nf-core sub-wf update

1da66a1

update tests organization

5f1a2cd

add bismark dump tags

a163515

move fasta_index ch creation out of bwameth scope

34ea759

prepare_genome local sub-wf tests

725d0ea

mv fasta_index to bwameth scope and update tests

1e8aff2

separate sub-wf configs

5967bd0

bismark local sub-wf test

f83b3c3

bwameth local sub-wf test

f9d9866

Update .gitpod.yml

8e4befd

conda update all --yes

pipeline level test snaps

8098992

update gitpod base to dev

0cac2e6

test: Skip deduplication

3ffcd77

Using this to confirm that's where the bug is

fix: Update syntax to deal with the meta ids

82573ea

sateeshperi and others added 6 commits September 18, 2024 15:32

update samtools mutliqc modules

6b06c1e

update test snaps after samtools update

916a47a

use nft-bam plugin - init default

428ae98

update UTILS.groovy and snaps

c726471

added type field to entry in nextflow_schema.json to address issue #361

1e36141

update CHANGELOG.md

663b0aa

sateeshperi linked an issue Sep 21, 2024 that may be closed by this pull request

samtools sort overwrites #328

Open

sateeshperi and others added 8 commits September 21, 2024 13:04

fix bismark samtools_sort overwrite #328

785ca56

bwameth samtool_index capture only index

7b2f2fe

update pipeline default test snap

92a4b70

add save_align_intermeds test snap

a8cce4e

module meta updates

6211ce4

Merge branch 'feat/workflow-updates' into dev

bb87872

Merge pull request #417 from scoughlan2/dev

a35d608

Add missing type field to entry in nextflow_schema.json

separate bismark tests

a02aa68

This was linked to issues Sep 21, 2024

bwameth pipeline skipping all bwameth steps when --fasta, --fasta_index, and --bwa_meth_index are provided. #396

Open

Missing type field in nextflow_schema.json file #361

Open

Something ain't right with that template... #309

Open

sateeshperi added 3 commits September 22, 2024 04:06

add dedup samtools_index step and update default test

63e4153

add UTILS_PIPELINE.groovy and semi-automate pipeline level tests and …

b837413

…snapshots

update pipeline level tests and snaps

8ddcf6d

sateeshperi marked this pull request as ready for review September 22, 2024 17:59

sateeshperi requested a review from a team as a code owner September 22, 2024 17:59

sateeshperi requested a review from FelixKrueger September 22, 2024 17:59

update workflow trace size assert and test configs + CHANGELOG

471d45d

sateeshperi requested a review from edmundmiller September 23, 2024 11:14

edmundmiller approved these changes Sep 23, 2024

View reviewed changes

workflows/methylseq/main.nf Outdated Show resolved Hide resolved

Update workflows/methylseq/main.nf

0cedbce

Co-authored-by: Edmund Miller <[email protected]>

sateeshperi merged commit 3f19e77 into dev Sep 23, 2024
7 checks passed

sateeshperi deleted the feat/workflow-updates branch September 23, 2024 15:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/workflow updates #414

Feat/workflow updates #414

sateeshperi commented Sep 6, 2024 •

edited

Loading

github-actions bot commented Sep 6, 2024 •

edited

Loading

❔ Tests ignored:

✅ Tests passed:

Run details

edmundmiller left a comment

Feat/workflow updates #414

Feat/workflow updates #414

Conversation

sateeshperi commented Sep 6, 2024 • edited Loading

github-actions bot commented Sep 6, 2024 • edited Loading

nf-core lint overall result: Passed ✅

❔ Tests ignored:

✅ Tests passed:

Run details

edmundmiller left a comment

Choose a reason for hiding this comment

sateeshperi commented Sep 6, 2024 •

edited

Loading

github-actions bot commented Sep 6, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅