Add skeleton for samplesheet generation #47

jfy133 · 2024-10-10T08:21:55Z

Closes #15

PR checklist

github-actions · 2024-10-10T08:23:35Z

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 488fcda

+| ✅ 194 tests passed       |+
#| ❔   2 tests were ignored |#
!| ❗  22 tests had warnings |!

❗ Test warnings:

files_exist - File not found: conf/igenomes.config
files_exist - File not found: conf/igenomes_ignored.config
readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
pipeline_todos - TODO string in nextflow.config: Specify your pipeline's command line flags
pipeline_todos - TODO string in README.md: TODO nf-core:
pipeline_todos - TODO string in README.md: Include a figure that guides the user through the major workflow steps. Many nf-core
pipeline_todos - TODO string in README.md: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
pipeline_todos - TODO string in README.md: update the following command to include all required parameters for a minimal example
pipeline_todos - TODO string in README.md: If applicable, make list of people who have also contributed
pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
pipeline_todos - TODO string in output.md: Write this documentation describing your workflow's output
pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
pipeline_todos - TODO string in test.config: Specify the paths to your test data on nf-core/test-datasets
pipeline_todos - TODO string in test.config: Give any required params for the test so that command line flags are not needed
pipeline_todos - TODO string in base.config: Check the defaults for all processes
pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required

❔ Tests ignored:

files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
actions_ci - actions_ci

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-createtaxdb_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-createtaxdb_logo_light.png
files_exist - File found: docs/images/nf-core-createtaxdb_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-createtaxdb_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowCreatetaxdb.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-schema plugin
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: validation.help.enabled
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable found: validation.help.beforeText
nextflow_config - Config variable found: validation.help.afterText
nextflow_config - Config variable found: validation.help.command
nextflow_config - Config variable found: validation.summary.beforeText
nextflow_config - Config variable found: validation.summary.afterText
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config variable (correctly) not found: params.max_cpus
nextflow_config - Config variable (correctly) not found: params.max_memory
nextflow_config - Config variable (correctly) not found: params.max_time
nextflow_config - Config variable (correctly) not found: params.validationFailUnrecognisedParams
nextflow_config - Config variable (correctly) not found: params.validationLenientMode
nextflow_config - Config variable (correctly) not found: params.validationSchemaIgnoreParams
nextflow_config - Config variable (correctly) not found: params.validationShowHiddenParams
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 1.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.malt_sequencetype= DNA
nextflow_config - Config default value correct: params.generate_samplesheet_dbtype= raw
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-createtaxdb_logo_light.png matches the template
files_unchanged - docs/images/nf-core-createtaxdb_logo_light.png matches the template
files_unchanged - docs/images/nf-core-createtaxdb_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 24.04.2, Config: 24.04.2
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (0 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: template_version_comment.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - KAIJU_MKFMI found in conf/base.config and Nextflow scripts.
modules_config - conf/modules.config found and not ignored.
modules_config - MULTIQC found in conf/modules.config and Nextflow scripts.
modules_config - CAT_CAT_DNA found in conf/modules.config and Nextflow scripts.
modules_config - CAT_CAT_AA found in conf/modules.config and Nextflow scripts.
modules_config - BRACKEN_BUILD found in conf/modules.config and Nextflow scripts.
modules_config - CENTRIFUGE_BUILD found in conf/modules.config and Nextflow scripts.
modules_config - DIAMOND_MAKEDB found in conf/modules.config and Nextflow scripts.
modules_config - KAIJU_MKFMI found in conf/modules.config and Nextflow scripts.
modules_config - KRAKEN2_ADD found in conf/modules.config and Nextflow scripts.
modules_config - KRAKEN2_BUILD found in conf/modules.config and Nextflow scripts.
modules_config - KRAKENUNIQ_BUILD found in conf/modules.config and Nextflow scripts.
modules_config - MALT_BUILD found in conf/modules.config and Nextflow scripts.
modules_config - TAR found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.0.2

Run details

nf-core/tools version 3.0.2
Run at 2024-10-22 11:12:40

CarsonJM · 2024-10-10T14:04:48Z

subworkflows/local/generate_downstream_samplesheets/main.nf

+        format = 'csv'
+        format_sep = ','


Does this need to be defined twice?

No, I was leaving it in as an example but can remove this now :)

…tory, update snapshot

jfy133 · 2024-10-11T11:43:50Z

Thanks for adding the test @CarsonJM !

…ix bracken test warning by installing untar module, update tests

jfy133 · 2024-10-12T13:41:02Z

TODO:
- Add TAR support to generate tar.gz versions (partly addresses Generate taxprofiler samplesheet #15)
- Make the nf-test of generate_samplesheet properly using remote test data (use outpt from previous tep)
- Test output samplehseet against taxprofielr

jfy133 · 2024-10-14T12:39:28Z

OK tested, and a createtaxdb generated samplesheet is accepted with taxprofiler!

Joon-Klaps

A minor suggestion regarding the new subworkflow which will allow easier integration of more downstream pipelines for samplesheet generation

subworkflows/local/generate_downstream_samplesheets/main.nf

…tiple pipelines samplesheet generation

subworkflows/local/generate_downstream_samplesheets/tests/main.nf.test.snap

Co-authored-by: Joon Klaps <[email protected]>

…om check)

Joon-Klaps

Minor suggestions.

conf/test_nothing.config

subworkflows/local/utils_nfcore_createtaxdb_pipeline/main.nf

workflows/createtaxdb.nf

Co-authored-by: Joon Klaps <[email protected]>

workflows/createtaxdb.nf

Joon-Klaps

LGTM

Joon-Klaps

Nice tutorial - I think its clear.

CarsonJM

This looks really good! Nice idea using the tar outputs 🎉

CarsonJM · 2024-10-17T12:52:09Z

subworkflows/local/generate_downstream_samplesheets/main.nf

+    }
+}
+
+def channelToSamplesheet(ch_list_for_samplesheet, path, format) {


This looks really clean now!

CarsonJM · 2024-10-17T12:53:50Z

subworkflows/local/generate_downstream_samplesheets/tests/main.nf.test

I think this looks good 👍

nextflow.config

… contents can be found by KRAKEN2_BUILD with container mounting

jfy133 · 2024-10-25T12:24:02Z

TODO: complete tutorial (double check commands, outputs etc.)

Add template

0ac36bf

jfy133 marked this pull request as draft October 10, 2024 09:04

jfy133 added 5 commits October 10, 2024 11:08

Resolve the full path of the db

1947e20

Start adding new parameters everywhere

edcff76

Add output docs, fix kraken2/bracken and a warning

8086eba

Remoe debugging view

bc91756

Tweak schema

4c87427

jfy133 linked an issue Oct 10, 2024 that may be closed by this pull request

Generate taxprofiler samplesheet #15

Open

CarsonJM self-assigned this Oct 10, 2024

CarsonJM reviewed Oct 10, 2024

View reviewed changes

jfy133 and others added 3 commits October 10, 2024 20:52

Fix MALT missing map, update Centrifuge build to put files in a direc…

ea00267

…tory, update snapshot

First pass at adding nf-test

9800e76

Add samplesheet to tests

981e822

jfy133 added 2 commits October 11, 2024 14:07

Restructure generate_samplesheet subworkflow to new draft standard, f…

5af6ed0

…ix bracken test warning by installing untar module, update tests

Merge branch 'dev' into generate-samplesheet

d8ad456

jfy133 added 5 commits October 12, 2024 15:12

Udpdate snap due to change in centriguie build output

b8b003f

Update snapshot because outdir test directory changes per context

30164b0

Add TAR support for creating TARd archives

f71c6cd

update test profiole

a0f7385

Add working nf-test for the subworkflow

3128c72

jfy133 marked this pull request as ready for review October 14, 2024 12:38

Change default samplesheet to generate for to false, and update test

e4184bc

Joon-Klaps reviewed Oct 14, 2024

View reviewed changes

Replace single string/option with comma separated list to support mul…

c98514a

…tiple pipelines samplesheet generation

Joon-Klaps reviewed Oct 14, 2024

View reviewed changes

subworkflows/local/generate_downstream_samplesheets/tests/main.nf.test.snap Outdated Show resolved Hide resolved

Apply suggestions from code review

4e52e0a

Co-authored-by: Joon Klaps <[email protected]>

More changes after review from @Joon-Klaps (use schema regex not cust…

818a1f1

…om check)

Joon-Klaps reviewed Oct 14, 2024

View reviewed changes

conf/test_nothing.config Outdated Show resolved Hide resolved

subworkflows/local/utils_nfcore_createtaxdb_pipeline/main.nf Outdated Show resolved Hide resolved

workflows/createtaxdb.nf Show resolved Hide resolved

jfy133 and others added 2 commits October 14, 2024 16:06

Apply suggestions from code review

656d6e3

Co-authored-by: Joon Klaps <[email protected]>

Apply suggestions from code review

db6ddfc

Co-authored-by: Joon Klaps <[email protected]>

jfy133 commented Oct 14, 2024

View reviewed changes

workflows/createtaxdb.nf Outdated Show resolved Hide resolved

jfy133 added 2 commits October 14, 2024 16:08

Update workflows/createtaxdb.nf

18bcb3f

Fix default in schema and config

6d9f011

jfy133 requested review from Joon-Klaps and CarsonJM October 15, 2024 07:17

Joon-Klaps approved these changes Oct 15, 2024

View reviewed changes

jfy133 added 3 commits October 15, 2024 11:33

Bug fixes for optional auxiliary, and start adding tutorial

7729415

Update tutorial, add tutorail directory to gitignore

5368c72

Use Joon's latest function

70387f0

Joon-Klaps approved these changes Oct 15, 2024

View reviewed changes

Formatting and fix URL

7ce0358

CarsonJM approved these changes Oct 17, 2024

View reviewed changes

jfy133 added 3 commits October 17, 2024 15:32

Fix schema in preparation for new nf-schema patch

dbe44c1

Bump version of schema

bdc765e

Set KRAKEN2_ADD to hardlink input to ensure symlinked taxonomy folder…

488fcda

… contents can be found by KRAKEN2_BUILD with container mounting

Joon-Klaps mentioned this pull request Oct 29, 2024

create template GENERATE_DOWNSTREAM_SAMPLESHEETS for tools nf-core/tools#3259

Open

jfy133 marked this pull request as draft January 31, 2025 14:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add skeleton for samplesheet generation #47

Add skeleton for samplesheet generation #47

jfy133 commented Oct 10, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

CarsonJM Oct 10, 2024

jfy133 Oct 11, 2024

jfy133 commented Oct 11, 2024

jfy133 commented Oct 12, 2024 •

edited

Loading

jfy133 commented Oct 14, 2024

Joon-Klaps left a comment

Joon-Klaps left a comment

Joon-Klaps left a comment

Joon-Klaps left a comment

CarsonJM left a comment

CarsonJM Oct 17, 2024

CarsonJM Oct 17, 2024

jfy133 commented Oct 25, 2024

Add skeleton for samplesheet generation #47

Are you sure you want to change the base?

Add skeleton for samplesheet generation #47

Conversation

jfy133 commented Oct 10, 2024 • edited Loading

PR checklist

github-actions bot commented Oct 10, 2024 • edited Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

CarsonJM Oct 10, 2024

Choose a reason for hiding this comment

jfy133 Oct 11, 2024

Choose a reason for hiding this comment

jfy133 commented Oct 11, 2024

jfy133 commented Oct 12, 2024 • edited Loading

jfy133 commented Oct 14, 2024

Joon-Klaps left a comment

Choose a reason for hiding this comment

Joon-Klaps left a comment

Choose a reason for hiding this comment

Joon-Klaps left a comment

Choose a reason for hiding this comment

Joon-Klaps left a comment

Choose a reason for hiding this comment

CarsonJM left a comment

Choose a reason for hiding this comment

CarsonJM Oct 17, 2024

Choose a reason for hiding this comment

CarsonJM Oct 17, 2024

Choose a reason for hiding this comment

jfy133 commented Oct 25, 2024

jfy133 commented Oct 10, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024 •

edited

Loading

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

jfy133 commented Oct 12, 2024 •

edited

Loading