Fix meta.yml structure for input and output tuples

[ewels]

The current contents of the meta.yml files does not correspond to the actual processes in most modules. This limits how useful these metadata files can be.

For example, consider abacas. The meta.yml file shows 3 inputs:

modules/modules/nf-core/abacas/meta.yml

Lines 17 to 30 in 53a9794

	input:
	- meta:
	type: map
	description: |
	Groovy Map containing sample information
	e.g. [ id:'test', single_end:false ]
	- scaffold:
	type: file
	description: Fasta file containing scaffold
	pattern: "*.{fasta,fa}"
	- fasta:
	type: file
	description: FASTA reference file
	pattern: "*.{fasta,fa}"

However, there are only 2 inputs in main.nf (the first one being a tuple with two elements):

modules/modules/nf-core/abacas/main.nf

Lines 10 to 12 in 53a9794

	input:
	tuple val(meta), path(scaffold)
	path fasta

The correct YAML structure to describe this should be:

 input: 
   -
       - meta: 
           type: map 
           description: | 
             Groovy Map containing sample information 
             e.g. [ id:'test', single_end:false ] 
       - scaffold: 
           type: file 
           description: Fasta file containing scaffold 
           pattern: "*.{fasta,fa}" 
   - fasta: 
       type: file 
       description: FASTA reference file 
       pattern: "*.{fasta,fa}" 

The difference is pretty difficult to see in YAML (*cough* see #11 *cough*), but now we have two inputs. The first is an array containing two elements.

This error seems to be present in basically any modules containing a tuple, as all meta.yml files seems to have a flat array of inputs.

We should be doing better automation of this file from nf-core/tools in order to solve this problem. I started a brief proof of concept on how we can do this in nf-core/tools#2789 and described a rough outline of what needs doing.

[nvnieuwk]

I fully agree with this! Maybe using [] instead of - for arrays will make this a bit more clear to read?

[nvnieuwk]

Although I'm not sure it's possible to create multiline lists that way

[nvnieuwk]

https://www.w3schools.io/file/yaml-multiline-arrays/

[mashehu]

tbh: also quite easy to overlook what is inside the array in json:

{
  "input": [
    [
      {
        "meta": {
          "type": "map",
          "description": "Groovy Map containing sample information \ne.g. [ id:'test', single_end:false ] \n"
        }
      },
      {
        "scaffold": {
          "type": "file",
          "description": "Fasta file containing scaffold",
          "pattern": "*.{fasta,fa}"
        }
      }
    ],
    {
      "fasta": {
        "type": "file",
        "description": "FASTA reference file",
        "pattern": "*.{fasta,fa}"
      }
    }
  ]
}

how about adding comment, so it is not just one lonely dash, helping to distinguish things:

 input: 
   - #input tuple
       - meta: 
           type: map 
           description: | 
             Groovy Map containing sample information 
             e.g. [ id:'test', single_end:false ] 
       - scaffold: 
           type: file 
           description: Fasta file containing scaffold 
           pattern: "*.{fasta,fa}" 
   - fasta: 
       type: file 
       description: FASTA reference file 
       pattern: "*.{fasta,fa}" 

[ewels]

tbh: also quite easy to overlook what is inside the array in json:

Overlook yes, but harder to break it by accidentally messing with indentation at least. Like the comment idea 👍🏻

@nvnieuwk - remember that we will be running stuff through Prettier which has a habit of destroying any fancy formatting like that 😉 But yes - JSON is valid YAML, so makes sense that we could write it in a JSON-like way.

I'm not super fussed about the language we use - hopefully we can do more automation and linting so will pick up anything that looks wrong automatically. Main thing is to fix the structure.

[nvnieuwk]

I like @mashehu's idea a lot, maybe the best of both worlds (in YAML)

[jfy133]

@pinin4fjords (maintainers team disccusion) suggests not a comment, but just duplicate the name first element of tuple

 input: 
   - scaffold
       - meta: 
           type: map 
           description: | 
             Groovy Map containing sample information 
             e.g. [ id:'test', single_end:false ] 
       - scaffold: 
           type: file 
           description: Fasta file containing scaffold 
           pattern: "*.{fasta,fa}" 
   - fasta: 
       type: file 
       description: FASTA reference file 
       pattern: "*.{fasta,fa}" 

OR

input: 
  - bam
      - meta: 
          type: map 
          description: | 
            Groovy Map containing sample information 
            e.g. [ id:'test', single_end:false ] 
      - bam: 
          type: file 
          description: BAM
          pattern: "*.{bam},
      - bai: 
          type: file 
          description: BAM index file
          pattern: "*.{bai}" 
  - fasta: 
      type: file 
      description: FASTA reference file 
      pattern: "*.{fasta,fa}" 

[JoseEspinosa]

I would be more inclined to adopt @mashehu suggestion of using a comment and it seems to me more machine-readable. My point is that if you use the first element of the tuple what if you have a channel that has a structure that encodes something different in the the first element of the tuple. Maybe the indentation is enough though to parse it...
Anyhow, my point is that we should keep in mind that at some point we may want to use these files to automatically generate a pipeline or subworkflow and it would be nice that the yaml file is as machine-readable as possible.

[ewels]

Agree, my vote is YAML with comment or switch to JSON.