🚸 Simplify tags to declare rules/test cases

.github/workflows/poltergust.yaml

@@ -20,8 +20,8 @@ jobs:
     - name: Install poltergust
       run: npm ci
 
-    - name: Test the example ruleset
-      run: npx poltergust test ./example/valid
+    - name: Test the valid example ruleset
+      run: npx poltergust test ./examples/valid
 
   build-and-test-invalid-example:
     needs: build-and-test-valid-example
@@ -38,5 +38,5 @@ jobs:
       run: npm ci
 
     - name: Test the invalid example ruleset (should exit 1)
-      run: npx poltergust test ./example/invalid || (exitcode=$?; [ $exitcode -eq 1 ] || exit $exitcode)
+      run: npx poltergust test ./examples/invalid || (exitcode=$?; [ $exitcode -eq 1 ] || exit $exitcode)
       shell: bash

README.md

@@ -2,48 +2,23 @@
 
 # 👻 poltergust
 
-An npm CLI to extract Spectral rules from .md files, merge them and test them.
+An npm CLI to extract, test and merge Spectral rules from .md files.
 
-- Each yaml codeblock starting by `#spectral` is considered a spectral rule and will be aggregated in the spectral.yaml file
-- Each yaml codeblock starting by `#✅-test-for: some-rule-name` is considered as an OpenAPI snippet that not should fail the corresponding spectral rule
-- Each yaml codeblock starting by `#❌-test-for: some-rule-name` is considered as an OpenAPI snippet that should fail the corresponding spectral rule
+- **Rules**: A YAML codeblock starting by `#👻-rule` is considered a spectral rule and will be aggregated in the spectral.yaml file
+- **Test cases**: A YAML codeblock containing `#👻-failures:` or `#👻-fails-here:` is considered an OpenAPI test case
+  - `#👻-failures: X some-rule-name` expects the given rule to return only X failures for this test case. For example `#👻-failures: 0 some-rule-name` can be used to assert that a test case do not trigger a rule.
+  - `#👻-fails-here: some-rule-name` expects the given rule to be trigger at the very line where the comment is set
+- **Base ruleset**: to merge the rules, a `spectral.base.yaml` is needed. It should be a standard Spectral ruleset files, with empty rules. But it can includes extends, aliases, functions...
 
-## Install the poltergust CLI
+## Setup and run
 
 ```sh
 npm install
-npm link
+npx poltergust test ./examples/valid
+npx poltergust merge ./examples/valid
 ```
 
-## Test and merge the rules
+## Examples
 
-```sh
-cd ..
-poltergust test ./example
-poltergust merge ./example
-```
-
-### Example of a test output
-
-```
-$ poltergust merge ./example
-🔎 Testing the spectral rules from the .md files in the directory: ./rules
-👻 base-path-must-start-with-slash (rules\api.md:42)
-  ✅ Test OK (rules\api.md:17)
-  ✅ Test OK (rules\api.md:28)
-👻 operation-parameters-must-have-description (rules\api.md:94)
-  ✅ Test OK (rules\api.md:60)
-  ✅ Test OK (rules\api.md:75)
-👻 operation-must-have-description (rules\api.md:110)
-👻 operation-must-have-no-summary (rules\api.md:125)
-👻 operation-must-have-at-least-one-response (rules\api.md:138)
-👻 request-bodies-must-have-a-content (rules\api.md:155)
-👻 path-parameters-must-be-kebab-case (rules\parameters.md:75)
-  ✅ Test OK (rules\parameters.md:9)
-  ❌ Was expecting to fail rule path-parameters-must-be-kebab-case at line 24 in test (rules\parameters.md:62)
-  But failed there instead:
-   { start: 8, end: 8 } (rules\parameters.md:46)
-   { start: 15, end: 15 } (rules\parameters.md:53)
-👻 required-property-must-exist (rules\parameters.md:121)
-  ✅ Test OK (rules\parameters.md:92)
-```
+- [valid](examples/valid)
+- [invalid](examples/invalid)

examples/invalid/README.md

@@ -0,0 +1,15 @@
+# Invalid example
+
+* This example contains a rule in the [invalid.md](invalid.md) file
+  * ❌ The test for this rule is expected to NOT pass
+
+## Test ouput
+
+```
+npx poltergust test .\examples\invalid
+🔎 Testing the spectral rules from the .md files in the directory: .\examples\invalid
+👻 base-path-must-start-with-slash (examples\invalid\invalid.md:22)
+  ❌ Expected 0 failure(s) for rule base-path-must-start-with-slash in test (examples\invalid\invalid.md:9)
+  But got 1 instead:
+   { start: 6, end: 6 } (examples\invalid\invalid.md:15)
+```

example/invalid/invalid.md → examples/invalid/invalid.md

@@ -6,8 +6,7 @@
 
 ❌ This test should not pass, as the server's url fails the rule
 ```yaml
-#spectral-test
-#spectral-should-not-fail-anywhere-✅: base-path-must-start-with-slash
+#👻-failures: 0 base-path-must-start-with-slash
 openapi: 3.0.1
 info:
   title: Test
@@ -20,7 +19,7 @@ servers:
   <summary>Spectral rule 🤖</summary>
 
 ```yaml
-#spectral-rule
+#👻-rule
 base-path-must-start-with-slash:
   description: Base path must start with /.
   message: "{{description}}. But was {{value}}."

examples/valid/README.md

@@ -0,0 +1,28 @@
+# Valid example
+
+* This example contains rules accross several markdown files: [rules1.md](rules1.md) and [rules2.md](rules2.md)
+  * ✅ All the test declared in those files are expected to pass
+* Some of the rules use custom functions defined in the [functions](functions) directory
+* [spectral.base.yaml](spectral.base.yaml) contains the base rule files that will be used to merge the rules into. For example, it contains the extends, aliases and functions declarations
+* [spectral.yaml](spectral.yaml) is the output of the merge operation: [spectral.base.yaml](spectral.base.yaml) + all the rules found in [rules1.md](rules1.md) and [rules2.md](rules2.md)
+
+## Test ouput
+
+```
+npx poltergust test .\examples\valid
+🔎 Testing the spectral rules from the .md files in the directory: .\examples\valid
+👻 base-path-must-start-with-slash (examples\valid\rules1.md:40)
+  ✅ Test OK (examples\valid\rules1.md:17)
+  ✅ Test OK (examples\valid\rules1.md:27)
+👻 operation-parameters-must-have-description (examples\valid\rules1.md:91)
+  ✅ Test OK (examples\valid\rules1.md:58)
+  ✅ Test OK (examples\valid\rules1.md:73)
+👻 operation-must-have-description (examples\valid\rules1.md:107)
+👻 operation-must-have-no-summary (examples\valid\rules1.md:122)
+👻 operation-must-have-at-least-one-response (examples\valid\rules1.md:135)
+👻 request-bodies-must-have-a-content (examples\valid\rules1.md:152)
+👻 path-parameters-must-be-kebab-case (examples\valid\rules2.md:49)
+  ✅ Test OK (examples\valid\rules2.md:11)
+👻 required-property-must-exist (examples\valid\rules2.md:96)
+  ✅ Test OK (examples\valid\rules2.md:66)
+```

example/valid/rules1.md → examples/valid/rules1.md

@@ -14,8 +14,7 @@ We don't want static base path, because the endpoints will be aggregated and a c
 OpenAPI examples:
 
 ```yaml
-#spectral-test
-#spectral-should-fail-anywhere-❌: base-path-must-start-with-slash
+#👻-failures: 1 base-path-must-start-with-slash
 openapi: 3.0.1
 info:
   title: Test
@@ -25,8 +24,7 @@ servers:
 ```
 
 ```yaml
-#spectral-test
-#spectral-should-not-fail-anywhere-✅: base-path-must-start-with-slash
+#👻-failures: 0 base-path-must-start-with-slash
 openapi: 3.0.1
 info:
   title: Test
@@ -39,7 +37,7 @@ servers:
   <summary>Spectral rule 🤖</summary>
 
 ```yaml
-#spectral-rule
+#👻-rule
 base-path-must-start-with-slash:
   description: Base path must start with /.
   message: "{{description}}. But was {{value}}."
@@ -57,13 +55,13 @@ base-path-must-start-with-slash:
 ## operation-parameters-must-have-description
 
 ```yaml
-#spectral-test
+#👻-failures: 1 operation-parameters-must-have-description
 openapi: 3.0.1
 paths:
   /test/{id}:
     get: 
       parameters:
-      - name: id #spectral-should-fail-here-❌: operation-parameters-must-have-description
+      - name: id #👻-fails-here: operation-parameters-must-have-description
         in: path 
         # need a description
         required: true
@@ -72,8 +70,7 @@ paths:
 ```
 
 ```yaml
-#spectral-test
-#spectral-should-not-fail-anywhere-✅: operation-parameters-must-have-description
+#👻-failures: 0 operation-parameters-must-have-description
 openapi: 3.0.1
 paths:
   /test/{id}:
@@ -91,7 +88,7 @@ paths:
   <summary>Spectral rule 🤖</summary>
 
 ```yaml
-#spectral-rule
+#👻-rule
 operation-parameters-must-have-description:
   description: Operation parameters must have a description
   given: $.paths[*][*].parameters[*]
@@ -107,7 +104,7 @@ operation-parameters-must-have-description:
 ## operation-must-have-description
 
 ```yaml
-#spectral-rule
+#👻-rule
 operation-must-have-description:
   description: Operation must have a description
   given: $.paths[*][*]
@@ -122,7 +119,7 @@ operation-must-have-description:
 Summary on operations mess up the documentation portal :(
 
 ```yaml
-#spectral-rule
+#👻-rule
 operation-must-have-no-summary:
   description: Operation must not have a summary
   given: $.paths[*][*]
@@ -135,7 +132,7 @@ operation-must-have-no-summary:
 ## operation-must-have-at-least-one-response
 
 ```yaml
-#spectral-rule
+#👻-rule
 operation-must-have-at-least-one-response:
   description: Operation must have at least one response
   given: $.paths[*][*]
@@ -152,7 +149,7 @@ operation-must-have-at-least-one-response:
 ## request-bodies-must-have-a-content
 
 ```yaml
-#spectral-rule
+#👻-rule
 request-bodies-must-have-a-content:
   description: Request bodies must have a content
   given: $.paths[*][*].requestBody

example/valid/rules2.md → examples/valid/rules2.md

@@ -5,36 +5,10 @@
 ## path-parameters-must-be-kebab-case
 
 An example of valid kebab case parameters, both defined at the path or operation level
-```yaml
-#spectral-test
-#spectral-should-not-fail-anywhere-✅: path-parameters-must-be-kebab-case
-openapi: 3.0.1
-info:
-  title: Test
-  version: 1.0.0
-paths:
-  /cats/{dog-id}: 
-    get:
-      parameters:
-        - name: dog-id #good
-          in: path
-          required: true
-          schema:
-            type: number
-  /dogs/{cat-id}: 
-    parameters:
-      - name: cat-id #good
-        in: path
-        required: true
-        schema:
-          type: number
-    get: {}
-```
-
 
 Example of a wrongs parameters casing
 ```yaml
-#spectral-test
+#👻-failures: 2 path-parameters-must-be-kebab-case
 openapi: 3.0.1
 info:
   title: Test
@@ -43,14 +17,14 @@ paths:
   /cats/{dogId}: 
     get:
       parameters:
-        - name: dogId #spectral-should-fail-here-❌: path-parameters-must-be-kebab-case
+        - name: dogId #👻-fails-here: path-parameters-must-be-kebab-case
           in: path
           required: true
           schema:
             type: number
   /dogs/{catId}: 
     parameters:
-      - name: catId #spectral-should-fail-here-❌: path-parameters-must-be-kebab-case
+      - name: catId #👻-fails-here: path-parameters-must-be-kebab-case
         in: path
         required: true
         schema:
@@ -59,7 +33,7 @@ paths:
   /mouses/{mice-id}: 
     get:
       parameters:
-        - name: mice-id #spectral-should-not-fail-here-✅: path-parameters-must-be-kebab-case
+        - name: mice-id #this should not fail
           in: path
           required: true
           schema:
@@ -72,7 +46,7 @@ paths:
 This use the **pathParameters** alias to target both the parameters in the "paths" and the "operations.
 
 ```yaml
-#spectral-rule
+#👻-rule
 path-parameters-must-be-kebab-case:
   description: Path parameters must be kebab case
   given: "#parameters[?(@.in==\"path\")]"
@@ -89,7 +63,7 @@ path-parameters-must-be-kebab-case:
 ## required-property-must-exist
 
 ```yaml
-#spectral-test
+#👻-failures: 1 required-property-must-exist
 openapi: 3.0.1
 info:
   title: Test
@@ -106,8 +80,8 @@ components:
         nestedschema:
           type: object 
           required:
-            - id #spectral-should-not-fail-here-✅: required-property-must-exist
-            - nonexistent #spectral-should-fail-here-❌: required-property-must-exist
+            - id
+            - nonexistent #👻-fails-here: required-property-must-exist
           properties: 
             id: 
               type: string
@@ -119,7 +93,7 @@ components:
 This use the **isRequiredPropertyDefined** custom function.
 
 ```yaml
-#spectral-rule
+#👻-rule
 required-property-must-exist:
   description: Required property must exist
   message: "Required property must exist: {{error}}"