Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extractor variable support #5727

Open
wants to merge 2 commits into
base: dev
Choose a base branch
from
Open

Extractor variable support #5727

wants to merge 2 commits into from
+61 −18

Conversation

murat-kekij
Copy link

Proposed changes

Closes #2647

  • Adds variable support to regex extractors
  • Adds variable support to json extractors
Test Server 
package main

import (
  "encoding/json"
  "fmt"
  "log"
  "net/http"
  "strings"
)

// DomainResponse represents the structure of the response where the domain name is a key
type DomainResponse map[string]interface{}

// Static data for the example
var exampleData = map[string]interface{}{
  "subdomains": []string{"api", "www", "test"},
  "ip":         "192.168.1.1",
  "region":     "us-east",
}

func main() {
  http.HandleFunc("/json-test", jsonHandler)
  http.HandleFunc("/regex-test", regexHandler)

  fmt.Println("Server is running on http://127.0.0.1:5005")
  log.Fatal(http.ListenAndServe(":5005", nil))
}

// jsonHandler returns a JSON object with a dynamic key (domain name based on the request URL)
func jsonHandler(w http.ResponseWriter, r *http.Request) {
  // Get the host from the request
  host := strings.Split(r.Host, ":")[0]

  // Create the response with the dynamic host as the key
  response := DomainResponse{
  	host: exampleData,
  }

  // Set response header to application/json
  w.Header().Set("Content-Type", "application/json")
  w.WriteHeader(http.StatusOK)
  json.NewEncoder(w).Encode(response)
}

func regexHandler(w http.ResponseWriter, r *http.Request) {
  nonce := "abc123"
  scriptSrc := fmt.Sprintf("/static/main.%s.js", nonce)

  // HTML content with the script tag
  htmlContent := fmt.Sprintf(`
  <html>
  <head>
  	<title>Test Page</title>
  </head>
  <body>
  	<h1>Test Page With Dynamic Script Tag</h1>
  	<script src="%s"></script>
  </body>
  </html>
  `, scriptSrc)

  // Write HTML response
  w.Header().Set("Content-Type", "text/html")
  w.WriteHeader(http.StatusOK)
  w.Write([]byte(htmlContent))
}
Test Json Extractor 
id: http-variable-json-extractor

info:
name: HTTP Variable JSON Extractor
author: pdteam
severity: info

http:
- method: GET
  path:
    - "{{BaseURL}}/json-test"

  extractors:
    - type: json
      part: body
      name: subdomains
      json:
        - '."{{FQDN}}".subdomains[]'
Test Regex Extractor 
id: http-variable-regex-extractor

info:
name: HTTP Variable Regex Extractor
author: pdteam
severity: info

http:
- method: GET
  path:
    - "{{BaseURL}}/regex-test"

  extractors:
    - type: regex
      part: body
      name: mainjs
      regex:
        - '{{script_regex}}'

Command

nuclei -t ./http-variable-regex-extractor.yaml -u http://127.0.0.1:5005 -var "script_regex=/static/main\.[a-zA-Z0-9]+\.js"

Checklist

  • Pull request is created against the dev branch
  • All checks passed (lint, unit/integration/regression tests etc.) with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

@GeorginaReeder
Copy link

Thanks so much for your contribution @murat-kekij !

tarunKoyalwar

This comment was marked as outdated.

Sign in to view

@tarunKoyalwar tarunKoyalwar self-requested a review December 1, 2024 16:34
tarunKoyalwar
tarunKoyalwar requested changes Dec 1, 2024
View reviewed changes
Copy link
Member

@tarunKoyalwar tarunKoyalwar left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is a panic in unit test due to nil pointer dereference

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tarunKoyalwar tarunKoyalwar tarunKoyalwar requested changes

@dwisiswant0 dwisiswant0 dwisiswant0 approved these changes

@dogancanbakir dogancanbakir Awaiting requested review from dogancanbakir

@ehsandeep ehsandeep Awaiting requested review from ehsandeep

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

extractors add variable support
4 participants
@murat-kekij @GeorginaReeder @dwisiswant0 @tarunKoyalwar