Skip to content

fix: improve URL parsing error handling in ListInputProvider #6448

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: dev
Choose a base branch
from
Open

fix: improve URL parsing error handling in ListInputProvider #6448

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7b2094d
fix: improve URL parsing error handling in ListInputProvider
oleveloper Sep 1, 2025
9e19b5c
Merge branch 'projectdiscovery:dev' into dev
oleveloper Oct 10, 2025
f697aab
Merge branch 'dev' into pr/6448
Mzack9999 Oct 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 34 additions & 21 deletions pkg/input/provider/list/hmap.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ package list
import (
"bufio"
"context"
"fmt"
"io"
"os"
"regexp"
Expand Down Expand Up @@ -39,6 +38,7 @@ const DefaultMaxDedupeItemsCount = 10000
// it supports list type of input ex: urls,file,stdin,uncover,etc. (i.e just url not complete request/response)
type ListInputProvider struct {
ipOptions *ipOptions
options *types.Options
inputCount int64
excludedCount int64
dupeCount int64
Expand Down Expand Up @@ -71,6 +71,7 @@ func New(opts *Options) (*ListInputProvider, error) {

input := &ListInputProvider{
hostMap: hm,
options: options,
ipOptions: &ipOptions{
ScanAllIPs: options.ScanAllIPs,
IPV4: sliceutil.Contains(options.IPVersion, "4"),
Expand Down Expand Up @@ -147,16 +148,22 @@ func (i *ListInputProvider) Set(executionId string, value string) {
// parse hostname if url is given
urlx, err := urlutil.Parse(URL)
if err != nil || (urlx != nil && urlx.Host == "") {
gologger.Debug().Label("url").MsgFunc(func() string {
if err != nil {
return fmt.Sprintf("failed to parse url %v got %v skipping ip selection", URL, err)
}
return fmt.Sprintf("got empty hostname for %v skipping ip selection", URL)
})
metaInput := contextargs.NewMetaInput()
metaInput.Input = URL
i.setItem(metaInput)
return
if err != nil {
gologger.Error().Msgf("Failed to parse URL %v: %v", URL, err)
} else {
gologger.Error().Msgf("Invalid hostname for URL %v", URL)
}

// Check if this is single target mode (-u) or batch mode (-l)
if i.options.TargetsFilePath == "" && len(i.options.Targets) <= 1 {
// Single target mode: exit immediately
gologger.Info().Msg("Stopping scan due to invalid target")
os.Exit(1)
} else {
// Batch mode: log error and skip this target
gologger.Warning().Msgf("Skipping invalid target: %v", URL)
return
}
Comment on lines +151 to +166
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Invalid-target handling: wrong single-target detection, duplicate logs, and os.Exit in library code.

  • Misclassification: stdin/uncover-only runs (Targets==0, TargetsFilePath=="") are treated as “single-target”, causing premature exit on any bad line.
  • Duplicate logs: you emit an Error then an Info/Warning for the same invalid input.
  • os.Exit here makes the package kill the process; better keep exit decisions at the CLI/runner layer.

Minimum fix: correct detection and emit a single message; keep exit only if you must. Suggested patch:

-		if err != nil {
-			gologger.Error().Msgf("Failed to parse URL %v: %v", URL, err)
-		} else {
-			gologger.Error().Msgf("Invalid hostname for URL %v", URL)
-		}
-		
-		// Check if this is single target mode (-u) or batch mode (-l)
-		if i.options.TargetsFilePath == "" && len(i.options.Targets) <= 1 {
-			// Single target mode: exit immediately
-			gologger.Info().Msg("Stopping scan due to invalid target")
-			os.Exit(1)
-		} else {
-			// Batch mode: log error and skip this target
-			gologger.Warning().Msgf("Skipping invalid target: %v", URL)
-			return
-		}
+		// Single clear message + correct single/batch detection
+		reason := "empty hostname"
+		if err != nil {
+			reason = err.Error()
+		}
+		if i.isSingleTargetMode() {
+			gologger.Error().Msgf("Invalid target %v: %v", URL, reason)
+			// Ideally propagate to caller instead of exiting here.
+			os.Exit(1)
+		} else {
+			gologger.Warning().Msgf("Skipping invalid target %v: %v", URL, reason)
+			return
+		}

Add this helper once in the file:

func (i *ListInputProvider) isSingleTargetMode() bool {
	if i == nil || i.options == nil {
		return false
	}
	// Consider only explicit -u usage; stdin/-l/uncover are batch-like.
	return i.options.TargetsFilePath == "" &&
		!i.options.Stdin &&
		!i.options.Uncover &&
		len(i.options.Targets) == 1
}

If feasible, prefer bubbling an error to the caller instead of os.Exit to keep package-level code side-effect free. I can draft that refactor if you want.

🤖 Prompt for AI Agents 
In pkg/input/provider/list/hmap.go around lines 151 to 166: the current logic
misdetects single-target mode (treats stdin/uncover runs as single-target),
emits duplicate logs for the same invalid URL, and calls os.Exit from library
code; fix by adding a helper method isSingleTargetMode on ListInputProvider that
returns true only when TargetsFilePath=="" AND Targets length == 1 AND explicit
flags Stdin and Uncover are false (also guard i/options nil), replace the
current detection with that helper, remove the redundant Error+Info/Warning pair
so only one log entry is emitted for an invalid target, and do NOT call os.Exit
here — instead return an error or a sentinel so the CLI/runner layer can decide
to exit; if you prefer, change the function signature to return an error so
callers can handle exit behavior.

}

// Check if input is ip or hostname
Expand Down Expand Up @@ -391,16 +398,22 @@ func (i *ListInputProvider) Del(executionId string, value string) {
// parse hostname if url is given
urlx, err := urlutil.Parse(URL)
if err != nil || (urlx != nil && urlx.Host == "") {
gologger.Debug().Label("url").MsgFunc(func() string {
if err != nil {
return fmt.Sprintf("failed to parse url %v got %v skipping ip selection", URL, err)
}
return fmt.Sprintf("got empty hostname for %v skipping ip selection", URL)
})
metaInput := contextargs.NewMetaInput()
metaInput.Input = URL
i.delItem(metaInput)
return
if err != nil {
gologger.Error().Msgf("Failed to parse URL %v: %v", URL, err)
} else {
gologger.Error().Msgf("Invalid hostname for URL %v", URL)
}

// Check if this is single target mode (-u) or batch mode (-l)
if i.options.TargetsFilePath == "" && len(i.options.Targets) <= 1 {
// Single target mode: exit immediately
gologger.Info().Msg("Stopping scan due to invalid target")
os.Exit(1)
} else {
// Batch mode: log error and skip this target
gologger.Warning().Msgf("Skipping invalid target: %v", URL)
return
}
Comment on lines +401 to +416
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Don’t os.Exit from Del; treat invalid excludes as non-fatal and skip.

Exiting from Del can kill a scan due to an invalid exclude pattern even in -l/stdin runs. Keep it a single warning and continue.

-		if err != nil {
-			gologger.Error().Msgf("Failed to parse URL %v: %v", URL, err)
-		} else {
-			gologger.Error().Msgf("Invalid hostname for URL %v", URL)
-		}
-		
-		// Check if this is single target mode (-u) or batch mode (-l)
-		if i.options.TargetsFilePath == "" && len(i.options.Targets) <= 1 {
-			// Single target mode: exit immediately
-			gologger.Info().Msg("Stopping scan due to invalid target")
-			os.Exit(1)
-		} else {
-			// Batch mode: log error and skip this target
-			gologger.Warning().Msgf("Skipping invalid target: %v", URL)
-			return
-		}
+		reason := "empty hostname"
+		if err != nil {
+			reason = err.Error()
+		}
+		gologger.Warning().Msgf("Skipping invalid exclude target %v: %v", URL, reason)
+		return
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if err != nil {
gologger.Error().Msgf("Failed to parse URL %v: %v", URL, err)
} else {
gologger.Error().Msgf("Invalid hostname for URL %v", URL)
}
// Check if this is single target mode (-u) or batch mode (-l)
if i.options.TargetsFilePath == "" && len(i.options.Targets) <= 1 {
// Single target mode: exit immediately
gologger.Info().Msg("Stopping scan due to invalid target")
os.Exit(1)
} else {
// Batch mode: log error and skip this target
gologger.Warning().Msgf("Skipping invalid target: %v", URL)
return
}
// Treat invalid exclude patterns as non-fatal—just warn and skip
reason := "empty hostname"
if err != nil {
reason = err.Error()
}
gologger.Warning().Msgf("Skipping invalid exclude target %v: %v", URL, reason)
return
🤖 Prompt for AI Agents 
In pkg/input/provider/list/hmap.go around lines 401 to 416, the code currently
calls os.Exit(1) when encountering an invalid target, which terminates the
entire scan; instead, treat invalid excludes/non-fatal parse failures as
non-fatal: remove the os.Exit call, change the logic so that both single-target
and batch runs log an appropriate warning or info and return to skip processing
this target (for single-target mode, return an error to the caller or set a
non-fatal status rather than exiting), ensuring the function does not terminate
the process and batch/stdin runs continue.

}

// Check if input is ip or hostname
Expand Down
Loading