feat(cli): allow repeating -w to combine wordlists

CHANGELOG.md

@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
+- `-w` can be repeated to layer wordlists, for example a general list plus a target-specific one: `-w general.txt -w target.txt` reads both in order and scans an entry found in both only once (counted as a duplicate, as within one file). `-` stays valid as one of the entries. A `w` value from the config file or `SUBENUM_W` is replaced, not extended, by `-w` on the command line (#199).
 - `-max-duration 20m` gives a run a time budget. When it runs out no new names are tested, lookups already under way finish, and the run ends normally: results, the `-stats` report (`stopped_by: "duration"`, untested names in `skipped_by_cap`) and a `degraded` verdict. Under `-dL` one budget covers every target. The GitHub Action gains a `max-duration` input, so a scheduled scan can stop before the job's `timeout-minutes` kills it without a report, and `pkg/subenum` gains `Config.MaxDuration` and `Stats.Expired`. With `-max-queries`, the report now says `stopped_by: "queries"` (#190).
 - The progress line and the TUI status bar show an estimated time left (`| ETA 4m12s`), from the rate over the last 30 seconds, once a scan has run for 5 seconds; it disappears at 100% (#191).
 - `-diff-records` makes `-diff` compare the records of names found again, so a CNAME re-pointed at an unclaimed service or a moved address is reported as `changed` (`~ name` in text; `"change": "changed"` with `previous_records` in JSON and JSONL), counted in the report's `diff.changed`, and exits 4. When either side has a CNAME only the alias targets are compared, so CDN address rotation is not a change. The previous file must be JSON, JSONL or CSV. The GitHub Action gains a `changed` output (#189).

README.md

@@ -257,7 +257,7 @@ make help           # list all targets
 
 | Flag | Default | Description |
 | :--- | :---: | :--- |
-| `-w <file>` | bundled top-5000 | Wordlist file, one prefix per line; `#` comments allowed; `-` reads stdin. Omit it to use the built-in list (see [Choosing a wordlist](#choosing-a-wordlist)) |
+| `-w <file>` | bundled top-5000 | Wordlist file, one prefix per line; `#` comments allowed; `-` reads stdin. Repeat it to combine lists (`-w general.txt -w target.txt`); entries found in more than one are scanned once. Omit it to use the built-in list (see [Choosing a wordlist](#choosing-a-wordlist)) |
 | `-dL <file>` | n/a | File of apex domains to scan, one per line (`-` reads stdin), instead of a `<domain>` argument. Each domain is an independent scan; results share one output |
 | `-t <n>` | `100` | Concurrent worker goroutines |
 | `-timeout <ms>` | `1000` | DNS timeout in milliseconds, applied to each record-type lookup separately |

docs/ARCHITECTURE.md

@@ -100,7 +100,7 @@ tools/wordlist-gen.go          - Standalone wordlist generator
 *   **Implementation**: `run()` only calls `runWith(runArgs)`, which takes the arguments, stdin, stdout and stderr as injected values so tests can drive the whole CLI in-process. Per-run state lives in a `session` (`main_session.go`) whose methods are the run's phases: `parse`, `configure`, `loadInputs`, `openOutputs`, `scanAll` (which calls `scanOneTarget` per target) and `finish`. `cliFlags` holds only parsed flag values; what the run derives from them (attempts, record types, excludes, the resolver pool, the `-max-duration` deadline) is in `runtimeInputs`, and each pass over a target is a `scanPass`. `runWith` first handles the `completion` and `man` subcommands and `-resume`, then builds a `flag.FlagSet` with `parseFlagsWith`, which binds every flag into a `cliFlags` struct and applies config-file and environment defaults (see section 5). The standard `flag` package stops at the first non-flag argument, so `parseInterspersed` re-parses around each positional; flags written after the domain are honored, and everything after a `--` terminator is positional. `runWith` then:
     *   handles `-tui` (launches `tui.Start`, ignoring other flags), `-print-config` and `-version` first;
     *   parses `-format` (`output.ParseFormat`), `-type` (`dns.ParseTypes`), and merges `-attempts` with the deprecated `-retries` (`effectiveAttempts`);
-    *   calls `validateFlags`, which checks that `-w` and a target are present, that a positional domain and `-dL` are not both given, and that `-w` and `-dL` do not both read stdin. Numeric ranges and the resolver address are checked by `scan.Options.Validate`, the same validator the TUI uses. The positional domain is canonicalized with `validate.NormalizeDomain`.
+    *   calls `validateFlags`, which checks that `-w` and a target are present, that a positional domain and `-dL` are not both given, and that `-w` and `-dL` do not both read stdin (`-w` may repeat, but only one entry may be `-`). Numeric ranges and the resolver address are checked by `scan.Options.Validate`, the same validator the TUI uses. The positional domain is canonicalized with `validate.NormalizeDomain`.
 *   **Domain normalization**: `validate.NormalizeDomain` trims and lowercases the input, strips a URL scheme and path, a `:port`, a leading `*.` and a trailing dot (returning a note for each change), converts IDNs to punycode, and checks the result with `validate.Domain`. `validate.DNSServer` checks the resolver: `ip:port`, `tls://host[:port]` or `https://host/path`. Both are shared with the TUI.
 *   **Target list (`-dL`)**: `loadTargets` returns either the single positional domain or the entries of the `-dL` file (`-` for stdin). Blank lines, `#` comments and duplicates are skipped; invalid domains are reported and skipped. An empty result is an error.
 *   **Interactions**: `scanOptions` maps the parsed flags onto a `scan.Options`; its `Config` method produces the `scan.Config` passed to the scan engine.

docs/_includes/flags.md

@@ -41,4 +41,4 @@
 | `-type <string>` | A,AAAA | Comma-separated DNS record types to look up: A, AAAA, CNAME |
 | `-v` | - | Enable verbose output |
 | `-version` | - | Show version information |
-| `-w <string>` | - | Path to the wordlist file (- for stdin); omitted: the bundled top-5000 list |
+| `-w <file>` | - | Path to a wordlist file (- for stdin); repeat to combine several, deduplicated; omitted: the bundled top-5000 list |

main.go

@@ -111,7 +111,7 @@ func main() {
 type cliFlags struct {
 	tui          bool
 	printConfig  bool
-	wordlistFile string
+	wordlists    wordlistPaths // -w, repeatable (#199)
 	domainList   string
 	concurrency  int
 	timeoutMs    int
@@ -199,7 +199,7 @@ func parseFlagsWith(args []string, applyDefaults func(*flag.FlagSet) error, stde
 	}
 	fs.BoolVar(&f.tui, "tui", false, "Launch the interactive terminal UI (all other flags are ignored)")
 	fs.BoolVar(&f.printConfig, "print-config", false, "Print every setting's effective value and its source (flag, env, config or default), then exit")
-	fs.StringVar(&f.wordlistFile, "w", "", "Path to the wordlist file (- for stdin); omitted: the bundled top-5000 list")
+	fs.Var(&f.wordlists, "w", "Path to a wordlist `file` (- for stdin); repeat to combine several, deduplicated; omitted: the bundled top-5000 list")
 	fs.StringVar(&f.domainList, "dL", "", "File of apex domains to scan, one per line (- for stdin); replaces the <domain> argument")
 	fs.IntVar(&f.concurrency, "t", 100, "Number of concurrent workers")
 	fs.IntVar(&f.timeoutMs, "timeout", 1000, "DNS lookup timeout in milliseconds")
@@ -329,10 +329,14 @@ func validateFlags(f cliFlags, in *runtimeInputs, positionals []string, fs *flag
 	if !validateArgs(f, positionals, fs, out) {
 		return "", false
 	}
-	if f.wordlistFile == wordlist.Stdin && f.domainList == wordlist.Stdin {
+	if f.wordlists.readsStdin() && f.domainList == wordlist.Stdin {
 		out.Error("-w and -dL cannot both read standard input")
 		return "", false
 	}
+	if n := countStdin(f.wordlists.paths); n > 1 {
+		out.Error("-w - is given %d times; standard input can only be read once", n)
+		return "", false
+	}
 	if f.simZone != "" {
 		// Effective values, not just command-line flags: a resolver list or
 		// simulation mode from the config file or SUBENUM_* would otherwise
@@ -527,10 +531,10 @@ func logVerboseStart(f cliFlags, domain string, maxAttempts int, out *output.Wri
 		out.Info("Mode: LIVE DNS RESOLUTION")
 	}
 	out.Info("Target domain: %s", domain)
-	if f.wordlistFile == "" {
+	if len(f.wordlists.paths) == 0 {
 		out.Info("Wordlist: %s", bundledWordlistName)
 	} else {
-		out.Info("Wordlist: %s", f.wordlistFile)
+		out.Info("Wordlist: %s", f.wordlists.describe())
 	}
 	out.Info("Concurrency: %d workers", f.concurrency)
 	out.Info("Timeout: %d ms", f.timeoutMs)

main_config.go

@@ -111,6 +111,9 @@ func (d *defaultsLoader) apply(fs *flag.FlagSet) error {
 			errs = append(errs, fmt.Errorf("%w: %s value %q for %s: %w", errConfig, source, value, name, err))
 			return false
 		}
+		if v, ok := fs.Lookup(name).Value.(interface{ markDefault() }); ok {
+			v.markDefault()
+		}
 		d.values[name] = value
 		return true
 	}

main_resume.go

@@ -226,7 +226,8 @@ func inputDigests(f cliFlags) (map[string]string, error) {
 	if diff == f.outputFile {
 		diff = ""
 	}
-	for _, p := range []string{f.wordlistFile, f.domainList, f.excludeFile, f.resolvers, f.simZone, f.seedsFile, diff} {
+	paths := append(slices.Clone(f.wordlists.paths), f.domainList, f.excludeFile, f.resolvers, f.simZone, f.seedsFile, diff)
+	for _, p := range paths {
 		d, err := fileDigest(p)
 		if err != nil {
 			return nil, err
@@ -308,7 +309,7 @@ func mergeStats(a, b scan.Stats) scan.Stats {
 // run never reached keeps its saved state unchanged.
 func saveResumeState(s *session) {
 	f, out, prev := s.f, s.out, s.resume
-	if f.wordlistFile == "-" || f.domainList == "-" {
+	if f.wordlists.readsStdin() || f.domainList == "-" {
 		out.Info("Not saving resume state: the wordlist or domain list came from stdin")
 		return
 	}

main_resume_test.go

@@ -239,7 +239,7 @@ func TestSaveResumeStateKeepsEarlierPosition(t *testing.T) {
 		{Domain: "a.example", RootDone: 40, Processed: 40, Stats: scan.Stats{NXDomain: 38}, Results: []output.Result{{Result: scan.Result{Subdomain: "x.a.example"}}}},
 		{Domain: "b.example", RootDone: 7, Processed: 7, Permute: &resumePass{RootDone: 3}},
 	}}
-	f := cliFlags{stateFile: filepath.Join(t.TempDir(), "state.json"), wordlistFile: ""}
+	f := cliFlags{stateFile: filepath.Join(t.TempDir(), "state.json")}
 	out := output.New(nil, false, output.FormatText)
 	saveResumeState(&session{
 		f: f, out: out, resume: prev,

main_session.go

@@ -273,7 +273,7 @@ func (s *session) loadInputs() bool {
 	f, out := s.f, s.out
 	// The wordlist is read once (stdin can only be read once) and normalized
 	// per target, since the name-length limit depends on the domain.
-	if f.wordlistFile == "" {
+	if len(f.wordlists.paths) == 0 {
 		out.Info("No -w given: using the %s. Pass -w for a larger or target-specific list.", bundledWordlistName)
 	}
 	s.settings = effectiveSettings(s.fs)
@@ -297,7 +297,7 @@ func (s *session) loadInputs() bool {
 	// limit differs (#182).
 	s.words = wordlist.Prepare(wordLines)
 	if probe, _, _ := s.words.For(""); len(probe) == 0 {
-		out.Error("wordlist %s has no valid entries", f.wordlistFile)
+		out.Error("wordlist %s has no valid entries", f.wordlists.describe())
 		return false
 	}
 	if s.resume != nil {

main_test.go

@@ -45,8 +45,8 @@ func TestParseFlagsInterspersed(t *testing.T) {
 			if f.testMode != tc.wantSim {
 				t.Errorf("simulate = %v, want %v", f.testMode, tc.wantSim)
 			}
-			if f.wordlistFile != "wl.txt" {
-				t.Errorf("wordlist = %q, want wl.txt", f.wordlistFile)
+			if got := f.wordlists.paths; len(got) != 1 || got[0] != "wl.txt" {
+				t.Errorf("wordlist = %q, want [wl.txt]", got)
 			}
 		})
 	}
@@ -95,7 +95,7 @@ func TestValidateFlagsDomainListConflicts(t *testing.T) {
 	if _, ok := validateFlags(f, &runtimeInputs{maxAttempts: 1}, []string{"example.com"}, fs, out); ok {
 		t.Error("domain argument plus -dL should be rejected")
 	}
-	f.wordlistFile, f.domainList = "-", "-"
+	f.wordlists.paths, f.domainList = []string{"-"}, "-"
 	if _, ok := validateFlags(f, &runtimeInputs{maxAttempts: 1}, nil, fs, out); ok {
 		t.Error("-w - with -dL - should be rejected")
 	}

main_wordlist.go

@@ -10,11 +10,70 @@ import (
 // bundledWordlistName is how notices refer to the embedded list (#63).
 const bundledWordlistName = "bundled top-5000 list (SecLists)"
 
-// readWordlist returns the wordlist lines from -w, or the bundled list when
-// -w is not set.
+// wordlistPaths is the repeatable -w flag (#199). Each -w adds a file; the
+// files are read in order and their lines concatenated, so Prepare dedupes
+// across them.
+//
+// A config-file or SUBENUM_W value is a default: it is replaced, not added
+// to, by a later source or by the first -w on the command line.
+type wordlistPaths struct {
+	paths   []string
+	replace bool // the current paths are a default; the next Set replaces them
+}
+
+func (w *wordlistPaths) String() string {
+	if w == nil {
+		return ""
+	}
+	return strings.Join(w.paths, ",")
+}
+
+// Set adds a path. An empty value adds nothing, so -w "" still means the
+// bundled list.
+func (w *wordlistPaths) Set(v string) error {
+	if w.replace {
+		w.paths, w.replace = nil, false
+	}
+	if v != "" {
+		w.paths = append(w.paths, v)
+	}
+	return nil
+}
+
+// markDefault records that the current paths came from the config file or
+// the environment rather than the command line.
+func (w *wordlistPaths) markDefault() { w.replace = true }
+
+// readsStdin reports whether any -w entry is "-".
+func (w *wordlistPaths) readsStdin() bool { return countStdin(w.paths) > 0 }
+
+// describe names the wordlists for notices: the paths, comma-separated.
+func (w *wordlistPaths) describe() string { return strings.Join(w.paths, ", ") }
+
+// readWordlist returns the wordlist lines from every -w file in order, or the
+// bundled list when -w is not set.
 func readWordlist(f cliFlags, in *runtimeInputs) ([]string, error) {
-	if f.wordlistFile == "" {
+	if len(f.wordlists.paths) == 0 {
 		return wordlist.ReadLinesFrom(strings.NewReader(data.Subdomains5k))
 	}
-	return in.readLines(f.wordlistFile)
+	var lines []string
+	for _, p := range f.wordlists.paths {
+		l, err := in.readLines(p)
+		if err != nil {
+			return nil, err
+		}
+		lines = append(lines, l...)
+	}
+	return lines, nil
+}
+
+// countStdin counts the "-" entries in paths.
+func countStdin(paths []string) int {
+	n := 0
+	for _, p := range paths {
+		if p == wordlist.Stdin {
+			n++
+		}
+	}
+	return n
 }

main_wordlist_test.go

@@ -0,0 +1,108 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"slices"
+	"sort"
+	"strings"
+	"testing"
+)
+
+// TestE2EMultipleWordlists covers #199: -w repeats, the files are combined
+// and an entry found in both is scanned once.
+func TestE2EMultipleWordlists(t *testing.T) {
+	general := writeFile(t, "general.txt", "www\nmail\napi\n")
+	specific := writeFile(t, "specific.txt", "# target-specific\napi\nstaging\nWWW\n")
+	code, out := runCLI(t, "", "-simulate", "-hit-rate", "100", "-progress=false", "-w", general, "-w", specific, "example.com")
+	if code != 0 {
+		t.Fatalf("exit %d", code)
+	}
+	lines := strings.Fields(out)
+	sort.Strings(lines)
+	want := "api.example.com,mail.example.com,staging.example.com,www.example.com"
+	if got := strings.Join(lines, ","); got != want {
+		t.Errorf("results = %q, want %q", got, want)
+	}
+}
+
+// TestE2EMultipleWordlistsWithStdin: "-" stays valid as one of the entries.
+func TestE2EMultipleWordlistsWithStdin(t *testing.T) {
+	file := writeFile(t, "wl.txt", "www\n")
+	code, out := runCLI(t, "mail\nwww\n", "-simulate", "-hit-rate", "100", "-progress=false", "-w", file, "-w", "-", "example.com")
+	if code != 0 {
+		t.Fatalf("exit %d", code)
+	}
+	lines := strings.Fields(out)
+	sort.Strings(lines)
+	if got := strings.Join(lines, ","); got != "mail.example.com,www.example.com" {
+		t.Errorf("results = %q", got)
+	}
+}
+
+func TestMultipleWordlistsRejectedInputs(t *testing.T) {
+	file := writeFile(t, "wl.txt", "www\n")
+	for name, args := range map[string][]string{
+		"stdin twice":      {"-w", "-", "-w", "-", "example.com"},
+		"stdin with -dL -": {"-w", file, "-w", "-", "-dL", "-"},
+		"one file missing": {"-w", file, "-w", filepath.Join(t.TempDir(), "nope.txt"), "example.com"},
+	} {
+		t.Run(name, func(t *testing.T) {
+			code, _ := runCLI(t, "", append([]string{"-simulate", "-progress=false"}, args...)...)
+			if code == 0 {
+				t.Errorf("exit 0, want a failure")
+			}
+		})
+	}
+}
+
+// TestWordlistDefaultsReplaced: a config-file or SUBENUM_W wordlist is a
+// default, so a later source replaces it instead of adding to it.
+func TestWordlistDefaultsReplaced(t *testing.T) {
+	cfg := filepath.Join(t.TempDir(), "config.json")
+	if err := os.WriteFile(cfg, []byte(`{"w": "from-config.txt"}`), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	cases := []struct {
+		name string
+		env  map[string]string
+		args []string
+		want []string
+	}{
+		{"config only", nil, nil, []string{"from-config.txt"}},
+		{"env beats config", map[string]string{"SUBENUM_W": "from-env.txt"}, nil, []string{"from-env.txt"}},
+		{"flags beat env and config", map[string]string{"SUBENUM_W": "from-env.txt"}, []string{"-w", "a.txt", "-w", "b.txt"}, []string{"a.txt", "b.txt"}},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			d := &defaultsLoader{path: cfg, getenv: func(k string) string { return tc.env[k] }}
+			f, _, _, err := parseFlagsWith(append(tc.args, "example.com"), d.apply, nil)
+			if err != nil {
+				t.Fatal(err)
+			}
+			if !slices.Equal(f.wordlists.paths, tc.want) {
+				t.Errorf("wordlists = %q, want %q", f.wordlists.paths, tc.want)
+			}
+		})
+	}
+}
+
+func TestParseFlagsRepeatedWordlist(t *testing.T) {
+	f, _, fs, err := parseFlags([]string{"-w", "a.txt", "example.com", "-w", "b.txt"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if want := []string{"a.txt", "b.txt"}; !slices.Equal(f.wordlists.paths, want) {
+		t.Errorf("wordlists = %q, want %q", f.wordlists.paths, want)
+	}
+	if got := fs.Lookup("w").Value.String(); got != "a.txt,b.txt" {
+		t.Errorf("-w value = %q", got)
+	}
+	f, _, _, err = parseFlags([]string{"example.com"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(f.wordlists.paths) != 0 {
+		t.Errorf("no -w: wordlists = %q, want none (bundled list)", f.wordlists.paths)
+	}
+}