diff --git a/CHANGELOG.md b/CHANGELOG.md index 722eb2c..bab3fb6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Added +- `-w` can be repeated to layer wordlists, for example a general list plus a target-specific one: `-w general.txt -w target.txt` reads both in order and scans an entry found in both only once (counted as a duplicate, as within one file). `-` stays valid as one of the entries. A `w` value from the config file or `SUBENUM_W` is replaced, not extended, by `-w` on the command line (#199). - `-max-duration 20m` gives a run a time budget. When it runs out no new names are tested, lookups already under way finish, and the run ends normally: results, the `-stats` report (`stopped_by: "duration"`, untested names in `skipped_by_cap`) and a `degraded` verdict. Under `-dL` one budget covers every target. The GitHub Action gains a `max-duration` input, so a scheduled scan can stop before the job's `timeout-minutes` kills it without a report, and `pkg/subenum` gains `Config.MaxDuration` and `Stats.Expired`. With `-max-queries`, the report now says `stopped_by: "queries"` (#190). - The progress line and the TUI status bar show an estimated time left (`| ETA 4m12s`), from the rate over the last 30 seconds, once a scan has run for 5 seconds; it disappears at 100% (#191). - `-diff-records` makes `-diff` compare the records of names found again, so a CNAME re-pointed at an unclaimed service or a moved address is reported as `changed` (`~ name` in text; `"change": "changed"` with `previous_records` in JSON and JSONL), counted in the report's `diff.changed`, and exits 4. When either side has a CNAME only the alias targets are compared, so CDN address rotation is not a change. The previous file must be JSON, JSONL or CSV. The GitHub Action gains a `changed` output (#189). diff --git a/README.md b/README.md index 4a5e30a..c9e5d54 100644 --- a/README.md +++ b/README.md @@ -257,7 +257,7 @@ make help # list all targets | Flag | Default | Description | | :--- | :---: | :--- | -| `-w ` | bundled top-5000 | Wordlist file, one prefix per line; `#` comments allowed; `-` reads stdin. Omit it to use the built-in list (see [Choosing a wordlist](#choosing-a-wordlist)) | +| `-w ` | bundled top-5000 | Wordlist file, one prefix per line; `#` comments allowed; `-` reads stdin. Repeat it to combine lists (`-w general.txt -w target.txt`); entries found in more than one are scanned once. Omit it to use the built-in list (see [Choosing a wordlist](#choosing-a-wordlist)) | | `-dL ` | n/a | File of apex domains to scan, one per line (`-` reads stdin), instead of a `` argument. Each domain is an independent scan; results share one output | | `-t ` | `100` | Concurrent worker goroutines | | `-timeout ` | `1000` | DNS timeout in milliseconds, applied to each record-type lookup separately | diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 504b70d..fbca941 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -100,7 +100,7 @@ tools/wordlist-gen.go - Standalone wordlist generator * **Implementation**: `run()` only calls `runWith(runArgs)`, which takes the arguments, stdin, stdout and stderr as injected values so tests can drive the whole CLI in-process. Per-run state lives in a `session` (`main_session.go`) whose methods are the run's phases: `parse`, `configure`, `loadInputs`, `openOutputs`, `scanAll` (which calls `scanOneTarget` per target) and `finish`. `cliFlags` holds only parsed flag values; what the run derives from them (attempts, record types, excludes, the resolver pool, the `-max-duration` deadline) is in `runtimeInputs`, and each pass over a target is a `scanPass`. `runWith` first handles the `completion` and `man` subcommands and `-resume`, then builds a `flag.FlagSet` with `parseFlagsWith`, which binds every flag into a `cliFlags` struct and applies config-file and environment defaults (see section 5). The standard `flag` package stops at the first non-flag argument, so `parseInterspersed` re-parses around each positional; flags written after the domain are honored, and everything after a `--` terminator is positional. `runWith` then: * handles `-tui` (launches `tui.Start`, ignoring other flags), `-print-config` and `-version` first; * parses `-format` (`output.ParseFormat`), `-type` (`dns.ParseTypes`), and merges `-attempts` with the deprecated `-retries` (`effectiveAttempts`); - * calls `validateFlags`, which checks that `-w` and a target are present, that a positional domain and `-dL` are not both given, and that `-w` and `-dL` do not both read stdin. Numeric ranges and the resolver address are checked by `scan.Options.Validate`, the same validator the TUI uses. The positional domain is canonicalized with `validate.NormalizeDomain`. + * calls `validateFlags`, which checks that `-w` and a target are present, that a positional domain and `-dL` are not both given, and that `-w` and `-dL` do not both read stdin (`-w` may repeat, but only one entry may be `-`). Numeric ranges and the resolver address are checked by `scan.Options.Validate`, the same validator the TUI uses. The positional domain is canonicalized with `validate.NormalizeDomain`. * **Domain normalization**: `validate.NormalizeDomain` trims and lowercases the input, strips a URL scheme and path, a `:port`, a leading `*.` and a trailing dot (returning a note for each change), converts IDNs to punycode, and checks the result with `validate.Domain`. `validate.DNSServer` checks the resolver: `ip:port`, `tls://host[:port]` or `https://host/path`. Both are shared with the TUI. * **Target list (`-dL`)**: `loadTargets` returns either the single positional domain or the entries of the `-dL` file (`-` for stdin). Blank lines, `#` comments and duplicates are skipped; invalid domains are reported and skipped. An empty result is an error. * **Interactions**: `scanOptions` maps the parsed flags onto a `scan.Options`; its `Config` method produces the `scan.Config` passed to the scan engine. diff --git a/docs/_includes/flags.md b/docs/_includes/flags.md index a98ac95..09a672e 100644 --- a/docs/_includes/flags.md +++ b/docs/_includes/flags.md @@ -41,4 +41,4 @@ | `-type ` | A,AAAA | Comma-separated DNS record types to look up: A, AAAA, CNAME | | `-v` | - | Enable verbose output | | `-version` | - | Show version information | -| `-w ` | - | Path to the wordlist file (- for stdin); omitted: the bundled top-5000 list | +| `-w ` | - | Path to a wordlist file (- for stdin); repeat to combine several, deduplicated; omitted: the bundled top-5000 list | diff --git a/main.go b/main.go index 2e275d4..509175a 100644 --- a/main.go +++ b/main.go @@ -111,7 +111,7 @@ func main() { type cliFlags struct { tui bool printConfig bool - wordlistFile string + wordlists wordlistPaths // -w, repeatable (#199) domainList string concurrency int timeoutMs int @@ -199,7 +199,7 @@ func parseFlagsWith(args []string, applyDefaults func(*flag.FlagSet) error, stde } fs.BoolVar(&f.tui, "tui", false, "Launch the interactive terminal UI (all other flags are ignored)") fs.BoolVar(&f.printConfig, "print-config", false, "Print every setting's effective value and its source (flag, env, config or default), then exit") - fs.StringVar(&f.wordlistFile, "w", "", "Path to the wordlist file (- for stdin); omitted: the bundled top-5000 list") + fs.Var(&f.wordlists, "w", "Path to a wordlist `file` (- for stdin); repeat to combine several, deduplicated; omitted: the bundled top-5000 list") fs.StringVar(&f.domainList, "dL", "", "File of apex domains to scan, one per line (- for stdin); replaces the argument") fs.IntVar(&f.concurrency, "t", 100, "Number of concurrent workers") fs.IntVar(&f.timeoutMs, "timeout", 1000, "DNS lookup timeout in milliseconds") @@ -329,10 +329,14 @@ func validateFlags(f cliFlags, in *runtimeInputs, positionals []string, fs *flag if !validateArgs(f, positionals, fs, out) { return "", false } - if f.wordlistFile == wordlist.Stdin && f.domainList == wordlist.Stdin { + if f.wordlists.readsStdin() && f.domainList == wordlist.Stdin { out.Error("-w and -dL cannot both read standard input") return "", false } + if n := countStdin(f.wordlists.paths); n > 1 { + out.Error("-w - is given %d times; standard input can only be read once", n) + return "", false + } if f.simZone != "" { // Effective values, not just command-line flags: a resolver list or // simulation mode from the config file or SUBENUM_* would otherwise @@ -527,10 +531,10 @@ func logVerboseStart(f cliFlags, domain string, maxAttempts int, out *output.Wri out.Info("Mode: LIVE DNS RESOLUTION") } out.Info("Target domain: %s", domain) - if f.wordlistFile == "" { + if len(f.wordlists.paths) == 0 { out.Info("Wordlist: %s", bundledWordlistName) } else { - out.Info("Wordlist: %s", f.wordlistFile) + out.Info("Wordlist: %s", f.wordlists.describe()) } out.Info("Concurrency: %d workers", f.concurrency) out.Info("Timeout: %d ms", f.timeoutMs) diff --git a/main_config.go b/main_config.go index b15a03d..10562f5 100644 --- a/main_config.go +++ b/main_config.go @@ -111,6 +111,9 @@ func (d *defaultsLoader) apply(fs *flag.FlagSet) error { errs = append(errs, fmt.Errorf("%w: %s value %q for %s: %w", errConfig, source, value, name, err)) return false } + if v, ok := fs.Lookup(name).Value.(interface{ markDefault() }); ok { + v.markDefault() + } d.values[name] = value return true } diff --git a/main_resume.go b/main_resume.go index a4680c8..de86ccc 100644 --- a/main_resume.go +++ b/main_resume.go @@ -226,7 +226,8 @@ func inputDigests(f cliFlags) (map[string]string, error) { if diff == f.outputFile { diff = "" } - for _, p := range []string{f.wordlistFile, f.domainList, f.excludeFile, f.resolvers, f.simZone, f.seedsFile, diff} { + paths := append(slices.Clone(f.wordlists.paths), f.domainList, f.excludeFile, f.resolvers, f.simZone, f.seedsFile, diff) + for _, p := range paths { d, err := fileDigest(p) if err != nil { return nil, err @@ -308,7 +309,7 @@ func mergeStats(a, b scan.Stats) scan.Stats { // run never reached keeps its saved state unchanged. func saveResumeState(s *session) { f, out, prev := s.f, s.out, s.resume - if f.wordlistFile == "-" || f.domainList == "-" { + if f.wordlists.readsStdin() || f.domainList == "-" { out.Info("Not saving resume state: the wordlist or domain list came from stdin") return } diff --git a/main_resume_test.go b/main_resume_test.go index 18f0760..35e4fcb 100644 --- a/main_resume_test.go +++ b/main_resume_test.go @@ -239,7 +239,7 @@ func TestSaveResumeStateKeepsEarlierPosition(t *testing.T) { {Domain: "a.example", RootDone: 40, Processed: 40, Stats: scan.Stats{NXDomain: 38}, Results: []output.Result{{Result: scan.Result{Subdomain: "x.a.example"}}}}, {Domain: "b.example", RootDone: 7, Processed: 7, Permute: &resumePass{RootDone: 3}}, }} - f := cliFlags{stateFile: filepath.Join(t.TempDir(), "state.json"), wordlistFile: ""} + f := cliFlags{stateFile: filepath.Join(t.TempDir(), "state.json")} out := output.New(nil, false, output.FormatText) saveResumeState(&session{ f: f, out: out, resume: prev, diff --git a/main_session.go b/main_session.go index ff5a838..9b5bc51 100644 --- a/main_session.go +++ b/main_session.go @@ -273,7 +273,7 @@ func (s *session) loadInputs() bool { f, out := s.f, s.out // The wordlist is read once (stdin can only be read once) and normalized // per target, since the name-length limit depends on the domain. - if f.wordlistFile == "" { + if len(f.wordlists.paths) == 0 { out.Info("No -w given: using the %s. Pass -w for a larger or target-specific list.", bundledWordlistName) } s.settings = effectiveSettings(s.fs) @@ -297,7 +297,7 @@ func (s *session) loadInputs() bool { // limit differs (#182). s.words = wordlist.Prepare(wordLines) if probe, _, _ := s.words.For(""); len(probe) == 0 { - out.Error("wordlist %s has no valid entries", f.wordlistFile) + out.Error("wordlist %s has no valid entries", f.wordlists.describe()) return false } if s.resume != nil { diff --git a/main_test.go b/main_test.go index 34f8a0c..29b5874 100644 --- a/main_test.go +++ b/main_test.go @@ -45,8 +45,8 @@ func TestParseFlagsInterspersed(t *testing.T) { if f.testMode != tc.wantSim { t.Errorf("simulate = %v, want %v", f.testMode, tc.wantSim) } - if f.wordlistFile != "wl.txt" { - t.Errorf("wordlist = %q, want wl.txt", f.wordlistFile) + if got := f.wordlists.paths; len(got) != 1 || got[0] != "wl.txt" { + t.Errorf("wordlist = %q, want [wl.txt]", got) } }) } @@ -95,7 +95,7 @@ func TestValidateFlagsDomainListConflicts(t *testing.T) { if _, ok := validateFlags(f, &runtimeInputs{maxAttempts: 1}, []string{"example.com"}, fs, out); ok { t.Error("domain argument plus -dL should be rejected") } - f.wordlistFile, f.domainList = "-", "-" + f.wordlists.paths, f.domainList = []string{"-"}, "-" if _, ok := validateFlags(f, &runtimeInputs{maxAttempts: 1}, nil, fs, out); ok { t.Error("-w - with -dL - should be rejected") } diff --git a/main_wordlist.go b/main_wordlist.go index 29adca1..438714c 100644 --- a/main_wordlist.go +++ b/main_wordlist.go @@ -10,11 +10,70 @@ import ( // bundledWordlistName is how notices refer to the embedded list (#63). const bundledWordlistName = "bundled top-5000 list (SecLists)" -// readWordlist returns the wordlist lines from -w, or the bundled list when -// -w is not set. +// wordlistPaths is the repeatable -w flag (#199). Each -w adds a file; the +// files are read in order and their lines concatenated, so Prepare dedupes +// across them. +// +// A config-file or SUBENUM_W value is a default: it is replaced, not added +// to, by a later source or by the first -w on the command line. +type wordlistPaths struct { + paths []string + replace bool // the current paths are a default; the next Set replaces them +} + +func (w *wordlistPaths) String() string { + if w == nil { + return "" + } + return strings.Join(w.paths, ",") +} + +// Set adds a path. An empty value adds nothing, so -w "" still means the +// bundled list. +func (w *wordlistPaths) Set(v string) error { + if w.replace { + w.paths, w.replace = nil, false + } + if v != "" { + w.paths = append(w.paths, v) + } + return nil +} + +// markDefault records that the current paths came from the config file or +// the environment rather than the command line. +func (w *wordlistPaths) markDefault() { w.replace = true } + +// readsStdin reports whether any -w entry is "-". +func (w *wordlistPaths) readsStdin() bool { return countStdin(w.paths) > 0 } + +// describe names the wordlists for notices: the paths, comma-separated. +func (w *wordlistPaths) describe() string { return strings.Join(w.paths, ", ") } + +// readWordlist returns the wordlist lines from every -w file in order, or the +// bundled list when -w is not set. func readWordlist(f cliFlags, in *runtimeInputs) ([]string, error) { - if f.wordlistFile == "" { + if len(f.wordlists.paths) == 0 { return wordlist.ReadLinesFrom(strings.NewReader(data.Subdomains5k)) } - return in.readLines(f.wordlistFile) + var lines []string + for _, p := range f.wordlists.paths { + l, err := in.readLines(p) + if err != nil { + return nil, err + } + lines = append(lines, l...) + } + return lines, nil +} + +// countStdin counts the "-" entries in paths. +func countStdin(paths []string) int { + n := 0 + for _, p := range paths { + if p == wordlist.Stdin { + n++ + } + } + return n } diff --git a/main_wordlist_test.go b/main_wordlist_test.go new file mode 100644 index 0000000..fbc1b36 --- /dev/null +++ b/main_wordlist_test.go @@ -0,0 +1,108 @@ +package main + +import ( + "os" + "path/filepath" + "slices" + "sort" + "strings" + "testing" +) + +// TestE2EMultipleWordlists covers #199: -w repeats, the files are combined +// and an entry found in both is scanned once. +func TestE2EMultipleWordlists(t *testing.T) { + general := writeFile(t, "general.txt", "www\nmail\napi\n") + specific := writeFile(t, "specific.txt", "# target-specific\napi\nstaging\nWWW\n") + code, out := runCLI(t, "", "-simulate", "-hit-rate", "100", "-progress=false", "-w", general, "-w", specific, "example.com") + if code != 0 { + t.Fatalf("exit %d", code) + } + lines := strings.Fields(out) + sort.Strings(lines) + want := "api.example.com,mail.example.com,staging.example.com,www.example.com" + if got := strings.Join(lines, ","); got != want { + t.Errorf("results = %q, want %q", got, want) + } +} + +// TestE2EMultipleWordlistsWithStdin: "-" stays valid as one of the entries. +func TestE2EMultipleWordlistsWithStdin(t *testing.T) { + file := writeFile(t, "wl.txt", "www\n") + code, out := runCLI(t, "mail\nwww\n", "-simulate", "-hit-rate", "100", "-progress=false", "-w", file, "-w", "-", "example.com") + if code != 0 { + t.Fatalf("exit %d", code) + } + lines := strings.Fields(out) + sort.Strings(lines) + if got := strings.Join(lines, ","); got != "mail.example.com,www.example.com" { + t.Errorf("results = %q", got) + } +} + +func TestMultipleWordlistsRejectedInputs(t *testing.T) { + file := writeFile(t, "wl.txt", "www\n") + for name, args := range map[string][]string{ + "stdin twice": {"-w", "-", "-w", "-", "example.com"}, + "stdin with -dL -": {"-w", file, "-w", "-", "-dL", "-"}, + "one file missing": {"-w", file, "-w", filepath.Join(t.TempDir(), "nope.txt"), "example.com"}, + } { + t.Run(name, func(t *testing.T) { + code, _ := runCLI(t, "", append([]string{"-simulate", "-progress=false"}, args...)...) + if code == 0 { + t.Errorf("exit 0, want a failure") + } + }) + } +} + +// TestWordlistDefaultsReplaced: a config-file or SUBENUM_W wordlist is a +// default, so a later source replaces it instead of adding to it. +func TestWordlistDefaultsReplaced(t *testing.T) { + cfg := filepath.Join(t.TempDir(), "config.json") + if err := os.WriteFile(cfg, []byte(`{"w": "from-config.txt"}`), 0o600); err != nil { + t.Fatal(err) + } + cases := []struct { + name string + env map[string]string + args []string + want []string + }{ + {"config only", nil, nil, []string{"from-config.txt"}}, + {"env beats config", map[string]string{"SUBENUM_W": "from-env.txt"}, nil, []string{"from-env.txt"}}, + {"flags beat env and config", map[string]string{"SUBENUM_W": "from-env.txt"}, []string{"-w", "a.txt", "-w", "b.txt"}, []string{"a.txt", "b.txt"}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + d := &defaultsLoader{path: cfg, getenv: func(k string) string { return tc.env[k] }} + f, _, _, err := parseFlagsWith(append(tc.args, "example.com"), d.apply, nil) + if err != nil { + t.Fatal(err) + } + if !slices.Equal(f.wordlists.paths, tc.want) { + t.Errorf("wordlists = %q, want %q", f.wordlists.paths, tc.want) + } + }) + } +} + +func TestParseFlagsRepeatedWordlist(t *testing.T) { + f, _, fs, err := parseFlags([]string{"-w", "a.txt", "example.com", "-w", "b.txt"}) + if err != nil { + t.Fatal(err) + } + if want := []string{"a.txt", "b.txt"}; !slices.Equal(f.wordlists.paths, want) { + t.Errorf("wordlists = %q, want %q", f.wordlists.paths, want) + } + if got := fs.Lookup("w").Value.String(); got != "a.txt,b.txt" { + t.Errorf("-w value = %q", got) + } + f, _, _, err = parseFlags([]string{"example.com"}) + if err != nil { + t.Fatal(err) + } + if len(f.wordlists.paths) != 0 { + t.Errorf("no -w: wordlists = %q, want none (bundled list)", f.wordlists.paths) + } +}