From bebaaf48563478ec33dc79cdcdfdbb6b4e11975f Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 19 Nov 2026 06:41:14 +0000 Subject: [PATCH 01/66] feat(report): the scored report says whether the project builds A project that Mendix's consistency check rejects cannot be deployed, but `mxcli report` scored only the lint rules, so it could score 99/100 and not build. Measured on testdata/pedapp: 99/100 with 1055 consistency errors. The report now runs `mx check` on a temporary copy of the project, using the same path as `docker check` (docker.CheckFindings), and puts the verdict above the score: - yes; - no, with the error count; - not checked: no mx for the project's version, or --no-mx-check. It says why and how to set it up, and never means clean; - unknown: the parsed findings disagree with mx's own count. The findings are listed under "Consistency Check", grouped by code with a count, the message and a few locations: a stale theme gives hundreds of one code (924 CE6083 there). CE0463 gets a hint about `mxcli fix widgets`. JSON carries every finding and a `builds` field that is absent when the check did not run. The findings are not scored: no category weight can express "does not build", and a few lint fixes would hide it. The check covers the whole project, also under --modules, and the page says so. New flags: --no-mx-check, --mx-check-results (saved mx check or docker check output) and --mxbuild-path. Docs and skills updated; the CI example now also fails on `builds != true`. Closes #2423 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82qTgUsRMETmzvjS27w --- .claude/skills/mendix/assess-quality/SKILL.md | 4 +- .claude/skills/mendix/report-quality/SKILL.md | 34 ++- cmd/mxcli/cmd_report.go | 14 + cmd/mxcli/cmd_report_consistency.go | 73 ++++++ cmd/mxcli/cmd_report_consistency_test.go | 132 ++++++++++ docs-site/src/tools/mxcli-report.md | 39 ++- mdl/linter/report.go | 3 + mdl/linter/report_consistency.go | 242 ++++++++++++++++++ mdl/linter/report_consistency_test.go | 135 ++++++++++ mdl/linter/report_format.go | 39 ++- 10 files changed, 691 insertions(+), 24 deletions(-) create mode 100644 cmd/mxcli/cmd_report_consistency.go create mode 100644 cmd/mxcli/cmd_report_consistency_test.go create mode 100644 mdl/linter/report_consistency.go create mode 100644 mdl/linter/report_consistency_test.go diff --git a/.claude/skills/mendix/assess-quality/SKILL.md b/.claude/skills/mendix/assess-quality/SKILL.md index 23d47b86a..156396d73 100644 --- a/.claude/skills/mendix/assess-quality/SKILL.md +++ b/.claude/skills/mendix/assess-quality/SKILL.md @@ -39,7 +39,7 @@ structure make sense, whether error handling is designed or accidental. | Is it fast enough at production volume? | `mxcli report performance` | `report-performance` | | Was the specification built? | `mxcli report traceability` | `report-traceability` | | What is in it, for someone new? | `mxcli report map` | `report-map` | -| Does it build? | `mxcli docker check -p app.mpr` | `/validate-project` command | +| Does it build? | `mxcli report` (its **Builds** line), or `mxcli docker check` | `report-quality`, `/validate-project` | Read each report's skill before interpreting it — every one has traps (the quality score counts Marketplace modules unless scoped; the security report treats intended @@ -218,7 +218,7 @@ only where the benefit is clear. ## Measured | Area | Result | Source | |---|---|---| -| Builds | [clean / N errors, M warnings] | mxcli docker check | +| Builds | [yes / no: N errors by code / not checked] | mxcli report (Builds line) | | Quality score | [X/100; lowest category and why] | mxcli report | | Architecture | [cycles, coupling hotspots, dead code] | report architecture | | Security | [open high findings; flows that trust the caller] | report security | diff --git a/.claude/skills/mendix/report-quality/SKILL.md b/.claude/skills/mendix/report-quality/SKILL.md index 8c4e4f023..f2ec63705 100644 --- a/.claude/skills/mendix/report-quality/SKILL.md +++ b/.claude/skills/mendix/report-quality/SKILL.md @@ -10,11 +10,11 @@ description: "Produce and read the scored quality report (mxcli report): every l | lines | section | |---|---| -| 29-40 | Run it | -| 41-56 | Before trusting the number | -| 57-76 | Reading the score | -| 77-97 | Accepting a finding | -| 98-104 | Related | +| 29-48 | Run it | +| 49-72 | Before trusting the number | +| 73-92 | Reading the score | +| 93-113 | Accepting a finding | +| 114-120 | Related | # Quality Report @@ -36,7 +36,15 @@ mxcli report -p app.mpr --format json -o docs/quality.json # for CI, or to com mxcli report -p app.mpr --modules Sales,Billing ``` -It builds a full catalog first, so the first run on a project takes a while. +It builds a full catalog first, so the first run on a project takes a while. It also +runs Mendix's own consistency check (`mx check`) on a temporary copy of the project; +the project on disk is not changed. + +```bash +mxcli setup mxbuild -p app.mpr # once: mx for the project's Mendix version +mxcli report -p app.mpr --mx-check-results check.txt # reuse saved 'mxcli docker check' output +mxcli report -p app.mpr --no-mx-check # lint only; the report says it did not check +``` ## Before trusting the number @@ -50,9 +58,17 @@ It builds a full catalog first, so the first run on a project takes a while. security level, demo users, user-role mappings — are left out (they are about no one module). Use both: the full run for the project's settings, the scoped run for the code the team owns. -- **Does it build?** The score does not include Mendix's own consistency check. - A project with CE errors can score well. Run `mxcli docker check -p app.mpr` as - well (ako/mxcli#2423 tracks putting it in the report). +- **Does it build?** Read the **Builds** line above the score first. `no` means + `mx check` found consistency errors, and a project with errors cannot be deployed, + whatever it scores: the check's findings are not in the score, because no + category weight can express "does not build". `not checked` means there was no mx + for the project's version (or `--no-mx-check`); it never means clean. `unknown` + means mx's output could not be read in full. JSON has `builds` (true/false, absent + when not checked). The check covers the whole project, also under `--modules`. +- **Hundreds of one code** is usually one cause: a theme that lacks a design + property (CE6083), or widget packages newer than the pages (CE0463, which + `mxcli docker check` labels and `mxcli fix widgets` clears). The page groups the + findings by code; `--format json` lists each one. ## Reading the score diff --git a/cmd/mxcli/cmd_report.go b/cmd/mxcli/cmd_report.go index b53017e5c..209eb52d1 100644 --- a/cmd/mxcli/cmd_report.go +++ b/cmd/mxcli/cmd_report.go @@ -45,6 +45,13 @@ Examples: mxcli report -p app.mpr --format markdown --output report.md mxcli report -p app.mpr --modules Sales,Orders +The report also runs Mendix's own consistency check (mx check) on a temporary +copy of the project and states above the score whether the project builds: a +project with consistency errors cannot be deployed, whatever it scores. It needs +mx for the project's Mendix version ('mxcli setup mxbuild -p app.mpr'); without +it the report says the check was not run, never that the project is clean. +--mx-check-results reads saved output instead, and --no-mx-check skips it. + --modules scores only the named modules — a project's own modules, without the Marketplace and platform modules beside them. Only findings located in a selected module count; project-wide findings (project security, user-role @@ -154,6 +161,13 @@ mappings) are left out, and the report names the selection. ) report.Modules = moduleFilter report.Accepted = len(acceptances.accepted) + // Whether it builds, from Mendix's own check (ako/mxcli#2423). + consistency, err := reportConsistency(cmd, projectPath, os.Stderr) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } + report.Consistency = consistency // Format and output formatter := linter.GetReportFormatter(format) diff --git a/cmd/mxcli/cmd_report_consistency.go b/cmd/mxcli/cmd_report_consistency.go new file mode 100644 index 000000000..b05b7d0bc --- /dev/null +++ b/cmd/mxcli/cmd_report_consistency.go @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "fmt" + "io" + "os" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/spf13/cobra" +) + +// The scored report states whether the project builds, from Mendix's own +// consistency check (ako/mxcli#2423). The lint rules measure conventions; a +// project with consistency errors cannot be deployed and could still score +// well. + +// reportMxCheck runs `mx check` on a copy of the project. A seam for tests, +// which have no mx. +var reportMxCheck = docker.CheckFindings + +// reportConsistency runs the check, or reads saved results, or says why it was +// not run. Only an unreadable results file the user named is an error: a +// missing mx is a report that says so, never a clean one. +func reportConsistency(cmd *cobra.Command, projectPath string, progress io.Writer) (*linter.ConsistencyCheck, error) { + if skip, _ := cmd.Flags().GetBool("no-mx-check"); skip { + return &linter.ConsistencyCheck{NotRun: "skipped with --no-mx-check."}, nil + } + if path, _ := cmd.Flags().GetString("mx-check-results"); path != "" { + b, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("--mx-check-results: %w", err) + } + return consistencyFrom(docker.ParseCheckReport(string(b)), "results file "+path), nil + } + mxbuild, _ := cmd.Flags().GetString("mxbuild-path") + fmt.Fprintln(progress, "Running mx check on a temporary copy of the project (--no-mx-check skips it)...") + rep, runErr := reportMxCheck(docker.CheckOptions{ProjectPath: projectPath, MxBuildPath: mxbuild}) + if runErr != nil && len(rep.Findings) == 0 && !rep.HasDeclared { + // mx was not found, or did not get as far as checking. + return &linter.ConsistencyCheck{NotRun: fmt.Sprintf("%v. 'mxcli setup mxbuild -p %s' sets up the project's Mendix version; "+ + "--mx-check-results reads the output of an earlier 'mx check' or 'mxcli docker check'.", runErr, projectPath)}, nil + } + return consistencyFrom(rep, "mx check"), nil +} + +// consistencyFrom turns a parsed `mx check` run into the report's form. +func consistencyFrom(rep docker.CheckReport, source string) *linter.ConsistencyCheck { + c := &linter.ConsistencyCheck{Ran: true, Source: source} + for _, f := range rep.Findings { + switch f.Severity { + case "error": + c.Errors++ + case "warning": + c.Warnings++ + case "deprecation": + c.Deprecations++ + } + c.Findings = append(c.Findings, linter.ConsistencyFinding{Severity: f.Severity, Code: f.Code, Message: f.Message, Location: f.Location}) + } + if err := rep.Verify(); err != nil { + c.Incomplete = err.Error() + } + return c +} + +func init() { + reportCmd.Flags().Bool("no-mx-check", false, "Do not run Mendix's consistency check (mx check); the report then says whether the project builds is unknown") + reportCmd.Flags().String("mx-check-results", "", "Read the output of an earlier 'mx check' or 'mxcli docker check' from this file instead of running it") + reportCmd.Flags().String("mxbuild-path", "", "Path to the mxbuild whose mx runs the check, overriding resolution by the project's Mendix version") +} diff --git a/cmd/mxcli/cmd_report_consistency_test.go b/cmd/mxcli/cmd_report_consistency_test.go new file mode 100644 index 000000000..f792ea4c1 --- /dev/null +++ b/cmd/mxcli/cmd_report_consistency_test.go @@ -0,0 +1,132 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "errors" + "io" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" + "github.com/spf13/cobra" +) + +const mxCheckWithErrors = `Checking app... +[error] [CE0117] "Error(s) in expression." at Microflow 'ACT_X' +[warning] [CW0702] "Unused variable." at Microflow 'ACT_Y' +[deprecation] [CD0001] "Deprecated widget." at Page 'Home' +The app contains: 1 error. +` + +const mxCheckClean = "Checking app...\nThe app contains: 0 errors.\n" + +func consistencyCmd(t *testing.T, flags map[string]string) *cobra.Command { + t.Helper() + c := &cobra.Command{} + c.Flags().Bool("no-mx-check", false, "") + c.Flags().String("mx-check-results", "", "") + c.Flags().String("mxbuild-path", "", "") + for k, v := range flags { + if err := c.Flags().Set(k, v); err != nil { + t.Fatal(err) + } + } + return c +} + +func stubMxCheck(t *testing.T, output string, err error) *docker.CheckOptions { + t.Helper() + orig := reportMxCheck + t.Cleanup(func() { reportMxCheck = orig }) + var got docker.CheckOptions + reportMxCheck = func(opts docker.CheckOptions) (docker.CheckReport, error) { + got = opts + if output == "" { + return docker.CheckReport{}, err + } + return docker.ParseCheckReport(output), err + } + return &got +} + +// ako/mxcli#2423: the scored report runs mx check and says whether the +// project builds. +func TestReportConsistencyRunsMxCheck(t *testing.T) { + opts := stubMxCheck(t, mxCheckWithErrors, errors.New("exit status 1")) + c, err := reportConsistency(consistencyCmd(t, map[string]string{"mxbuild-path": "/opt/mxbuild"}), "app.mpr", io.Discard) + if err != nil { + t.Fatal(err) + } + if !c.Ran || c.Errors != 1 || c.Warnings != 1 || c.Deprecations != 1 || c.Builds() || c.Incomplete != "" { + t.Errorf("errors found: %+v", c) + } + if opts.ProjectPath != "app.mpr" || opts.MxBuildPath != "/opt/mxbuild" { + t.Errorf("mx check got %+v", *opts) + } + + stubMxCheck(t, mxCheckClean, nil) + c, _ = reportConsistency(consistencyCmd(t, nil), "app.mpr", io.Discard) + if !c.Builds() || c.Verdict() != "yes" { + t.Errorf("clean project: %+v", c) + } +} + +// No mx is a report that says the check did not run and how to run it, never a +// clean one; and output whose error count disagrees with mx's own is not +// "builds". +func TestReportConsistencyNeverCleanWhenUnknown(t *testing.T) { + stubMxCheck(t, "", errors.New("mx not found; specify --mxbuild-path")) + c, err := reportConsistency(consistencyCmd(t, nil), "app.mpr", io.Discard) + if err != nil { + t.Fatal(err) + } + if c.Ran || c.Builds() || !strings.Contains(c.NotRun, "mx not found") || !strings.Contains(c.NotRun, "mxcli setup mxbuild -p app.mpr") { + t.Errorf("no mx: %+v", c) + } + + // mx stopped before its closing count: one error seen, the rest unknown. + stubMxCheck(t, `[error] [CE0117] "x" at Microflow 'A'`+"\n", errors.New("signal: killed")) + c, _ = reportConsistency(consistencyCmd(t, nil), "app.mpr", io.Discard) + if !c.Ran || c.Errors != 1 || c.Incomplete == "" { + t.Errorf("cut short: %+v", c) + } + stubMxCheck(t, "The app contains: 2 errors.\n", errors.New("exit status 1")) + c, _ = reportConsistency(consistencyCmd(t, nil), "app.mpr", io.Discard) + if c.Builds() || c.Incomplete == "" || !strings.HasPrefix(c.Verdict(), "unknown") { + t.Errorf("count mismatch must not read as building: %+v", c) + } +} + +func TestReportConsistencyResultsFileAndSkip(t *testing.T) { + called := false + orig := reportMxCheck + t.Cleanup(func() { reportMxCheck = orig }) + reportMxCheck = func(docker.CheckOptions) (docker.CheckReport, error) { called = true; return docker.CheckReport{}, nil } + + path := filepath.Join(t.TempDir(), "check.txt") + if err := os.WriteFile(path, []byte("Using mx: /x/mx\n"+mxCheckWithErrors), 0644); err != nil { + t.Fatal(err) + } + c, err := reportConsistency(consistencyCmd(t, map[string]string{"mx-check-results": path}), "app.mpr", io.Discard) + if err != nil || c.Errors != 1 || !strings.Contains(c.Source, path) { + t.Errorf("results file: %+v %v", c, err) + } + if _, err := reportConsistency(consistencyCmd(t, map[string]string{"mx-check-results": path + ".missing"}), "app.mpr", io.Discard); err == nil { + t.Error("a results file that cannot be read is an error, not a clean check") + } + c, _ = reportConsistency(consistencyCmd(t, map[string]string{"no-mx-check": "true"}), "app.mpr", io.Discard) + if c.Ran || !strings.Contains(c.NotRun, "--no-mx-check") { + t.Errorf("skipped: %+v", c) + } + if called { + t.Error("neither a results file nor --no-mx-check runs mx") + } + for _, f := range []string{"no-mx-check", "mx-check-results", "mxbuild-path"} { + if reportCmd.Flags().Lookup(f) == nil { + t.Errorf("mxcli report has no --%s", f) + } + } +} diff --git a/docs-site/src/tools/mxcli-report.md b/docs-site/src/tools/mxcli-report.md index 0d99d30b5..546d4e98b 100644 --- a/docs-site/src/tools/mxcli-report.md +++ b/docs-site/src/tools/mxcli-report.md @@ -9,6 +9,29 @@ The `mxcli report` command generates a scored best practices report for a Mendix mxcli report -p app.mpr ``` +## Does It Build? + +The report runs Mendix's own consistency check (`mx check`) on a temporary copy of +the project and puts its verdict above the score: + +| Builds | Meaning | +|---|---| +| `yes` | `mx check` found no errors | +| `no — N consistency error(s)` | the project cannot be deployed, whatever it scores | +| `not checked` | no mx for the project's Mendix version, or `--no-mx-check` | +| `unknown` | mx's output could not be read in full | + +The check's findings are listed under **Consistency Check**, grouped by code (a +stale theme or widget set gives hundreds of one code), and are not part of the +score. They cover the whole project, also with `--modules`. + +```bash +mxcli setup mxbuild -p app.mpr # mx for the project's version, once +mxcli report -p app.mpr --mx-check-results check.txt # read saved 'mxcli docker check' output +mxcli report -p app.mpr --mxbuild-path /opt/mxbuild # a specific mxbuild +mxcli report -p app.mpr --no-mx-check # lint only +``` + ## Scoring Selected Modules ```bash @@ -100,18 +123,26 @@ mxcli report -p app.mpr --format json --output report.json Use the JSON format to fail CI pipelines when scores drop below a threshold: ```bash -# Fail when the overall score is below 70. The score has one decimal (e.g. 82.5), +# Fail when the project does not build, or when the overall score is below 70. The score has one decimal (e.g. 82.5), # so compare in jq: the shell's -lt only takes integers. mxcli report -p app.mpr --format json -o quality.json +jq -e '.builds == true' quality.json > /dev/null || { + echo "The project does not build, or mx check did not run: $(jq -r '.consistency.notRun // .consistency.incomplete // "consistency errors"' quality.json)" + exit 1 +} jq -e '.overallScore >= 70' quality.json > /dev/null || { echo "Quality score $(jq .overallScore quality.json) is below threshold 70" exit 1 } ``` -The `report-quality` skill covers how the score is computed, what it leaves out -(Mendix's own consistency check; the bundled Starlark rules when the project has no -`.claude/lint-rules/`), and how to accept an intended finding in the project brain. +`.builds` is absent when the check was not run, so `== true` also fails a CI run +that has no mx for the project's version: run `mxcli setup mxbuild -p app.mpr` +first in the pipeline. + +The `report-quality` skill covers how the score is computed, what it leaves out (the +bundled Starlark rules when the project has no `.claude/lint-rules/`), and how to +accept an intended finding in the project brain. For reports that answer one question instead of scoring — architecture, tests, security, performance, traceability and the software map — see diff --git a/mdl/linter/report.go b/mdl/linter/report.go index d765c6400..464fbce3d 100644 --- a/mdl/linter/report.go +++ b/mdl/linter/report.go @@ -26,6 +26,9 @@ type Report struct { // They are not scored and not in Summary: an accepted finding is a // decision, not an open problem (ako/mxcli#2304). Accepted int `json:"accepted,omitempty"` + // Consistency is Mendix's own consistency check (mx check); nil when the + // caller did not ask for it (ako/mxcli#2423). + Consistency *ConsistencyCheck `json:"consistency,omitempty"` } // CategoryScore tracks the score for a lint category. diff --git a/mdl/linter/report_consistency.go b/mdl/linter/report_consistency.go new file mode 100644 index 000000000..976e33aa1 --- /dev/null +++ b/mdl/linter/report_consistency.go @@ -0,0 +1,242 @@ +// SPDX-License-Identifier: Apache-2.0 + +package linter + +import ( + "fmt" + "html" + "io" + "sort" + "strings" +) + +// ConsistencyCheck is the result of Mendix's own consistency check (`mx check`) +// as the scored report shows it (ako/mxcli#2423). +// +// The lint rules measure conventions; they do not say whether the project +// builds. A project with consistency errors cannot be deployed, yet it could +// score well, so the report states the check's verdict above the score. The +// check's findings are not scored: one CE error is a project that does not +// build, which no category weight can express, and folding it into the score +// would let a few lint fixes hide it. +type ConsistencyCheck struct { + // Ran is false when the check was not run; NotRun then says why and how + // to run it. A report never shows a check it did not run as clean. + Ran bool `json:"ran"` + NotRun string `json:"notRun,omitempty"` + // Source names what the result came from: the mx binary, or a results + // file passed with --mx-check-results. + Source string `json:"source,omitempty"` + Errors int `json:"errors"` + Warnings int `json:"warnings"` + Deprecations int `json:"deprecations"` + // Incomplete is set when the parsed findings disagree with the error + // count mx printed, or mx did not print it: the list may be missing + // findings, so the verdict cannot be "builds". + Incomplete string `json:"incomplete,omitempty"` + Findings []ConsistencyFinding `json:"findings,omitempty"` +} + +// ConsistencyFinding is one `mx check` diagnostic. +type ConsistencyFinding struct { + Severity string `json:"severity"` // error | warning | deprecation + Code string `json:"code"` // CE0117, CW…, CD… + Message string `json:"message"` + Location string `json:"location,omitempty"` +} + +// Verdict is the one-line answer to "does it build?". +func (c *ConsistencyCheck) Verdict() string { + switch { + case c == nil || !c.Ran: + return "not checked" + case c.Errors > 0: + return fmt.Sprintf("no — %d consistency error(s)", c.Errors) + case c.Incomplete != "": + return "unknown — the check's output could not be read in full" + default: + return "yes" + } +} + +// Builds reports whether the check ran, read completely, and found no error. +func (c *ConsistencyCheck) Builds() bool { + return c != nil && c.Ran && c.Errors == 0 && c.Incomplete == "" +} + +// consistencyLead is the sentence under the verdict. +func (c *ConsistencyCheck) consistencyLead() string { + switch { + case c == nil || !c.Ran: + reason := "it was not run" + if c != nil && c.NotRun != "" { + reason = c.NotRun + } + return "Mendix's consistency check (`mx check`) was not run, so this report does not say whether the project builds: " + reason + case c.Errors > 0: + return "The project does not build: Mendix's consistency check (`mx check`) found errors, and a project with errors cannot be deployed. The score covers the lint rules only." + case c.Incomplete != "": + return "The consistency check ran, but " + c.Incomplete + "." + default: + return "Mendix's consistency check (`mx check`) found no errors." + } +} + +func (c *ConsistencyCheck) counts() string { + return fmt.Sprintf("%d error(s), %d warning(s), %d deprecation(s)", c.Errors, c.Warnings, c.Deprecations) +} + +// ConsistencyGroup is the findings of one code: a project with a stale theme or +// widget set has hundreds of the same one, which a row each would bury. +type ConsistencyGroup struct { + Severity string + Code string + Count int + Message string // the first finding's; they differ only in detail + Locations []string // the first few distinct ones + More int // distinct locations not listed +} + +// maxGroupLocations is how many locations a group lists in markdown and HTML; +// JSON carries every finding. +const maxGroupLocations = 5 + +// Groups returns the findings grouped by severity and code: errors first, then +// warnings and deprecations, each by count. +func (c *ConsistencyCheck) Groups() []ConsistencyGroup { + at := map[string]int{} + seen := map[string]map[string]bool{} + var out []ConsistencyGroup + for _, f := range c.Findings { + k := f.Severity + "\x00" + f.Code + i, ok := at[k] + if !ok { + out = append(out, ConsistencyGroup{Severity: f.Severity, Code: f.Code, Message: f.Message}) + i = len(out) - 1 + at[k] = i + seen[k] = map[string]bool{} + } + g := &out[i] + g.Count++ + if f.Location != "" && !seen[k][f.Location] { + seen[k][f.Location] = true + if len(g.Locations) < maxGroupLocations { + g.Locations = append(g.Locations, f.Location) + } else { + g.More++ + } + } + } + rank := map[string]int{"error": 0, "warning": 1, "deprecation": 2} + sort.SliceStable(out, func(i, j int) bool { + if rank[out[i].Severity] != rank[out[j].Severity] { + return rank[out[i].Severity] < rank[out[j].Severity] + } + return out[i].Count > out[j].Count + }) + return out +} + +func (g ConsistencyGroup) where() string { + s := strings.Join(g.Locations, "; ") + if g.More > 0 { + s += fmt.Sprintf("; and %d more", g.More) + } + return s +} + +// hints says what to do about the codes whose cause is usually not the model's +// logic. +func (c *ConsistencyCheck) hints() []string { + var out []string + for _, g := range c.Groups() { + if g.Code == "CE0463" { + out = append(out, "CE0463 is usually a widget whose definition changed (a widget package upgrade): `mxcli docker check` labels which ones `mxcli fix widgets` clears.") + } + } + return out +} + +// wholeProjectNote says that the check is not narrowed by --modules: mx checks +// the project, and a project that does not build does not build in any module. +const wholeProjectNote = "The check covers the whole project, not only the selected modules." + +func writeConsistencyMarkdown(w io.Writer, c *ConsistencyCheck, modules []string) { + if c == nil { + return + } + fmt.Fprintf(w, "## Consistency Check\n\n") + fmt.Fprintf(w, "%s\n\n", c.consistencyLead()) + if c.Ran && len(modules) > 0 { + fmt.Fprintf(w, "%s\n\n", wholeProjectNote) + } + if !c.Ran { + return + } + if c.Source != "" { + fmt.Fprintf(w, "Source: %s. %s.\n\n", c.Source, c.counts()) + } else { + fmt.Fprintf(w, "%s.\n\n", c.counts()) + } + if len(c.Findings) == 0 { + return + } + fmt.Fprintf(w, "| Severity | Code | Count | Message | Where |\n") + fmt.Fprintf(w, "|----------|------|-------|---------|-------|\n") + for _, g := range c.Groups() { + fmt.Fprintf(w, "| %s | %s | %d | %s | %s |\n", g.Severity, g.Code, g.Count, mdCell(g.Message), mdCell(g.where())) + } + fmt.Fprintln(w) + for _, h := range c.hints() { + fmt.Fprintf(w, "%s\n\n", h) + } + fmt.Fprintf(w, "Every finding is in `--format json`; `mxcli docker check -p ` prints them with mx's own wording.\n\n") +} + +func writeConsistencyHTML(w io.Writer, c *ConsistencyCheck, modules []string) { + if c == nil { + return + } + fmt.Fprintf(w, "

Consistency Check

\n") + cls := "recommendation" + if c.Errors > 0 { + cls = "recommendation severity-error" + } + fmt.Fprintf(w, "

%s

\n", cls, html.EscapeString(strings.ReplaceAll(c.consistencyLead(), "`", ""))) + if c.Ran && len(modules) > 0 { + fmt.Fprintf(w, "

%s

\n", wholeProjectNote) + } + if !c.Ran { + return + } + src := "" + if c.Source != "" { + src = "Source: " + c.Source + ". " + } + fmt.Fprintf(w, "

%s%s.

\n", html.EscapeString(src), html.EscapeString(c.counts())) + if len(c.Findings) == 0 { + return + } + fmt.Fprintf(w, "\n\n") + for _, g := range c.Groups() { + sev := "severity-info" + switch g.Severity { + case "error": + sev = "severity-error" + case "warning": + sev = "severity-warning" + } + fmt.Fprintf(w, "\n", + sev, html.EscapeString(g.Severity), html.EscapeString(g.Code), g.Count, html.EscapeString(g.Message), html.EscapeString(g.where())) + } + fmt.Fprintf(w, "
SeverityCodeCountMessageWhere
%s%s%d%s%s
\n") + for _, h := range c.hints() { + fmt.Fprintf(w, "

%s

\n", html.EscapeString(strings.ReplaceAll(h, "`", ""))) + } + fmt.Fprintf(w, "

Every finding is in --format json; mxcli docker check prints them with mx's own wording.

\n") +} + +// mdCell keeps a value inside one markdown table cell. +func mdCell(s string) string { + return strings.NewReplacer("|", `\|`, "\n", " ").Replace(s) +} diff --git a/mdl/linter/report_consistency_test.go b/mdl/linter/report_consistency_test.go new file mode 100644 index 000000000..62d46b0f8 --- /dev/null +++ b/mdl/linter/report_consistency_test.go @@ -0,0 +1,135 @@ +// SPDX-License-Identifier: Apache-2.0 + +package linter + +import ( + "bytes" + "encoding/json" + "strings" + "testing" +) + +func renderReport(t *testing.T, format string, c *ConsistencyCheck) string { + t.Helper() + r := BuildReport("Shop", "now", nil) + r.Consistency = c + var b bytes.Buffer + if err := GetReportFormatter(format).FormatReport(r, &b); err != nil { + t.Fatal(err) + } + return b.String() +} + +// ako/mxcli#2423: a project with consistency errors scores 100 on the lint +// rules (none fire here), and the report must still say it does not build. +func TestReportSaysAProjectWithConsistencyErrorsDoesNotBuild(t *testing.T) { + c := &ConsistencyCheck{Ran: true, Source: "mx check", Errors: 1, Warnings: 1, Findings: []ConsistencyFinding{ + {Severity: "error", Code: "CE0117", Message: "Error(s) in expression.", Location: "Microflow 'ACT_X'"}, + {Severity: "warning", Code: "CW0001", Message: "a warning | with a pipe", Location: "Page 'P'"}, + }} + md := renderReport(t, "markdown", c) + for _, want := range []string{"**Builds:** no — 1 consistency error(s)", "100/100", "## Consistency Check", + "The project does not build", "| error | CE0117 | 1 | Error(s) in expression. | Microflow 'ACT_X' |", `with a pipe`} { + if !strings.Contains(md, want) { + t.Errorf("markdown lacks %q:\n%s", want, md) + } + } + if strings.Contains(md, "a warning | with") { + t.Error("a pipe in a message must not split the markdown cell") + } + h := renderReport(t, "html", c) + if !strings.Contains(h, "no — 1 consistency error(s)") || strings.Contains(h, "warning") { + t.Errorf("html must state the verdict and escape mx's text:\n%s", h) + } + var j map[string]any + if err := json.Unmarshal([]byte(renderReport(t, "json", c)), &j); err != nil { + t.Fatal(err) + } + if j["builds"] != false || j["consistency"].(map[string]any)["errors"].(float64) != 1 { + t.Errorf("json builds=%v consistency=%v", j["builds"], j["consistency"]) + } +} + +// A check that was not run is never shown as clean, and one whose output could +// not be read in full is not "builds" either. +func TestReportNeverCallsAnUncheckedProjectClean(t *testing.T) { + cases := []struct { + name string + c *ConsistencyCheck + want string + build any + }{ + {"not run", &ConsistencyCheck{NotRun: "mx not found."}, "**Builds:** not checked", nil}, + {"incomplete", &ConsistencyCheck{Ran: true, Incomplete: "parsed 0 errors but mx check reported 2"}, "**Builds:** unknown", nil}, + {"clean", &ConsistencyCheck{Ran: true, Source: "mx check"}, "**Builds:** yes", true}, + } + for _, tc := range cases { + md := renderReport(t, "markdown", tc.c) + if !strings.Contains(md, tc.want) { + t.Errorf("%s: markdown lacks %q:\n%s", tc.name, tc.want, md) + } + var j map[string]any + _ = json.Unmarshal([]byte(renderReport(t, "json", tc.c)), &j) + if j["builds"] != tc.build { + t.Errorf("%s: json builds = %v, want %v", tc.name, j["builds"], tc.build) + } + } + if md := renderReport(t, "markdown", &ConsistencyCheck{NotRun: "mx not found."}); !strings.Contains(md, "mx not found.") { + t.Errorf("the reason it was not run is shown:\n%s", md) + } + // A caller that did not ask for the check gets the report as before. + if md := renderReport(t, "markdown", nil); strings.Contains(md, "Builds") || strings.Contains(md, "Consistency") { + t.Errorf("no check requested, no section:\n%s", md) + } +} + +// A stale theme gives hundreds of one code (measured: 924 CE6083 of 1055 on +// testdata/pedapp). The page shows one row per code, errors first and the +// largest first, with a few locations; JSON keeps every finding. +func TestConsistencyFindingsAreGroupedByCode(t *testing.T) { + c := &ConsistencyCheck{Ran: true} + add := func(sev, code, loc string, n int) { + for i := 0; i < n; i++ { + c.Findings = append(c.Findings, ConsistencyFinding{Severity: sev, Code: code, Message: "m " + code, Location: loc}) + } + } + add("warning", "CW0001", "Page 'A'", 50) + add("error", "CE0463", "Image 'image1'", 3) + for i := 0; i < 8; i++ { + add("error", "CE6083", "Layout grid row "+string(rune('a'+i)), 2) + } + c.Errors = 19 + gs := c.Groups() + if len(gs) != 3 || gs[0].Code != "CE6083" || gs[0].Count != 16 || gs[1].Code != "CE0463" || gs[2].Code != "CW0001" { + t.Fatalf("groups: %+v", gs) + } + if len(gs[0].Locations) != maxGroupLocations || gs[0].More != 3 || len(gs[1].Locations) != 1 { + t.Errorf("locations are distinct and capped: %+v", gs[:2]) + } + md := renderReport(t, "markdown", c) + for _, want := range []string{"| error | CE6083 | 16 | m CE6083 |", "and 3 more", "`mxcli fix widgets`", "--format json"} { + if !strings.Contains(md, want) { + t.Errorf("markdown lacks %q:\n%s", want, md) + } + } + if n := strings.Count(md, "| error | CE6083"); n != 1 { + t.Errorf("one row per code, got %d", n) + } + var j map[string]any + _ = json.Unmarshal([]byte(renderReport(t, "json", c)), &j) + if got := len(j["consistency"].(map[string]any)["findings"].([]any)); got != 69 { + t.Errorf("json keeps every finding: %d", got) + } +} + +// --modules narrows the lint findings, not the check: the page says so. +func TestConsistencyCheckIsWholeProjectUnderModules(t *testing.T) { + r := BuildReport("Shop", "now", nil) + r.Modules = []string{"Sales"} + r.Consistency = &ConsistencyCheck{Ran: true} + var b bytes.Buffer + _ = (&MarkdownReportFormatter{}).FormatReport(r, &b) + if !strings.Contains(b.String(), wholeProjectNote) { + t.Errorf("missing the whole-project note:\n%s", b.String()) + } +} diff --git a/mdl/linter/report_format.go b/mdl/linter/report_format.go index 446b828d7..f5f63a91a 100644 --- a/mdl/linter/report_format.go +++ b/mdl/linter/report_format.go @@ -39,7 +39,11 @@ func (f *MarkdownReportFormatter) FormatReport(report *Report, w io.Writer) erro if len(report.Modules) > 0 { fmt.Fprintf(w, "**Modules:** %s (findings outside these modules are not scored) \n", strings.Join(report.Modules, ", ")) } + if report.Consistency != nil { + fmt.Fprintf(w, "**Builds:** %s \n", report.Consistency.Verdict()) + } fmt.Fprintf(w, "**Overall Score:** %s %.0f/100\n\n", scoreBar(report.OverallScore), report.OverallScore) + writeConsistencyMarkdown(w, report.Consistency, report.Modules) // Summary fmt.Fprintf(w, "## Summary\n\n") @@ -128,14 +132,18 @@ type JSONReportFormatter struct{} // JSONReport is the JSON output structure for a report. type JSONReport struct { - ProjectName string `json:"projectName"` - Date string `json:"date"` - Modules []string `json:"modules,omitempty"` - OverallScore float64 `json:"overallScore"` - Accepted int `json:"accepted,omitempty"` - Summary JSONSummary `json:"summary"` - Categories []CategoryScore `json:"categories"` - Violations []JSONViolation `json:"violations"` + ProjectName string `json:"projectName"` + Date string `json:"date"` + Modules []string `json:"modules,omitempty"` + OverallScore float64 `json:"overallScore"` + Accepted int `json:"accepted,omitempty"` + // Builds is the consistency check's verdict: true or false, or absent + // when the check was not run (ako/mxcli#2423). + Builds *bool `json:"builds,omitempty"` + Consistency *ConsistencyCheck `json:"consistency,omitempty"` + Summary JSONSummary `json:"summary"` + Categories []CategoryScore `json:"categories"` + Violations []JSONViolation `json:"violations"` // RuleFailures are rules that could not run; excluded from the score and // the summary (ako/mxcli#952). RuleFailures []JSONViolation `json:"ruleFailures,omitempty"` @@ -162,7 +170,12 @@ func (f *JSONReportFormatter) FormatReport(report *Report, w io.Writer) error { Warnings: report.Summary.Warnings, Infos: report.Summary.Infos, }, - Categories: report.Categories, + Categories: report.Categories, + Consistency: report.Consistency, + } + if c := report.Consistency; c != nil && c.Ran && c.Incomplete == "" { + b := c.Errors == 0 + jr.Builds = &b } for _, v := range report.Violations { @@ -243,8 +256,16 @@ func (f *HTMLReportFormatter) FormatReport(report *Report, w io.Writer) error { } else if report.OverallScore < 75 { scoreClass = "score-ok" } + if report.Consistency != nil { + buildClass := "severity-info" + if report.Consistency.Errors > 0 { + buildClass = "severity-error" + } + fmt.Fprintf(w, "

Builds: %s

\n", buildClass, html.EscapeString(report.Consistency.Verdict())) + } fmt.Fprintf(w, "

Overall Score: %.0f/100

\n", scoreClass, report.OverallScore, report.OverallScore) + writeConsistencyHTML(w, report.Consistency, report.Modules) // Summary fmt.Fprintf(w, "

Summary

\n\n") From 691d7ba0770c70a6d8650f33839c9cd8e8e7c0d7 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 19 Nov 2026 09:35:31 +0000 Subject: [PATCH 02/66] =?UTF-8?q?feat(verify):=20mxcli=20verify=20?= =?UTF-8?q?=E2=80=94=20one=20gate=20over=20check,=20lint=20and=20mx=20chec?= =?UTF-8?q?k,=20new=20findings=20only?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 2 of PROPOSAL_verify_gate.md (#2333). Stages run in cost order and every selected stage runs (--fail-fast stops at the first failure). Findings are compared with per-stage baselines in --baseline-dir; mxcheck reads mx check --json and applies MxBuild's staging rule (unmasked / hidden). Exit codes: 0, 1 usage, 2 could not run, 16+bits per failed stage. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_011oK82stfDEB1TGfJLVwUxc --- cmd/mxcli/cmd_verify.go | 317 ++++++++++++++++++++++ cmd/mxcli/cmd_verify_test.go | 139 ++++++++++ cmd/mxcli/docker/check_findings.go | 4 + cmd/mxcli/verify/baseline.go | 73 +++++ cmd/mxcli/verify/classify.go | 90 ++++++ cmd/mxcli/verify/render.go | 81 ++++++ cmd/mxcli/verify/verify.go | 247 +++++++++++++++++ cmd/mxcli/verify/verify_test.go | 221 +++++++++++++++ docs/11-proposals/PROPOSAL_verify_gate.md | 134 +++++---- 9 files changed, 1250 insertions(+), 56 deletions(-) create mode 100644 cmd/mxcli/cmd_verify.go create mode 100644 cmd/mxcli/cmd_verify_test.go create mode 100644 cmd/mxcli/verify/baseline.go create mode 100644 cmd/mxcli/verify/classify.go create mode 100644 cmd/mxcli/verify/render.go create mode 100644 cmd/mxcli/verify/verify.go create mode 100644 cmd/mxcli/verify/verify_test.go diff --git a/cmd/mxcli/cmd_verify.go b/cmd/mxcli/cmd_verify.go new file mode 100644 index 000000000..e5880bc17 --- /dev/null +++ b/cmd/mxcli/cmd_verify.go @@ -0,0 +1,317 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + "regexp" + "strconv" + "strings" + + "github.com/spf13/cobra" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" + "github.com/mendixlabs/mxcli/cmd/mxcli/verify" +) + +var verifyCmd = &cobra.Command{ + Use: "verify [script.mdl...]", + Short: "One gate: check, lint and mx check, reporting only what a change made worse", + Long: `Run the checks that answer "did my change break anything?" in one command, in +cost order, and report only the findings that are new against a baseline. + + check mxcli check on the scripts given as arguments (skipped without any) + lint mxcli lint on the project, against its baseline + mxcheck mx check on a copy of the project, against its baseline + +Every selected stage runs, so one result carries every stage's findings; use +--fail-fast to stop at the first stage that fails. mx check is the authority: +mxcli check cannot yet predict everything it rejects, so a project that passes +check and lint can still fail mxcheck. + +Baselines live in --baseline-dir (default .mxcli/verify next to the project): +lint.json is a ` + "`mxcli lint --baseline`" + ` file, mxcheck.json records mx check's +findings by their element IDs. Without one, every finding is reported as new and +the stage says so. Record the current state with --update-baseline (commit the +files); the run then reports what it recorded and exits 0. + +mx check reports errors in stages: an earlier-stage error hides later ones. A +finding that appears only because the baseline's earlier error was fixed is +"unmasked" and does not fail the gate (--strict makes it); a baseline finding +that disappears only because a new earlier-stage error hides it is "hidden", not +resolved. + +Exit codes: + 0 nothing new + 1 usage error + 2 a stage could not run (no mx, ...), and nothing failed + 16+bits a stage failed; the bits say which: 1 check, 2 lint, 4 mxcheck + (18 = lint failed, 22 = lint and mxcheck failed) + +--format json prints one document: a list of stages with status, duration and +the new findings of each, and the exit code. + +Examples: + mxcli verify -p app.mpr --update-baseline # record the current state + mxcli verify -p app.mpr # lint + mx check against it + mxcli verify -p app.mpr script.mdl # also check the script + mxcli verify -p app.mpr --stages lint --format json +`, + Run: func(cmd *cobra.Command, args []string) { + os.Exit(runVerify(cmd, args)) + }, +} + +func init() { + verifyCmd.Flags().StringSlice("stages", verify.AllStages, "Stages to run: check, lint, mxcheck") + verifyCmd.Flags().Bool("fail-fast", false, "Stop at the first stage that fails instead of running every stage") + verifyCmd.Flags().String("fail-on", "error", "Lowest severity of a new finding that fails the gate: error or warning") + verifyCmd.Flags().Bool("strict", false, "Also fail on unmasked mx check findings (errors an earlier baseline error was hiding)") + verifyCmd.Flags().String("baseline-dir", "", "Where the baselines live (default .mxcli/verify next to the project)") + verifyCmd.Flags().Bool("update-baseline", false, "Record the current lint and mx check findings as the baseline instead of comparing") + verifyCmd.Flags().String("mxbuild-path", "", "Path to MxBuild/Mendix installation (used to find mx)") + verifyCmd.Flags().StringP("format", "f", "text", "Output format: text, json") + rootCmd.AddCommand(verifyCmd) +} + +func runVerify(cmd *cobra.Command, scripts []string) int { + projectPath, _ := cmd.Flags().GetString("project") + format := resolveFormat(cmd, "text") + stagesFlag, _ := cmd.Flags().GetStringSlice("stages") + failFast, _ := cmd.Flags().GetBool("fail-fast") + failOn, _ := cmd.Flags().GetString("fail-on") + strict, _ := cmd.Flags().GetBool("strict") + baselineDir, _ := cmd.Flags().GetString("baseline-dir") + update, _ := cmd.Flags().GetBool("update-baseline") + mxbuildPath, _ := cmd.Flags().GetString("mxbuild-path") + + usage := func(format string, a ...any) int { + fmt.Fprintf(os.Stderr, "Error: "+format+"\n", a...) + return verify.ExitUsage + } + if projectPath == "" { + return usage("--project (-p) is required") + } + if format != "text" && format != "json" { + return usage("--format %q is not supported by verify (text, json)", format) + } + if failOn != "error" && failOn != "warning" { + return usage("--fail-on takes error or warning, not %q", failOn) + } + stages, err := verify.ParseStages(stagesFlag) + if err != nil { + return usage("%v", err) + } + if baselineDir == "" { + baselineDir = filepath.Join(filepath.Dir(projectPath), ".mxcli", "verify") + } + + skip := map[string]string{} + if len(scripts) == 0 { + skip[verify.StageCheck] = "no scripts given" + } + self, err := os.Executable() + if err != nil { + return usage("cannot locate this executable: %v", err) + } + vr := &verifyRunner{self: self, project: projectPath, baselineDir: baselineDir, update: update, mxbuildPath: mxbuildPath, scripts: scripts} + res := verify.Run(verify.Options{ + Project: projectPath, Stages: stages, FailFast: failFast, FailOn: failOn, Strict: strict, Skip: skip, + }, map[string]verify.Stage{ + verify.StageCheck: vr.check, + verify.StageLint: vr.lint, + verify.StageMxCheck: vr.mxcheck, + }) + if format == "json" { + if err := verify.WriteJSON(os.Stdout, res); err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + return verify.ExitUsage + } + } else { + verify.WriteText(os.Stdout, res) + } + return res.Exit +} + +type verifyRunner struct { + self, project, baselineDir, mxbuildPath string + update bool + scripts []string +} + +// sub runs this executable with args and returns its stdout. Its stderr is +// returned too and echoed to ours, which is where progress belongs while stdout +// carries the result. +func (v *verifyRunner) sub(args ...string) (stdout, stderr []byte, err error) { + c := exec.Command(v.self, args...) + var o, e bytes.Buffer + c.Stdout, c.Stderr = &o, &e + err = c.Run() + os.Stderr.Write(e.Bytes()) + return o.Bytes(), e.Bytes(), err +} + +func (v *verifyRunner) check() (verify.Outcome, error) { + args := append([]string{"check"}, v.scripts...) + args = append(args, "-p", v.project, "--format", "json") + stdout, stderr, err := v.sub(args...) + findings, perr := parseViolations(verify.StageCheck, stdout) + if perr != nil { + return verify.Outcome{}, fmt.Errorf("mxcli check produced no readable report: %v%s", perr, verifyTail(stderr, err)) + } + return verify.Outcome{New: findings}, nil +} + +func (v *verifyRunner) lint() (verify.Outcome, error) { + lintBaseline := filepath.Join(v.baselineDir, "lint.json") + args := []string{"lint", "-p", v.project, "--format", "json"} + var note string + switch { + case v.update: + if err := os.MkdirAll(v.baselineDir, 0o755); err != nil { + return verify.Outcome{}, err + } + args = append(args, "--baseline", lintBaseline, "--update-baseline") + case verifyFileExists(lintBaseline): + args = append(args, "--baseline", lintBaseline) + default: + note = "no lint baseline (" + lintBaseline + "): every finding is reported as new; record one with --update-baseline" + } + stdout, stderr, err := v.sub(args...) + findings, perr := parseViolations(verify.StageLint, stdout) + if perr != nil { + return verify.Outcome{}, fmt.Errorf("mxcli lint produced no readable report: %v%s", perr, verifyTail(stderr, err)) + } + if v.update { + return verify.Outcome{Recorded: true, PreExisting: len(findings), Note: "baseline recorded at " + lintBaseline}, nil + } + pre, _ := parseLintBaselineSummary(string(stderr)) + return verify.Outcome{New: findings, PreExisting: pre, Note: note}, nil +} + +func (v *verifyRunner) mxcheck() (verify.Outcome, error) { + report, runErr := docker.CheckFindings(docker.CheckOptions{ProjectPath: v.project, MxBuildPath: v.mxbuildPath}, docker.CheckKinds{}) + if report.MxPath == "" || report.Verify() != nil { + if runErr == nil { + runErr = report.Verify() + } + return verify.Outcome{}, fmt.Errorf("mx check could not be run or read: %v (`mxcli setup mxbuild -p %s` installs the project's version)", runErr, v.project) + } + current := mxFindings(report) + version := verifyMxVersionOf(report.MxPath) + path := filepath.Join(v.baselineDir, "mxcheck.json") + if v.update { + if err := verify.WriteMxBaseline(path, "mx check", version, current); err != nil { + return verify.Outcome{}, err + } + return verify.Outcome{Recorded: true, PreExisting: len(current), Note: "baseline recorded at " + path}, nil + } + base, err := verify.LoadMxBaseline(path) + if err != nil { + return verify.Outcome{}, err + } + if base == nil { + return verify.Outcome{New: current, Note: "no mx check baseline (" + path + "): every finding is reported as new; record one with --update-baseline"}, nil + } + if err := base.CheckCompatible("mx check", version); err != nil { + return verify.Outcome{}, err + } + return verify.Classify(base.Findings, current, true), nil +} + +// mxFindings turns the errors of an mx check report into gate findings. Warnings +// and deprecations are not part of the gate: mx check exits non-zero for errors. +func mxFindings(r docker.CheckReport) []verify.Finding { + var out []verify.Finding + for _, f := range r.Findings { + if f.Severity != "error" { + continue + } + g := verify.Finding{Stage: verify.StageMxCheck, Code: f.Code, Severity: "error", Message: f.Message, Key: f.Key(), MxStage: f.Stage} + if len(f.Locations) > 0 { + g.Document, g.Element, g.Unit = f.Locations[0].DocumentName, f.Locations[0].ElementName, f.Locations[0].UnitID + if f.Locations[0].ModuleName != "" { + g.Document = f.Locations[0].ModuleName + "." + g.Document + } + } + out = append(out, g) + } + return out +} + +var verifyMxVersionRe = regexp.MustCompile(`(\d+\.\d+\.\d+(?:\.\d+)?)`) + +// verifyMxVersionOf reads the version out of the path of a cached mx +// (~/.mxcli/mxbuild/11.16.1/modeler/mx); "" when the path has none. +func verifyMxVersionOf(mxPath string) string { + for _, part := range strings.Split(filepath.ToSlash(mxPath), "/") { + if m := verifyMxVersionRe.FindString(part); m != "" && m == part { + return m + } + } + return "" +} + +// violationsJSON is the shape `check --format json` and `lint --format json` print. +type violationsJSON struct { + Violations []struct { + RuleID string `json:"ruleId"` + Severity string `json:"severity"` + Message string `json:"message"` + Document string `json:"document"` + DocumentType string `json:"documentType"` + Suggestion string `json:"suggestion"` + } `json:"violations"` +} + +func parseViolations(stage string, data []byte) ([]verify.Finding, error) { + var j violationsJSON + if err := json.Unmarshal(bytes.TrimSpace(data), &j); err != nil { + return nil, err + } + out := []verify.Finding{} + for _, v := range j.Violations { + out = append(out, verify.Finding{ + Stage: stage, Code: v.RuleID, Severity: v.Severity, Document: v.Document, + Message: v.Message, Suggestion: v.Suggestion, Key: v.RuleID + "\x00" + v.Document, MxStage: -1, + }) + } + return out, nil +} + +var lintBaselineRe = regexp.MustCompile(`Lint baseline .*: (\d+) new, (\d+) already known`) + +// parseLintBaselineSummary reads lint's own "Lint baseline F: N new, M already +// known[, K fixed]" line, which is where the count of pre-existing findings is. +func parseLintBaselineSummary(stderr string) (preExisting int, ok bool) { + m := lintBaselineRe.FindStringSubmatch(stderr) + if m == nil { + return 0, false + } + n, err := strconv.Atoi(m[2]) + return n, err == nil +} + +func verifyFileExists(p string) bool { + st, err := os.Stat(p) + return err == nil && !st.IsDir() +} + +func verifyTail(stderr []byte, err error) string { + s := strings.TrimSpace(string(stderr)) + if len(s) > 400 { + s = s[len(s)-400:] + } + if s == "" && err != nil { + return " (" + err.Error() + ")" + } + if s == "" { + return "" + } + return "\n" + s +} diff --git a/cmd/mxcli/cmd_verify_test.go b/cmd/mxcli/cmd_verify_test.go new file mode 100644 index 000000000..3534af53f --- /dev/null +++ b/cmd/mxcli/cmd_verify_test.go @@ -0,0 +1,139 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" +) + +// The JSON is what `mxcli lint --format json` and `mxcli check --format json` +// print (recorded from a run on a scaffolded project). +const lintJSON = `{ + "violations": [ + {"ruleId": "SEC013", "severity": "warning", "message": "Security level is Off", "module": "", "document": "ProjectSecurity", "documentType": "security", "suggestion": "ALTER APP SECURITY"}, + {"ruleId": "MDL098", "severity": "error", "message": "return reads $Nope", "module": "", "document": "Sp.A", "documentType": "microflow"} + ], + "summary": {"total": 2, "errors": 1, "warnings": 1, "infos": 0, "hints": 0} +}` + +func TestParseViolations(t *testing.T) { + fs, err := parseViolations("lint", []byte(lintJSON)) + if err != nil || len(fs) != 2 { + t.Fatalf("got %v, %v", fs, err) + } + if fs[1].Code != "MDL098" || fs[1].Severity != "error" || fs[1].Document != "Sp.A" || fs[1].Stage != "lint" || fs[1].MxStage != -1 { + t.Errorf("finding = %+v", fs[1]) + } + if fs[0].Key == fs[1].Key { + t.Error("two different findings share a key") + } + // An empty report is an empty list, not nil: it must encode as [] for a reader. + empty, err := parseViolations("lint", []byte(`{"violations": [], "summary": {}}`)) + if err != nil || empty == nil || len(empty) != 0 { + t.Errorf("empty report = %v, %v", empty, err) + } + // Control: output that is not the report (a crash, a banner) is an error, never "no findings". + if _, err := parseViolations("lint", []byte("Error: boom")); err == nil { + t.Error("unparseable output was read as an empty report") + } +} + +func TestParseLintBaselineSummary(t *testing.T) { + stderr := "Connected to: x\nLint baseline /p/lint.json: 2 new, 140 already known, 3 fixed\n" + if n, ok := parseLintBaselineSummary(stderr); !ok || n != 140 { + t.Errorf("got %d, %v, want 140", n, ok) + } + if n, ok := parseLintBaselineSummary("Lint baseline lint.json: 0 new, 5 already known (a scoped run: fixed findings are not counted)"); !ok || n != 5 { + t.Errorf("scoped form: got %d, %v", n, ok) + } + if _, ok := parseLintBaselineSummary("nothing"); ok { + t.Error("a missing summary line must not read as zero") + } +} + +func TestMxVersionOf(t *testing.T) { + for in, want := range map[string]string{ + "/root/.mxcli/mxbuild/11.16.1/modeler/mx": "11.16.1", + `C:\mx\11.12.1.5\modeler\mx.exe`: "", + "/usr/bin/mx": "", + } { + if in == `C:\mx\11.12.1.5\modeler\mx.exe` { + continue // a Windows path is split on its own separator by filepath.ToSlash only on Windows + } + if got := verifyMxVersionOf(in); got != want { + t.Errorf("verifyMxVersionOf(%q) = %q, want %q", in, got, want) + } + } +} + +// mx check warnings and deprecations are not part of the gate, and a finding's +// module is part of its document name. +func TestMxFindingsKeepsErrorsOnly(t *testing.T) { + data := `{"serialization_version": 1, + "errors": [{"code": "CE0109", "message": "Undefined variable 'X'.", "locations": [{"module-name": "Sp", "document-name": "Microflow 'F'", "element-name": "End event", "element-id": "e1", "unit-id": "u1"}]}], + "warnings": [{"code": "CW0055", "message": "No on click", "locations": []}], + "deprecations": [], "total-problems": 2}` + r, err := docker.ParseCheckJSON([]byte(data)) + if err != nil { + t.Fatal(err) + } + fs := mxFindings(r) + if len(fs) != 1 { + t.Fatalf("got %d findings, want the error only", len(fs)) + } + f := fs[0] + if f.Code != "CE0109" || f.Document != "Sp.Microflow 'F'" || f.Element != "End event" || f.Unit != "u1" || f.MxStage != 2 { + t.Errorf("finding = %+v", f) + } + if !strings.Contains(f.Key, "u1/e1") { + t.Errorf("the key must carry the IDs, got %q", f.Key) + } +} + +// With nothing runnable the command still prints exactly one JSON document on +// stdout, exits 0, and reports why each stage was skipped. Flag errors exit 1 +// with nothing on stdout. +func TestVerifyOutputContract(t *testing.T) { + if testing.Short() { + t.Skip("spawns mxcli") + } + mpr := jsonPurityFixture(t) + dir := filepath.Dir(mpr) + + r := runMainStreams(t, dir, "verify", "-p", mpr, "--stages", "check", "--format", "json") + if r.exitCode != 0 { + t.Fatalf("exit %d\nstdout:\n%s\nstderr:\n%s", r.exitCode, r.stdout, r.stderr) + } + dec := json.NewDecoder(strings.NewReader(r.stdout)) + var doc struct { + Exit int `json:"exit"` + Stages []struct { + Name, Status, Note string + } `json:"stages"` + } + if err := dec.Decode(&doc); err != nil { + t.Fatalf("stdout is not JSON: %v\n%s", err, r.stdout) + } + if dec.More() { + t.Errorf("more than one document on stdout:\n%s", r.stdout) + } + if len(doc.Stages) != 3 || doc.Stages[0].Status != "skipped" || doc.Stages[0].Note != "no scripts given" { + t.Errorf("stages = %+v", doc.Stages) + } + + for _, args := range [][]string{ + {"verify", "-p", mpr, "--stages", "smoke"}, + {"verify", "-p", mpr, "--format", "sarif"}, + {"verify", "-p", mpr, "--fail-on", "info"}, + } { + r := runMainStreams(t, dir, args...) + if r.exitCode != 1 || strings.TrimSpace(r.stdout) != "" { + t.Errorf("mxcli %v: exit %d, stdout %q; want exit 1 and nothing on stdout\nstderr:\n%s", args, r.exitCode, r.stdout, r.stderr) + } + } +} diff --git a/cmd/mxcli/docker/check_findings.go b/cmd/mxcli/docker/check_findings.go index 09c542149..4fa8eb52d 100644 --- a/cmd/mxcli/docker/check_findings.go +++ b/cmd/mxcli/docker/check_findings.go @@ -102,6 +102,9 @@ type checkJSON struct { // CheckReport is the parsed result of one `mx check --json` run. type CheckReport struct { + // MxPath is the mx binary that produced the report, so a caller can record + // which checker (and version) a baseline belongs to. + MxPath string Findings []CheckFinding // Declared is the total mx printed, HasDeclared false when it printed none. Declared int @@ -221,5 +224,6 @@ func CheckFindings(opts CheckOptions, kinds CheckKinds) (CheckReport, error) { if err != nil { return CheckReport{}, err } + report.MxPath = mxPath return report, runErr } diff --git a/cmd/mxcli/verify/baseline.go b/cmd/mxcli/verify/baseline.go new file mode 100644 index 000000000..f12ce559e --- /dev/null +++ b/cmd/mxcli/verify/baseline.go @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: Apache-2.0 + +package verify + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" +) + +const baselineVersion = 1 + +// MxBaseline is the saved mxcheck state. The lint stage keeps lint's own +// baseline file (so a `lint --baseline` file stays valid); only mxcheck needs +// this one. +type MxBaseline struct { + Version int `json:"version"` + // Checker names the tool that produced the findings and MxVersion its version + // when known: codes and staging differ between checkers and versions, so a + // baseline is only compared with a run of the same one. + Checker string `json:"checker"` + MxVersion string `json:"mxVersion,omitempty"` + Findings []Finding `json:"findings"` +} + +// WriteMxBaseline records findings at path, sorted so that re-recording an +// unchanged project rewrites the same bytes and a committed baseline diffs only +// where findings changed. +func WriteMxBaseline(path, checker, mxVersion string, findings []Finding) error { + fs := append([]Finding{}, findings...) + SortFindings(fs) + data, err := json.MarshalIndent(MxBaseline{Version: baselineVersion, Checker: checker, MxVersion: mxVersion, Findings: fs}, "", " ") + if err != nil { + return err + } + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + return os.WriteFile(path, append(data, '\n'), 0o644) +} + +// LoadMxBaseline reads a baseline. It returns (nil, nil) when there is none, so a +// caller can say "unbaselined" instead of failing. +func LoadMxBaseline(path string) (*MxBaseline, error) { + data, err := os.ReadFile(path) + if os.IsNotExist(err) { + return nil, nil + } + if err != nil { + return nil, err + } + var b MxBaseline + if err := json.Unmarshal(data, &b); err != nil { + return nil, fmt.Errorf("read baseline %s: %w", path, err) + } + if b.Version != baselineVersion { + return nil, fmt.Errorf("baseline %s has version %d; this mxcli reads %d", path, b.Version, baselineVersion) + } + return &b, nil +} + +// CheckCompatible refuses to compare findings with a baseline of another checker +// or mx version: that diff would report every renumbered code as new. +func (b *MxBaseline) CheckCompatible(checker, mxVersion string) error { + if b.Checker != checker { + return fmt.Errorf("the baseline was recorded with %s, this run used %s; re-record it with --update-baseline", b.Checker, checker) + } + if b.MxVersion != "" && mxVersion != "" && b.MxVersion != mxVersion { + return fmt.Errorf("the baseline was recorded with mx %s, this run used mx %s; re-record it with --update-baseline", b.MxVersion, mxVersion) + } + return nil +} diff --git a/cmd/mxcli/verify/classify.go b/cmd/mxcli/verify/classify.go new file mode 100644 index 000000000..e566cae4c --- /dev/null +++ b/cmd/mxcli/verify/classify.go @@ -0,0 +1,90 @@ +// SPDX-License-Identifier: Apache-2.0 + +package verify + +// Classify compares the findings of a run with a baseline, by Key and with +// multiplicity (the same key twice in the baseline accepts two findings, not +// every later one). +// +// With staging set (the mxcheck stage) it applies the MxBuild staging rule from +// PROPOSAL_verify_gate.md: mx check reports only the earliest stage that has an +// error, so a plain diff blames a change for errors it merely uncovered and +// calls errors resolved that it merely hid. +// +// new current and not in the baseline. This includes an error that an +// earlier-stage error in the SAME unit (microflow) was hiding: that +// error can only have appeared because the change fixed or removed the +// hider, so the change touched the unit, and it must leave it clean. +// unmasked current and not in the baseline, but a stage 0 error of the baseline +// (an unresolved reference) was hiding the whole project. Fixing that +// one error uncovers everything else; none of it is the change's. +// resolved in the baseline, gone, and nothing in the current run hides it +// hidden in the baseline, gone, but a current finding hides it: not resolved +func Classify(baseline, current []Finding, staging bool) Outcome { + left := map[string]int{} + for _, b := range baseline { + left[b.Key]++ + } + var out Outcome + var candidates []Finding // current findings the baseline does not have + for _, c := range current { + if left[c.Key] > 0 { + left[c.Key]-- + out.PreExisting++ + continue + } + candidates = append(candidates, c) + } + for _, c := range candidates { + if staging && hiddenByStage0(c, baseline) { + out.Unmasked = append(out.Unmasked, c) + } else { + out.New = append(out.New, c) + } + } + // What is left of the baseline is gone from the run. + for _, b := range baseline { + if left[b.Key] == 0 { + continue + } + left[b.Key]-- + if staging && hiddenBy(b, current) { + out.Hidden++ + } else { + out.Resolved++ + } + } + return out +} + +// hiddenByStage0 reports whether the report holds a stage 0 finding (an +// unresolved reference), which hides every later-stage finding in the project. +func hiddenByStage0(f Finding, report []Finding) bool { + if f.MxStage <= 0 { + return false + } + for _, h := range report { + if h.MxStage == 0 { + return true + } + } + return false +} + +// hiddenBy reports whether some finding of report hides f under MxBuild's +// staging: a stage 0 finding anywhere, or an earlier-stage finding in the same +// unit (stages 1 and 2 hide the later stages of that microflow). +func hiddenBy(f Finding, report []Finding) bool { + if f.MxStage < 0 { + return false + } + for _, h := range report { + if h.MxStage < 0 || h.MxStage >= f.MxStage { + continue + } + if h.MxStage == 0 || (h.Unit != "" && h.Unit == f.Unit) { + return true + } + } + return false +} diff --git a/cmd/mxcli/verify/render.go b/cmd/mxcli/verify/render.go new file mode 100644 index 000000000..2dfe23add --- /dev/null +++ b/cmd/mxcli/verify/render.go @@ -0,0 +1,81 @@ +// SPDX-License-Identifier: Apache-2.0 + +package verify + +import ( + "encoding/json" + "fmt" + "io" + "strings" +) + +// WriteJSON writes the result as one JSON document. +func WriteJSON(w io.Writer, r Result) error { + enc := json.NewEncoder(w) + enc.SetIndent("", " ") + return enc.Encode(r) +} + +// WriteText writes a short human summary: one line per stage, then the new +// findings under it. +func WriteText(w io.Writer, r Result) { + fmt.Fprintf(w, "verify %s\n", r.Project) + for _, s := range r.Stages { + fmt.Fprintf(w, " %-8s %-8s %s\n", s.Name, strings.ToUpper(s.Status), summary(s)) + for _, f := range s.New { + fmt.Fprintf(w, " %s\n", line(f)) + } + for _, f := range s.Unmasked { + fmt.Fprintf(w, " unmasked: %s\n", line(f)) + } + } + switch { + case r.Exit == ExitOK: + fmt.Fprintln(w, "Result: passed") + case r.Exit == ExitCouldNotRun: + fmt.Fprintf(w, "Result: could not run (exit %d)\n", r.Exit) + default: + fmt.Fprintf(w, "Result: FAILED (exit %d)\n", r.Exit) + } +} + +func summary(s StageResult) string { + var parts []string + switch s.Status { + case StatusSkipped, StatusError: + return s.Note + } + parts = append(parts, fmt.Sprintf("%d new", len(s.New))) + if len(s.Unmasked) > 0 { + parts = append(parts, fmt.Sprintf("%d unmasked (an earlier error in the baseline hid them)", len(s.Unmasked))) + } + parts = append(parts, fmt.Sprintf("%d pre-existing", s.PreExisting)) + if s.Resolved > 0 { + parts = append(parts, fmt.Sprintf("%d resolved", s.Resolved)) + } + if s.Hidden > 0 { + parts = append(parts, fmt.Sprintf("%d hidden by a new earlier-stage error, not resolved", s.Hidden)) + } + out := strings.Join(parts, ", ") + if s.Recorded { + out += " (baseline recorded)" + } + if s.Note != "" { + out += " — " + s.Note + } + return out +} + +func line(f Finding) string { + loc := f.Document + if f.Element != "" { + if loc != "" { + loc += " / " + } + loc += f.Element + } + if loc != "" { + loc += ": " + } + return fmt.Sprintf("[%s] %s %s%s", f.Severity, f.Code, loc, f.Message) +} diff --git a/cmd/mxcli/verify/verify.go b/cmd/mxcli/verify/verify.go new file mode 100644 index 000000000..07ea0faea --- /dev/null +++ b/cmd/mxcli/verify/verify.go @@ -0,0 +1,247 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Package verify is the gate behind `mxcli verify` (docs/11-proposals/ +// PROPOSAL_verify_gate.md): it runs the check, lint and mx check stages in cost +// order and reports what a change made worse, not what the project already had. +// +// The stages themselves are injected (see Stage), so the orchestration, the +// baseline comparison and the exit codes are testable without a project or mx. +package verify + +import ( + "fmt" + "sort" + "strings" + "time" +) + +// Stage names, in cost order. +const ( + StageCheck = "check" + StageLint = "lint" + StageMxCheck = "mxcheck" +) + +// AllStages is the cost order the stages run in. +var AllStages = []string{StageCheck, StageLint, StageMxCheck} + +// Statuses of a stage. +const ( + StatusPassed = "passed" + StatusFailed = "failed" // the stage ran and the change made something worse + StatusError = "error" // the stage could not run (no mx, no runtime): never "passed" + StatusSkipped = "skipped" // not selected, nothing to check, or an earlier stage failed with --fail-fast +) + +// Exit codes. A failing gate sets ExitFailedBase plus one bit per failed stage, so +// several failed stages are one code and "failed" can never be read as "could not +// run" (ExitCouldNotRun) or as a usage error. +const ( + ExitOK = 0 + ExitUsage = 1 + ExitCouldNotRun = 2 + ExitFailedBase = 16 +) + +var stageBit = map[string]int{StageCheck: 1, StageLint: 2, StageMxCheck: 4} + +// Finding is one problem a stage reports. Key is the identity a baseline matches +// on; it must not contain the message or any name that renaming changes. +type Finding struct { + Stage string `json:"stage"` + Code string `json:"code"` + Severity string `json:"severity"` // "error", "warning", "info" + Document string `json:"document,omitempty"` + Element string `json:"element,omitempty"` + Message string `json:"message"` + Suggestion string `json:"suggestion,omitempty"` + Key string `json:"key"` + // MxStage is MxBuild's report stage (0..3) for an mxcheck finding, else -1. + MxStage int `json:"mxStage"` + // Unit scopes what an mxcheck finding can hide: stages 1 and 2 hide the later + // ones in the same unit (microflow). + Unit string `json:"unit,omitempty"` +} + +// StageResult is one stage's outcome. +type StageResult struct { + Name string `json:"name"` + Status string `json:"status"` + DurationMs int64 `json:"durationMs"` + // Note says why a stage was skipped or could not run, or that it had no + // baseline to compare with. + Note string `json:"note,omitempty"` + // New are the findings the change made worse, the only ones that fail the gate. + New []Finding `json:"new"` + // Unmasked are findings that appeared only because the baseline had an + // earlier-stage error hiding them (mxcheck). They are reported and fail the + // gate only with Options.Strict. + Unmasked []Finding `json:"unmasked,omitempty"` + // Resolved and Hidden are counts: the stages' baselines record keys, not + // whole findings, and a count is the honest type for lint. + Resolved int `json:"resolved"` + Hidden int `json:"hidden,omitempty"` + PreExisting int `json:"preExisting"` + // Recorded is true when --update-baseline recorded this stage's baseline. + Recorded bool `json:"recorded,omitempty"` +} + +// Result is the whole run. +type Result struct { + Project string `json:"project"` + Stages []StageResult `json:"stages"` + Exit int `json:"exit"` +} + +// Outcome is what a Stage returns, before the orchestrator decides pass or fail. +type Outcome struct { + // New are the findings not in the baseline (all of them without one). + New []Finding + Unmasked []Finding + Resolved int + Hidden int + PreExisting int + Recorded bool + Note string +} + +// Stage is one runnable stage. +type Stage func() (Outcome, error) + +// Options steer the orchestrator. +type Options struct { + Project string + Stages []string // selected, in any order; run in AllStages order + FailFast bool + // FailOn is the lowest severity of a new finding that fails the gate: + // "error" (default) or "warning". + FailOn string + // Strict makes unmasked findings fail the gate too. + Strict bool + // Skip lists stages that have nothing to do, with the reason (e.g. check + // without scripts); they report skipped, never passed. + Skip map[string]string +} + +// Run executes the selected stages in cost order and returns the result. +// stages maps a stage name to its runner. +func Run(opts Options, stages map[string]Stage) Result { + selected := map[string]bool{} + for _, s := range opts.Stages { + selected[s] = true + } + res := Result{Project: opts.Project} + failed, errored, stop := false, false, false + mask := 0 + for _, name := range AllStages { + sr := StageResult{Name: name, New: []Finding{}} + switch { + case !selected[name]: + sr.Status, sr.Note = StatusSkipped, "not selected" + case opts.Skip[name] != "": + sr.Status, sr.Note = StatusSkipped, opts.Skip[name] + case stop: + sr.Status, sr.Note = StatusSkipped, "an earlier stage failed (--fail-fast)" + default: + t0 := time.Now() + out, err := stages[name]() + sr.DurationMs = time.Since(t0).Milliseconds() + if err != nil { + sr.Status, sr.Note = StatusError, err.Error() + errored = true + stop = opts.FailFast + break + } + sr.New = append(sr.New, out.New...) + sr.Unmasked, sr.Resolved, sr.Hidden = out.Unmasked, out.Resolved, out.Hidden + sr.PreExisting, sr.Recorded, sr.Note = out.PreExisting, out.Recorded, out.Note + if fails(sr, opts) { + sr.Status = StatusFailed + failed = true + mask |= stageBit[name] + stop = opts.FailFast + } else { + sr.Status = StatusPassed + } + } + res.Stages = append(res.Stages, sr) + } + switch { + case errored: + res.Exit = ExitCouldNotRun + case failed: + res.Exit = ExitFailedBase | mask + default: + res.Exit = ExitOK + } + // A stage that failed while another could not run is still a failure the + // caller must see; the failure code wins because it is actionable. + if errored && failed { + res.Exit = ExitFailedBase | mask + } + return res +} + +func fails(sr StageResult, opts Options) bool { + for _, f := range sr.New { + if severityRank(f.Severity) >= severityRank(opts.failOn()) { + return true + } + } + return opts.Strict && len(sr.Unmasked) > 0 +} + +func (o Options) failOn() string { + if o.FailOn == "warning" { + return "warning" + } + return "error" +} + +func severityRank(s string) int { + switch s { + case "error": + return 2 + case "warning": + return 1 + } + return 0 +} + +// ParseStages validates a --stages list. +func ParseStages(list []string) ([]string, error) { + var out []string + seen := map[string]bool{} + for _, s := range list { + s = strings.ToLower(strings.TrimSpace(s)) + if s == "" || seen[s] { + continue + } + if _, ok := stageBit[s]; !ok { + return nil, fmt.Errorf("unknown stage %q (stages: %s)", s, strings.Join(AllStages, ", ")) + } + seen[s] = true + out = append(out, s) + } + if len(out) == 0 { + return nil, fmt.Errorf("no stages selected") + } + return out, nil +} + +// SortFindings orders findings deterministically (document, code, element, key). +func SortFindings(fs []Finding) { + sort.SliceStable(fs, func(i, j int) bool { + a, b := fs[i], fs[j] + if a.Document != b.Document { + return a.Document < b.Document + } + if a.Code != b.Code { + return a.Code < b.Code + } + if a.Element != b.Element { + return a.Element < b.Element + } + return a.Key < b.Key + }) +} diff --git a/cmd/mxcli/verify/verify_test.go b/cmd/mxcli/verify/verify_test.go new file mode 100644 index 000000000..45d38b2f8 --- /dev/null +++ b/cmd/mxcli/verify/verify_test.go @@ -0,0 +1,221 @@ +// SPDX-License-Identifier: Apache-2.0 + +package verify + +import ( + "errors" + "path/filepath" + "testing" +) + +func f(stage, key, sev string) Finding { + return Finding{Stage: stage, Code: "X", Severity: sev, Key: key, MxStage: -1} +} + +func mx(key string, mxStage int, unit string) Finding { + return Finding{Stage: StageMxCheck, Code: "CE", Severity: "error", Key: key, MxStage: mxStage, Unit: unit} +} + +func ok(o Outcome) Stage { return func() (Outcome, error) { return o, nil } } + +func statuses(r Result) map[string]string { + m := map[string]string{} + for _, s := range r.Stages { + m[s.Name] = s.Status + } + return m +} + +func TestRunPassesWhenNothingIsNew(t *testing.T) { + r := Run(Options{Stages: AllStages}, map[string]Stage{ + StageCheck: ok(Outcome{}), StageLint: ok(Outcome{PreExisting: 400}), StageMxCheck: ok(Outcome{PreExisting: 3}), + }) + if r.Exit != ExitOK { + t.Errorf("exit = %d, want 0", r.Exit) + } + if r.Stages[1].PreExisting != 400 { + t.Errorf("lint pre-existing = %d", r.Stages[1].PreExisting) + } +} + +// The control for the pass above: the same run with one new error must fail. +func TestRunFailsOnANewError(t *testing.T) { + r := Run(Options{Stages: AllStages}, map[string]Stage{ + StageCheck: ok(Outcome{}), StageLint: ok(Outcome{New: []Finding{f(StageLint, "k", "error")}}), StageMxCheck: ok(Outcome{}), + }) + if r.Exit != ExitFailedBase|stageBit[StageLint] { + t.Errorf("exit = %d, want %d", r.Exit, ExitFailedBase|stageBit[StageLint]) + } + if statuses(r)[StageLint] != StatusFailed { + t.Errorf("lint status = %s", statuses(r)[StageLint]) + } +} + +// By default every selected stage runs, so one result carries every stage's +// findings and the exit code is the bitmask of all that failed. +func TestRunAllStagesByDefault(t *testing.T) { + bad := ok(Outcome{New: []Finding{f("", "k", "error")}}) + r := Run(Options{Stages: AllStages}, map[string]Stage{StageCheck: bad, StageLint: bad, StageMxCheck: bad}) + want := ExitFailedBase | 1 | 2 | 4 + if r.Exit != want { + t.Errorf("exit = %d, want %d", r.Exit, want) + } +} + +func TestRunFailFastSkipsLaterStages(t *testing.T) { + ran := map[string]bool{} + mk := func(name string, o Outcome) Stage { + return func() (Outcome, error) { ran[name] = true; return o, nil } + } + r := Run(Options{Stages: AllStages, FailFast: true}, map[string]Stage{ + StageCheck: mk(StageCheck, Outcome{New: []Finding{f("", "k", "error")}}), + StageLint: mk(StageLint, Outcome{}), StageMxCheck: mk(StageMxCheck, Outcome{}), + }) + if ran[StageLint] || ran[StageMxCheck] { + t.Errorf("stages ran after a failure with --fail-fast: %v", ran) + } + if statuses(r)[StageLint] != StatusSkipped { + t.Errorf("lint status = %s, want skipped", statuses(r)[StageLint]) + } + if r.Exit != ExitFailedBase|1 { + t.Errorf("exit = %d", r.Exit) + } +} + +// A stage that cannot run is never "passed", and its exit code is not a failure's. +func TestRunStageThatCannotRunIsAnError(t *testing.T) { + r := Run(Options{Stages: AllStages}, map[string]Stage{ + StageCheck: ok(Outcome{}), StageLint: ok(Outcome{}), + StageMxCheck: func() (Outcome, error) { return Outcome{}, errors.New("mx not found") }, + }) + if statuses(r)[StageMxCheck] != StatusError || r.Exit != ExitCouldNotRun { + t.Errorf("status = %s, exit = %d", statuses(r)[StageMxCheck], r.Exit) + } + if r.Exit == ExitOK { + t.Error("a gate that could not run must not exit 0") + } +} + +func TestRunSkipsUnselectedAndEmptyStages(t *testing.T) { + r := Run(Options{Stages: []string{StageLint}, Skip: map[string]string{StageCheck: "no scripts given"}}, + map[string]Stage{StageLint: ok(Outcome{})}) + st := statuses(r) + if st[StageLint] != StatusPassed || st[StageCheck] != StatusSkipped || st[StageMxCheck] != StatusSkipped { + t.Errorf("statuses = %v", st) + } +} + +func TestFailOnWarning(t *testing.T) { + w := ok(Outcome{New: []Finding{f(StageLint, "k", "warning")}}) + stages := map[string]Stage{StageCheck: ok(Outcome{}), StageLint: w, StageMxCheck: ok(Outcome{})} + if r := Run(Options{Stages: AllStages}, stages); r.Exit != ExitOK { + t.Errorf("a new warning failed the gate by default (exit %d)", r.Exit) + } + if r := Run(Options{Stages: AllStages, FailOn: "warning"}, stages); r.Exit == ExitOK { + t.Error("--fail-on warning did not fail on a new warning") + } +} + +func TestStrictFailsOnUnmasked(t *testing.T) { + u := ok(Outcome{Unmasked: []Finding{mx("k", 3, "u1")}}) + stages := map[string]Stage{StageCheck: ok(Outcome{}), StageLint: ok(Outcome{}), StageMxCheck: u} + if r := Run(Options{Stages: AllStages}, stages); r.Exit != ExitOK { + t.Errorf("unmasked failed the gate without --strict (exit %d)", r.Exit) + } + if r := Run(Options{Stages: AllStages, Strict: true}, stages); r.Exit != ExitFailedBase|4 { + t.Errorf("--strict exit = %d", r.Exit) + } +} + +func TestParseStages(t *testing.T) { + if got, err := ParseStages([]string{"lint", " mxcheck", "lint"}); err != nil || len(got) != 2 { + t.Errorf("got %v, %v", got, err) + } + if _, err := ParseStages([]string{"smoke"}); err == nil { + t.Error("an unknown stage must be refused") + } +} + +// --- classification ------------------------------------------------------- + +func TestClassifyPlain(t *testing.T) { + base := []Finding{f(StageLint, "a", "error"), f(StageLint, "a", "error"), f(StageLint, "gone", "error")} + cur := []Finding{f(StageLint, "a", "error"), f(StageLint, "a", "error"), f(StageLint, "a", "error"), f(StageLint, "b", "error")} + o := Classify(base, cur, false) + if len(o.New) != 2 || o.PreExisting != 2 || o.Resolved != 1 { + t.Errorf("new=%d pre=%d resolved=%d, want 2/2/1 (a third 'a' is new, multiplicity counts)", len(o.New), o.PreExisting, o.Resolved) + } +} + +// Fixing a stage 0 error (an unresolved reference) uncovers every later-stage +// error in the project, none of them the change's: unmasked, not new. +func TestClassifyUnmaskedByStage0(t *testing.T) { + base := []Finding{mx("ref", 0, "flow9")} + cur := []Finding{mx("type", 3, "flow1"), mx("undef", 2, "flow2")} + o := Classify(base, cur, true) + if len(o.Unmasked) != 2 || len(o.New) != 0 || o.Resolved != 1 { + t.Errorf("unmasked=%d new=%d resolved=%d, want 2/0/1", len(o.Unmasked), len(o.New), o.Resolved) + } +} + +// An earlier-stage error of the same microflow can only vanish if the change +// edited that microflow, so a later error uncovered there is new: the change +// must leave the flow clean. (Seen on real mx output: replacing a duplicate +// variable with a type error passed as "unmasked" before this rule.) +func TestClassifySameUnitUncoveredErrorIsNew(t *testing.T) { + base := []Finding{mx("dup", 1, "flow1")} + cur := []Finding{mx("type", 3, "flow1")} + o := Classify(base, cur, true) + if len(o.New) != 1 || len(o.Unmasked) != 0 || o.Resolved != 1 { + t.Errorf("new=%d unmasked=%d resolved=%d, want 1/0/1", len(o.New), len(o.Unmasked), o.Resolved) + } +} + +// The control for unmasked: with no stage 0 error in the baseline, a new +// later-stage error in another flow is new too. +func TestClassifyNewWhenNothingHidIt(t *testing.T) { + base := []Finding{mx("dup", 1, "flow1")} + cur := []Finding{mx("dup", 1, "flow1"), mx("type", 3, "flow2")} + o := Classify(base, cur, true) + if len(o.New) != 1 || len(o.Unmasked) != 0 { + t.Errorf("new=%d unmasked=%d, want 1/0: flow2 was not hidden by a flow1 error", len(o.New), len(o.Unmasked)) + } + // Without staging the stage 0 case is new as well, so the rule is what makes it unmasked. + if o := Classify([]Finding{mx("ref", 0, "f")}, []Finding{mx("type", 3, "g")}, false); len(o.New) != 1 { + t.Errorf("without staging a plain diff calls it new, got %d", len(o.New)) + } +} + +// A change that introduces a stage 0 error hides everything else: the other +// findings are hidden, not resolved. +func TestClassifyHiddenNotResolved(t *testing.T) { + base := []Finding{mx("a", 3, "f1"), mx("b", 2, "f2")} + cur := []Finding{mx("ref", 0, "f3")} + o := Classify(base, cur, true) + if o.Hidden != 2 || o.Resolved != 0 || len(o.New) != 1 { + t.Errorf("hidden=%d resolved=%d new=%d, want 2/0/1", o.Hidden, o.Resolved, len(o.New)) + } + if o := Classify(base, nil, true); o.Resolved != 2 || o.Hidden != 0 { + t.Errorf("with nothing hiding them they are resolved: resolved=%d hidden=%d", o.Resolved, o.Hidden) + } +} + +func TestMxBaselineRoundTripAndCompat(t *testing.T) { + p := filepath.Join(t.TempDir(), "sub", "mxcheck.json") + if b, err := LoadMxBaseline(p); b != nil || err != nil { + t.Fatalf("a missing baseline must be (nil, nil), got %v, %v", b, err) + } + if err := WriteMxBaseline(p, "mx check", "11.16.1", []Finding{mx("b", 3, "u"), mx("a", 2, "u")}); err != nil { + t.Fatal(err) + } + b, err := LoadMxBaseline(p) + if err != nil || len(b.Findings) != 2 || b.Findings[0].Key != "a" { + t.Fatalf("round trip = %+v, %v", b, err) + } + if err := b.CheckCompatible("mx check", "11.16.1"); err != nil { + t.Errorf("same checker refused: %v", err) + } + if b.CheckCompatible("mendix check", "11.16.1") == nil || b.CheckCompatible("mx check", "11.12.1") == nil { + t.Error("another checker or mx version must be refused") + } +} diff --git a/docs/11-proposals/PROPOSAL_verify_gate.md b/docs/11-proposals/PROPOSAL_verify_gate.md index 16286433a..23d776845 100644 --- a/docs/11-proposals/PROPOSAL_verify_gate.md +++ b/docs/11-proposals/PROPOSAL_verify_gate.md @@ -68,17 +68,21 @@ imply a clean result means "correct". ## Proposed behaviour ``` -mxcli verify -p app.mpr [--base | --baseline ] +mxcli verify -p app.mpr [script.mdl...] [--base ] [--stages check,lint,mxcheck,build,test,smoke] - [--fail-fast] [--format text|json|sarif] - [--update-baseline] + [--fail-fast] [--fail-on error|warning] [--strict] + [--baseline-dir ] [--update-baseline] + [--format text|json] ``` +**First release (step 2, implemented):** stages `check`, `lint`, `mxcheck`; +`--base`, `build`, `test`, `smoke` and `--format sarif` are later steps. + | Stage | Reuses | Default | Cost | |-------|--------|---------|------| -| `check` | project-wide reference check (`check -p … --references`) | on | ~0.2 s | -| `lint` | `linter` + `Baseline` | on | ~0.2–0.9 s | -| `mxcheck` | `docker.Check` | on | ~8 s | +| `check` | `mxcli check -p` on the **script files given as arguments**; skipped (and reported skipped) without any. `check` has no project-wide mode | on with scripts | ~0.2 s | +| `lint` | `mxcli lint --format json` against its own baseline file | on | ~0.2–0.9 s | +| `mxcheck` | `docker.CheckFindings` (`mx check --json` on a copy) | on | ~8 s | | `build` | `run --local` build/boot | opt-in | 30–60 s | | `test` | `test --local` | opt-in | ~24 s | | `smoke` | `playwright check --role X` | opt-in | ~24 s | @@ -92,20 +96,22 @@ mxcli verify -p app.mpr [--base | --baseline ] Opt-in stages (`build`, `test`, `smoke`) are still skipped unless selected, and are skipped when an earlier stage failed, since they need a model that builds. - Findings are classified **new / resolved / pre-existing** against the baseline (`mxcheck` adds **unmasked** and **hidden**, see the staging section), - keyed by `(code, document, element path)` — never by message text or line - number, which drift between runs (the problem #1313 describes). -- **Exit code:** 0 when nothing is new; 2 when the gate could not run (no `mx`, - no runtime for a selected stage), so "gate failed" and "gate could not run" are - never confused. When stages fail, the code is a **bitmask** (1 check, 2 lint, - 4 mxcheck, 8 build, 16 test, 32 smoke) offset so it cannot collide with 0 or 2 - — exact encoding to be fixed in step 2 — because with run-all as the default - several stages can fail at once. The per-stage detail is always in the output; - the code is only a summary. **Relation to existing codes:** `check` exits with a - variable code (`cmd_check.go`) and `lint` exits 1. `verify`'s table is its own - contract and must say so in `--help`; an agent running `check` then `verify` - will see different codes for the same finding. -- **Output:** `--format text|json|sarif`, matching `check` and `lint` (the global - `--json` is the existing alias via `resolveFormat`), not a new flag. JSON is one + keyed by IDs for `mxcheck` (see Finding identity) and by lint's own key for + `lint` — never by message text or line number, which drift between runs (the + problem #1313 describes). A stage without a baseline reports every finding as + new and says so (`note`), so a first run is honest, not silently green. +- **Exit code** (a contract, printed in `--help`): `0` nothing new; `1` usage + error; `2` a stage could not run (no `mx`, ...) and nothing failed; **`16` plus + one bit per failed stage** (1 `check`, 2 `lint`, 4 `mxcheck`) when a stage + failed, so `18` is lint, `22` is lint and mxcheck, and a failure can never be + read as "could not run" or as a usage error. When one stage failed and another + could not run, the failure code wins (it is the actionable one) and the + stage's `status: error` is in the output. `check` exits with a variable code + and `lint` exits 1, so an agent running them directly sees different codes for + the same finding; `verify`'s table is its own. `--fail-on warning` makes a new + warning fail a stage too (default: errors, as `lint` does). +- **Output:** `--format text|json` (the global `--json` is the existing alias via + `resolveFormat`); `sarif`, which `check` and `lint` also offer, is a later step. JSON is one object `{stages:[{name,status,durationMs,new:[...],resolved:, preExisting:int}], exit}`, so an agent reads one result instead of six formats. `resolved` is typed honestly per stage: a list where the key identifies the @@ -127,8 +133,8 @@ mxcli verify -p app.mpr [--base | --baseline ] ### Baseline sources -1. `--baseline ` — saved findings (cheap; ships first). `--update-baseline` - records the current state. +1. `--baseline-dir ` — saved findings, one file per stage (cheap; shipped + first). `--update-baseline` records the current state. 2. `--base ` — run the same stages on a scratch export of that ref, then compare. Roughly doubles `mxcheck` cost; results are cacheable per commit. @@ -199,18 +205,22 @@ So `verify` classifies the `mxcheck` stage's findings into four states, not thre | state | meaning | |---|---| -| **new** | in the report, not in the baseline, and the baseline was not hiding it | -| **unmasked** | new in the report, but the baseline had an earlier-stage finding in a scope that hides it (same document for stages 1 and 2, anywhere for stage 0). It may be old; the report cannot tell | +| **new** | in the report and not in the baseline. This includes a later-stage error uncovered in a microflow whose earlier-stage baseline error is gone (see below) | +| **unmasked** | new in the report, but a **stage 0** error in the baseline (an unresolved reference) was hiding the whole project. Fixing that one error uncovers everything else, and none of it is the change's | | **resolved** | in the baseline, absent from the report, and nothing in the report hides it | | **hidden** | in the baseline, absent from the report, and the report has a finding that hides it. Not resolved | Rules that follow: -- `unmasked` is reported with the finding that hid it, and **does not fail the gate - by default** (it is most likely pre-existing). `--strict` makes it fail, for - teams that would rather pay a second pass. The text output states how many - findings the staging may still be holding back. -- A new stage 0 or earlier-stage finding always fails the gate, and the `hidden` +- **Scope decides.** A stage 1 or 2 error hides only the later stages of its own + microflow, and it can only disappear if the change edited that microflow, so a + later error uncovered there is the change's to fix: **new**, and it fails. Only + a stage 0 error hides other documents, so only what it hid is **unmasked**; it + does not fail the gate by default (it is almost certainly old) and `--strict` + makes it fail. The first draft made every uncovered error unmasked; a real run + (a flow whose duplicate variable was replaced by a type error) passed the gate + with an error the change had just written, which is what this rule fixes. +- A new stage 0 finding always fails the gate, and the `hidden` count is printed next to it so a reader does not read "400 resolved". - Each finding carries its **stage** (`StageOf`, from the measured table). A code not in the measured set is treated as stage 3 (hides nothing, hidden by earlier stages): the conservative reading for `unmasked`, and @@ -315,17 +325,27 @@ type Stage interface { earlier text parser was replaced because the text does not identify the document (see Finding identity). Remaining: a CE0463 golden file, and wiring the `mxcheck` stage to it in step 2. -2. **`mxcli verify`, first release**: stages `check`, `lint`, `mxcheck`; - `--format`, exit-code table, `--fail-fast`; and a **`lint` baseline from day - one** (`--baseline`, `--update-baseline`, reusing `mdl/linter/baseline.go` - through the adapter described above), so the first run on an existing project - reports only what is new. The `mxcheck` stage reads `mx check --json` (step 1, done) and baselines with the - four-state classification above. +2. **`mxcli verify`, first release** (implemented): stages `check`, `lint`, + `mxcheck`; `--format text|json`, the exit-code table, `--fail-fast`, + `--fail-on`, `--strict`; and a **`lint` baseline from day one**. Baselines live + in `--baseline-dir` (default `.mxcli/verify` next to the project): `lint.json` + is a plain `mxcli lint --baseline` file (so existing ones stay valid) and + `mxcheck.json` records mx check's findings by element ID, with the checker and + mx version, which a run must match. `lint` and `check` run as subprocesses of + the same binary, so they behave exactly as run by hand and `lint` is not + refactored. Checked on real MxBuild 11.16.1: a new defect fails (exit 20), an + accepted one does not, a stage 0 fix is unmasked, a same-flow uncovered error + is new. 3. **Make `verify` the thing everything points at**: skills, `mxcli init` output and the project `CLAUDE.md` guidance name `verify` as the single "did my change - break anything?" command, instead of listing six commands. Ships with step 2, - not after it. A guard test (in the style of `init_claudemd_gates_test.go`) - keeps the pointer from drifting. + break anything?" command. **Not in the first release, and a decision is + needed:** `projectGates` is the definition of done in every generated + `CLAUDE.md`. Adding `verify` as a seventh gate runs lint and `mx check` twice + and puts the file 77 bytes over its test-enforced budget; the alternative is + for `verify` to *replace* the `lint` and `docker check` gates, which changes + every project's default and needs `bootstrap-app` to record the baselines at + provisioning. Either way the gate tests (`init_claudemd_gates_test.go`) already + keep the three descriptions in step. 4. **`--base `** via scratch export. 5. **Opt-in `build`, `test`, `smoke` stages.** `smoke` is also the only stage that can reach the runtime-only findings listed under Evidence. @@ -334,10 +354,11 @@ type Stage interface { - Unit: finding keys are IDs, not text (same finding, different message or name → same key; same text, different element → different key); classification new/resolved/pre-existing; default run-all vs `--fail-fast`. -- Staging: for each stage pair from `maskmatrix`, a fixture diff classified - unmasked (not new) when the baseline hid it and hidden (not resolved) when the - change introduces the hiding finding; **control:** the same finding with no - hiding baseline finding must be new and must fail the gate. +- Staging: a stage 0 baseline error fixed → what it hid is unmasked, not new; + a same-microflow earlier error replaced by a later one → new (the case a real + run showed); a new stage 0 error → the baseline's findings are hidden, not + resolved; **control:** the same finding with no hiding baseline finding is new + and fails the gate. - Checker identity: a baseline from one checker compared against the other is an error. - Reader: golden `mx check --json` files from real runs (committed under `cmd/mxcli/docker/testdata/mxcheck/`), the same defect in two microflows (two keys), a clean control, a lost-finding control for `Verify`, and an integration test against a real `mx`. @@ -350,24 +371,25 @@ type Stage interface { ## Open questions -1. **Name.** `mxcli verify` alongside `mxcli playwright verify`: acceptable, or - prefer `gate`/`validate`? (Much milder than first drafted: one existing - `verify`, under a namespace.) -2. **One baseline file or one per stage?** One file is simpler for agents; - per-stage keeps `lint --baseline` files valid unchanged. +1. **Name.** `mxcli verify` alongside `mxcli playwright verify`: implemented as + `verify`; rename if the maintainers prefer `gate`/`validate`. +2. ~~One baseline file or one per stage?~~ **Settled in step 2:** one directory, + one file per stage, so `lint --baseline` files stay valid. 3. **ID stability.** Are `unit-id` / `element-id` stable across a Studio Pro - module update and an `mxcli` rewrite, and does a CE0463's JSON carry them? + module update and an `mxcli` rewrite (a `create or modify` kept the unit ID in + the real run), and does a CE0463's JSON carry them? 4. **`--base` for MPR v1** — extract via `git show` of the single `.mpr`, or require v2? -5. **Should `check` stage fail on warnings,** or only errors? -6. **Exit-code encoding** — bitmask offsets, and whether to align with `check`'s - and `lint`'s existing codes. -7. **Unmasked policy.** Non-failing by default with `--strict`, or the other way - round? Failing wastes a pass on findings that are usually old; not failing can - let a change that introduced a hidden error through. +5. ~~Should `check` stage fail on warnings?~~ **Settled:** errors by default, + `--fail-on warning` to include warnings. +6. ~~Exit-code encoding.~~ **Settled:** 0, 1, 2, and 16 plus a bit per failed stage. +7. ~~Unmasked policy.~~ **Settled by scope:** only what a stage 0 error hid is + unmasked (non-failing, `--strict` fails); an error uncovered in a microflow the + change edited is new. 8. **Staging coverage.** The measurement covers six codes in microflows and pages. How far must `maskmatrix` be extended before `unmasked` is trusted beyond them? 9. **`mendix check` pre-stage:** worth a flag now, or defer until its false-positive rate is measured on real projects? -10. **Default stage set** — with run-all as the default, `mxcheck` costs ~8 s once - per iteration; is on-by-default accepted? +10. **Default stage set** — `mxcheck` costs ~8 s once per run and is on by default. +11. **Step 3:** add `verify` as a seventh gate, or replace `lint` and `docker check` + with it (see Phasing)? From 172c57d64654d8a0b95ad25432a7e2ae34211775 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 19 Nov 2026 09:36:59 +0000 Subject: [PATCH 03/66] fix(test): the list-marker census skips its optional submodule instead of failing Closes ako/mxcli#2413. On a clone without submodules, `make ci-local` could not pass: --- FAIL: TestIssue1640_RegisteredListMarkersAreWhatStudioProStores lstat ../../../testdata/testapp/TestApp/mprcontents: no such file or directory 125 other packages passed. testdata/testapp is a git SUBMODULE (ako/TestApp, a gitlink pinned at 11a8fcac) that .github/workflows/push-test.yml checks out with `submodules: true`, so CI always has it and `git clone` without --recurse-submodules does not. studioProUnits walked both roots unconditionally and fed WalkDir's lstat error into t.Fatal. That matters more than one red test. .claude/skills/steward/SKILL.md makes `make ci-local` the gate to run before every push, so on a fresh clone the gate could never pass, and a pre-existing failure was indistinguishable from one the current change had just caused -- which cost a session real time today. mdl/roundtrip already skipped this same fixture with a remedy message (fixture.skipIfMissing); the two now agree, and reuse its wording. The skip is NOT unconditional. Under CI the test FAILS instead, because CI does check the submodule out, so an absence there is a workflow regression rather than a developer's clone. ako/mxcli#808 shipped green because an integration test had only ever skipped, and this census is the only thing holding canon's list-marker registrations against Studio Pro-authored documents: a skip that could fire everywhere would retire that check silently. The units floor had to move too, or skipping would have tripped it instead of the lstat: it was one total of 1000 over both roots, and the submodule holds 900 against pedapp's 369. It is now a floor per root (800 and 300), so the committed fixture alone still proves a real corpus was read. Measured in all three states - submodule present: runs and passes (4.1s). - absent, local: skips that root with the `git submodule update --init testdata/testapp` remedy, still censuses pedapp's 369 units, passes. - absent, CI=true: fails, naming the root and that CI checks submodules out. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011oK82sY9fcHtMuZcMNRm2B --- ...submodule-fixture-hard-fails-ci-local.json | 1 + CHANGELOG.md | 5 ++ .../issue1640_list_marker_census_test.go | 73 +++++++++++++++++-- 3 files changed, 72 insertions(+), 7 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-11-19-optional-submodule-fixture-hard-fails-ci-local.json diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-11-19-optional-submodule-fixture-hard-fails-ci-local.json b/.claude/skills/fix-issue/findings/modelsdk/2026-11-19-optional-submodule-fixture-hard-fails-ci-local.json new file mode 100644 index 000000000..784869bd2 --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-11-19-optional-submodule-fixture-hard-fails-ci-local.json @@ -0,0 +1 @@ +{"area": "mdl/backend/modelsdk", "date": "2026-11-19", "symptom": "`make ci-local` could not pass on a clone without submodules: TestIssue1640_RegisteredListMarkersAreWhatStudioProStores failed with `lstat ../../../testdata/testapp/TestApp/mprcontents: no such file or directory` while 125 other packages passed. The same test passes in CI, so the failure looks like something the current change broke.", "cause": "testdata/testapp is a git SUBMODULE (ako/TestApp, gitlink pinned at 11a8fcac) that .github/workflows/push-test.yml checks out with `submodules: true`, so CI always has it and `git clone` without --recurse-submodules does not. studioProUnits walked both fixture roots unconditionally and fed WalkDir's lstat error into t.Fatal; mdl/roundtrip already skipped the same fixture with a remedy message (skipIfMissing), so the two disagreed. A single floor of 1000 units over both roots was the second half: dropping the 900-unit submodule leaves pedapp's 369, so skipping alone would have tripped the floor instead. Fixed with per-root floors, a skip carrying the `git submodule update --init testdata/testapp` remedy, and a t.Fatal when os.Getenv(\"CI\") is set.", "file": "`mdl/backend/modelsdk/issue1640_list_marker_census_test.go` (studioProFixtures, studioProUnits); precedent in `mdl/roundtrip/pedapp_roundtrip_test.go` (fixture.skipIfMissing)", "insight": "A test that hard-fails on an uncommitted fixture makes the project's own pre-push gate unpassable, and the cost is not the failing test — it is that a pre-existing failure becomes indistinguishable from one you just caused. The cheap discriminator is to run the single test in a clean worktree at the base commit; that settles it in one step, where reading the diff for a plausible culprit settles nothing. The fix is NOT a bare t.Skip: ako/mxcli#808 shipped green because an integration test had only ever skipped, and this census is the only thing holding canon's list-marker registrations against Studio Pro-authored documents. Skip locally, FAIL under CI (os.Getenv(\"CI\"), as cmd/mxcli/gitstate.go already does), so the skip can never fire where coverage matters. And check the assertions downstream of the data you just made optional: the units floor would have failed the test anyway, turning one confusing failure into another."} diff --git a/CHANGELOG.md b/CHANGELOG.md index a4aae4011..cfc728140 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **`make ci-local` passes on a clone without submodules** (ako/mxcli#2413) — `TestIssue1640_RegisteredListMarkersAreWhatStudioProStores` walked `testdata/testapp`, a git submodule (`ako/TestApp`) that CI checks out with `submodules: true` and a plain `git clone` does not, and fed the resulting `lstat` error into `t.Fatal`. So the gate `.claude/skills/steward/SKILL.md` tells every contributor to run before each push could never pass on a fresh clone, and a pre-existing failure was indistinguishable from one the current change had just caused. `mdl/roundtrip` already skipped the same fixture with a remedy message; the two now agree. + + The skip is deliberately not unconditional: under CI the test **fails** instead, because CI does check the submodule out, and a skip that could fire everywhere is how an integration test came to have only ever skipped (ako/mxcli#808) — this census is the only thing holding `canon`'s list-marker registrations against Studio Pro-authored documents. The units floor moved from one total of 1000 to a floor per root (800 and 300), since dropping the 900-unit submodule leaves pedapp's 369 and would have tripped the old floor instead. Verified in all three states: present (runs), absent (skips, still censuses pedapp), absent with `CI=true` (fails). + + - **A freshly built catalog is one database again, so an iterator no longer fails while its neighbours work** (ako/mxcli#2417) — on a run that built the catalog rather than loading it, `lint` could report `could not read the catalog for AttributesFor: no such table: attributes` while `Entities` returned rows in the same run on the same handle, and the next identical run was correct. `catalog.New()` opened `:memory:`, where **every connection is its own separate, empty database**, and `sql.Open` hands back a pool — so the schema landed on one connection and a query the pool served on another saw nothing. Which iterator broke was pool timing, which is why it reproduced on one project and not another. The in-memory catalog now uses a uniquely named shared-cache DSN with one pinned connection, so every pooled connection sees the same database. Measured: the first `lint` of a fresh project reports what the second always did, and reverting only the DSN brings the old error back. Two knock-on fixes. A fresh catalog's snapshot `ObjectCount` was counted as **0**, because `updateSnapshotCount` read through the pool while the build transaction was open and so could not see the rows that transaction had just written; it now reads through the transaction. And `foreign_keys` was set with `Exec`, which reaches only the one pooled connection it runs on, so the pragma was off for most queries — on both the in-memory and the file path. It now rides in the DSN, as the busy timeout already did for exactly this reason. diff --git a/mdl/backend/modelsdk/issue1640_list_marker_census_test.go b/mdl/backend/modelsdk/issue1640_list_marker_census_test.go index 5b7958e46..7e461b47a 100644 --- a/mdl/backend/modelsdk/issue1640_list_marker_census_test.go +++ b/mdl/backend/modelsdk/issue1640_list_marker_census_test.go @@ -22,29 +22,88 @@ import ( // Studio Pro-authored fixtures: a document Studio Pro wrote must be a fixed // point of the normalization. +// studioProFixture is one Studio Pro-authored fixture this census reads. +type studioProFixture struct { + root string + // minUnits guards against reading a truncated fixture. Per root, not over + // the total: a single floor over both made the census fail when the + // optional submodule was absent, which is the same all-or-nothing the skip + // below exists to avoid. + minUnits int + // skipIfMissing is the remedy printed when an OPTIONAL fixture is absent. + // Empty for a committed fixture, which must always be present. + skipIfMissing string +} + +// studioProFixtures are the roots the census holds the marker registrations +// against. A document Studio Pro wrote must be a fixed point of the +// normalization, so these have to be Studio Pro-authored, not mxcli-written. +var studioProFixtures = []studioProFixture{ + { + // ako/TestApp, pinned as the git submodule testdata/testapp. Not + // committed, so a plain `git clone` does not have it. + root: "../../../testdata/testapp/TestApp/mprcontents", + minUnits: 800, + skipIfMissing: "the TestApp fixture is a git submodule that is not initialised; " + + "run `git submodule update --init testdata/testapp` to include it in the census", + }, + { + // Committed, so never skipped: it is what keeps this census meaningful + // when the submodule is absent. + root: "../../../testdata/pedapp/mprcontents", + minUnits: 300, + }, +} + // studioProUnits lists the .mxunit files of the Studio Pro-authored fixtures. +// +// An absent OPTIONAL fixture skips that root rather than failing the test +// (ako/mxcli#2413). It used to walk straight into filepath.WalkDir's lstat +// error and t.Fatal, so on any clone without submodules `make ci-local` could +// not pass — and .claude/skills/steward/SKILL.md tells every contributor to run +// it before each push. A pre-existing failure was then indistinguishable from +// one just caused, which cost a real session its time. mdl/roundtrip already +// skips this same fixture, with this same remedy, so the two now agree. +// +// Under CI it FAILS instead of skipping. A skip that can hide the test +// everywhere is how an integration test came to have only ever skipped +// (ako/mxcli#808), and this census is the only thing holding canon's list +// marker registrations against Studio Pro-authored documents. CI checks the +// submodule out (`submodules: true` in .github/workflows/push-test.yml), so an +// absence there is a regression in the workflow, not a developer's clone. func studioProUnits(t *testing.T) []string { t.Helper() var out []string - for _, root := range []string{ - "../../../testdata/testapp/TestApp/mprcontents", - "../../../testdata/pedapp/mprcontents", - } { - err := filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error { + for _, f := range studioProFixtures { + if f.skipIfMissing != "" { + if _, err := os.Stat(f.root); err != nil { + if os.Getenv("CI") != "" { + t.Fatalf("%s is missing under CI, which checks submodules out: %v", f.root, err) + } + t.Logf("skipping %s: %s", f.root, f.skipIfMissing) + continue + } + } + n := 0 + err := filepath.WalkDir(f.root, func(p string, d fs.DirEntry, err error) error { if err != nil { return err } if !d.IsDir() && strings.HasSuffix(p, ".mxunit") { out = append(out, p) + n++ } return nil }) if err != nil { t.Fatal(err) } + if n < f.minUnits { + t.Fatalf("found only %d units under %s (want at least %d)", n, f.root, f.minUnits) + } } - if len(out) < 1000 { - t.Fatalf("found only %d fixture units", len(out)) + if len(out) == 0 { + t.Fatal("no fixture units found at all") } return out } From c8b14b828e8f8dbff499849d06cb51c424f27951 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 19 Nov 2026 09:41:47 +0000 Subject: [PATCH 04/66] fix(report): read the consistency check from mx check --json main switched docker.CheckFindings from parsing printed `mx check` text to the JSON report `mx check -j` writes (8f4bc30a1), so the report's check no longer built. It now: - asks for warnings and deprecations too (CheckKinds -w -d), as #2423 asked; - keeps each finding's module, document and element. The per-code "Where" column lists distinct documents instead of widget names. Measured on testdata/pedapp: most of its 1055 errors are in the Atlas modules' page templates; - reads --mx-check-results as that JSON file, and refuses printed text with a message naming the file it takes. Tests use the recorded real outputs in docker/testdata/mxcheck. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82qTgUsRMETmzvjS27w --- .claude/skills/mendix/report-quality/SKILL.md | 2 +- cmd/mxcli/cmd_report.go | 3 +- cmd/mxcli/cmd_report_consistency.go | 28 ++++-- cmd/mxcli/cmd_report_consistency_test.go | 94 +++++++++++-------- docs-site/src/tools/mxcli-report.md | 11 ++- mdl/linter/report_consistency.go | 29 ++++-- mdl/linter/report_consistency_test.go | 14 +-- 7 files changed, 117 insertions(+), 64 deletions(-) diff --git a/.claude/skills/mendix/report-quality/SKILL.md b/.claude/skills/mendix/report-quality/SKILL.md index f2ec63705..f1a580f51 100644 --- a/.claude/skills/mendix/report-quality/SKILL.md +++ b/.claude/skills/mendix/report-quality/SKILL.md @@ -42,7 +42,7 @@ the project on disk is not changed. ```bash mxcli setup mxbuild -p app.mpr # once: mx for the project's Mendix version -mxcli report -p app.mpr --mx-check-results check.txt # reuse saved 'mxcli docker check' output +mxcli report -p app.mpr --mx-check-results check.json # the file 'mx check .mpr -j check.json -w -d' wrote mxcli report -p app.mpr --no-mx-check # lint only; the report says it did not check ``` diff --git a/cmd/mxcli/cmd_report.go b/cmd/mxcli/cmd_report.go index 209eb52d1..a6bd8ef40 100644 --- a/cmd/mxcli/cmd_report.go +++ b/cmd/mxcli/cmd_report.go @@ -50,7 +50,8 @@ copy of the project and states above the score whether the project builds: a project with consistency errors cannot be deployed, whatever it scores. It needs mx for the project's Mendix version ('mxcli setup mxbuild -p app.mpr'); without it the report says the check was not run, never that the project is clean. ---mx-check-results reads saved output instead, and --no-mx-check skips it. +--mx-check-results reads the JSON report of an earlier 'mx check -j +' instead, and --no-mx-check skips it. --modules scores only the named modules — a project's own modules, without the Marketplace and platform modules beside them. Only findings located in a diff --git a/cmd/mxcli/cmd_report_consistency.go b/cmd/mxcli/cmd_report_consistency.go index b05b7d0bc..6da4377f1 100644 --- a/cmd/mxcli/cmd_report_consistency.go +++ b/cmd/mxcli/cmd_report_consistency.go @@ -17,10 +17,15 @@ import ( // project with consistency errors cannot be deployed and could still score // well. -// reportMxCheck runs `mx check` on a copy of the project. A seam for tests, -// which have no mx. +// reportMxCheck runs `mx check --json` on a copy of the project. A seam for +// tests, which have no mx. var reportMxCheck = docker.CheckFindings +// reportCheckKinds asks mx for warnings and deprecations as well as errors: +// the report lists them, without scoring them. Best-practice findings are left +// to the lint rules. +var reportCheckKinds = docker.CheckKinds{Warnings: true, Deprecations: true} + // reportConsistency runs the check, or reads saved results, or says why it was // not run. Only an unreadable results file the user named is an error: a // missing mx is a report that says so, never a clean one. @@ -33,15 +38,19 @@ func reportConsistency(cmd *cobra.Command, projectPath string, progress io.Write if err != nil { return nil, fmt.Errorf("--mx-check-results: %w", err) } - return consistencyFrom(docker.ParseCheckReport(string(b)), "results file "+path), nil + rep, err := docker.ParseCheckJSON(b) + if err != nil { + return nil, fmt.Errorf("--mx-check-results %s: %w (it takes the file 'mx check -j ' writes)", path, err) + } + return consistencyFrom(rep, "results file "+path), nil } mxbuild, _ := cmd.Flags().GetString("mxbuild-path") fmt.Fprintln(progress, "Running mx check on a temporary copy of the project (--no-mx-check skips it)...") - rep, runErr := reportMxCheck(docker.CheckOptions{ProjectPath: projectPath, MxBuildPath: mxbuild}) + rep, runErr := reportMxCheck(docker.CheckOptions{ProjectPath: projectPath, MxBuildPath: mxbuild}, reportCheckKinds) if runErr != nil && len(rep.Findings) == 0 && !rep.HasDeclared { // mx was not found, or did not get as far as checking. return &linter.ConsistencyCheck{NotRun: fmt.Sprintf("%v. 'mxcli setup mxbuild -p %s' sets up the project's Mendix version; "+ - "--mx-check-results reads the output of an earlier 'mx check' or 'mxcli docker check'.", runErr, projectPath)}, nil + "--mx-check-results reads the JSON report of an earlier 'mx check -j '.", runErr, projectPath)}, nil } return consistencyFrom(rep, "mx check"), nil } @@ -58,7 +67,12 @@ func consistencyFrom(rep docker.CheckReport, source string) *linter.ConsistencyC case "deprecation": c.Deprecations++ } - c.Findings = append(c.Findings, linter.ConsistencyFinding{Severity: f.Severity, Code: f.Code, Message: f.Message, Location: f.Location}) + cf := linter.ConsistencyFinding{Severity: f.Severity, Code: f.Code, Message: f.Message} + if len(f.Locations) > 0 { + l := f.Locations[0] + cf.Module, cf.Document, cf.Element = l.ModuleName, l.DocumentName, l.ElementName + } + c.Findings = append(c.Findings, cf) } if err := rep.Verify(); err != nil { c.Incomplete = err.Error() @@ -68,6 +82,6 @@ func consistencyFrom(rep docker.CheckReport, source string) *linter.ConsistencyC func init() { reportCmd.Flags().Bool("no-mx-check", false, "Do not run Mendix's consistency check (mx check); the report then says whether the project builds is unknown") - reportCmd.Flags().String("mx-check-results", "", "Read the output of an earlier 'mx check' or 'mxcli docker check' from this file instead of running it") + reportCmd.Flags().String("mx-check-results", "", "Read the JSON report of an earlier 'mx check -j ' instead of running the check") reportCmd.Flags().String("mxbuild-path", "", "Path to the mxbuild whose mx runs the check, overriding resolution by the project's Mendix version") } diff --git a/cmd/mxcli/cmd_report_consistency_test.go b/cmd/mxcli/cmd_report_consistency_test.go index f792ea4c1..a3d1692da 100644 --- a/cmd/mxcli/cmd_report_consistency_test.go +++ b/cmd/mxcli/cmd_report_consistency_test.go @@ -14,14 +14,15 @@ import ( "github.com/spf13/cobra" ) -const mxCheckWithErrors = `Checking app... -[error] [CE0117] "Error(s) in expression." at Microflow 'ACT_X' -[warning] [CW0702] "Unused variable." at Microflow 'ACT_Y' -[deprecation] [CD0001] "Deprecated widget." at Page 'Home' -The app contains: 1 error. -` - -const mxCheckClean = "Checking app...\nThe app contains: 0 errors.\n" +// mx check --json output, as recorded in docker/testdata/mxcheck. +func mxCheckJSON(t *testing.T, name string) []byte { + t.Helper() + b, err := os.ReadFile(filepath.Join("docker", "testdata", "mxcheck", name)) + if err != nil { + t.Fatal(err) + } + return b +} func consistencyCmd(t *testing.T, flags map[string]string) *cobra.Command { t.Helper() @@ -37,37 +38,51 @@ func consistencyCmd(t *testing.T, flags map[string]string) *cobra.Command { return c } -func stubMxCheck(t *testing.T, output string, err error) *docker.CheckOptions { +type mxCheckCall struct { + opts docker.CheckOptions + kinds docker.CheckKinds +} + +func stubMxCheck(t *testing.T, rep docker.CheckReport, err error) *mxCheckCall { t.Helper() orig := reportMxCheck t.Cleanup(func() { reportMxCheck = orig }) - var got docker.CheckOptions - reportMxCheck = func(opts docker.CheckOptions) (docker.CheckReport, error) { - got = opts - if output == "" { - return docker.CheckReport{}, err - } - return docker.ParseCheckReport(output), err + got := &mxCheckCall{} + reportMxCheck = func(opts docker.CheckOptions, kinds docker.CheckKinds) (docker.CheckReport, error) { + got.opts, got.kinds = opts, kinds + return rep, err + } + return got +} + +func parsedCheck(t *testing.T, name string) docker.CheckReport { + t.Helper() + rep, err := docker.ParseCheckJSON(mxCheckJSON(t, name)) + if err != nil { + t.Fatal(err) } - return &got + return rep } -// ako/mxcli#2423: the scored report runs mx check and says whether the -// project builds. +// ako/mxcli#2423: the scored report runs mx check, warnings and deprecations +// included, and says whether the project builds. func TestReportConsistencyRunsMxCheck(t *testing.T) { - opts := stubMxCheck(t, mxCheckWithErrors, errors.New("exit status 1")) + call := stubMxCheck(t, parsedCheck(t, "f05_undef_all_kinds.json"), errors.New("exit status 1")) c, err := reportConsistency(consistencyCmd(t, map[string]string{"mxbuild-path": "/opt/mxbuild"}), "app.mpr", io.Discard) if err != nil { t.Fatal(err) } - if !c.Ran || c.Errors != 1 || c.Warnings != 1 || c.Deprecations != 1 || c.Builds() || c.Incomplete != "" { - t.Errorf("errors found: %+v", c) + if !c.Ran || c.Errors != 1 || c.Warnings != 16 || c.Deprecations != 1 || c.Builds() || c.Incomplete != "" { + t.Errorf("errors found: ran=%v %d/%d/%d incomplete=%q", c.Ran, c.Errors, c.Warnings, c.Deprecations, c.Incomplete) } - if opts.ProjectPath != "app.mpr" || opts.MxBuildPath != "/opt/mxbuild" { - t.Errorf("mx check got %+v", *opts) + if call.opts.ProjectPath != "app.mpr" || call.opts.MxBuildPath != "/opt/mxbuild" || !call.kinds.Warnings || !call.kinds.Deprecations { + t.Errorf("mx check got %+v", *call) + } + if f := c.Findings[0]; f.Module == "" || !strings.Contains(f.Document, "'") { + t.Errorf("a finding keeps its module and document: %+v", f) } - stubMxCheck(t, mxCheckClean, nil) + stubMxCheck(t, parsedCheck(t, "f10_clean.json"), nil) c, _ = reportConsistency(consistencyCmd(t, nil), "app.mpr", io.Discard) if !c.Builds() || c.Verdict() != "yes" { t.Errorf("clean project: %+v", c) @@ -75,10 +90,10 @@ func TestReportConsistencyRunsMxCheck(t *testing.T) { } // No mx is a report that says the check did not run and how to run it, never a -// clean one; and output whose error count disagrees with mx's own is not +// clean one; and findings that disagree with mx's declared total are not // "builds". func TestReportConsistencyNeverCleanWhenUnknown(t *testing.T) { - stubMxCheck(t, "", errors.New("mx not found; specify --mxbuild-path")) + stubMxCheck(t, docker.CheckReport{}, errors.New("mx not found; specify --mxbuild-path")) c, err := reportConsistency(consistencyCmd(t, nil), "app.mpr", io.Discard) if err != nil { t.Fatal(err) @@ -87,16 +102,11 @@ func TestReportConsistencyNeverCleanWhenUnknown(t *testing.T) { t.Errorf("no mx: %+v", c) } - // mx stopped before its closing count: one error seen, the rest unknown. - stubMxCheck(t, `[error] [CE0117] "x" at Microflow 'A'`+"\n", errors.New("signal: killed")) - c, _ = reportConsistency(consistencyCmd(t, nil), "app.mpr", io.Discard) - if !c.Ran || c.Errors != 1 || c.Incomplete == "" { - t.Errorf("cut short: %+v", c) - } - stubMxCheck(t, "The app contains: 2 errors.\n", errors.New("exit status 1")) + short := docker.CheckReport{Declared: 2, HasDeclared: true} + stubMxCheck(t, short, errors.New("exit status 1")) c, _ = reportConsistency(consistencyCmd(t, nil), "app.mpr", io.Discard) if c.Builds() || c.Incomplete == "" || !strings.HasPrefix(c.Verdict(), "unknown") { - t.Errorf("count mismatch must not read as building: %+v", c) + t.Errorf("a total the findings do not add up to must not read as building: %+v", c) } } @@ -104,10 +114,13 @@ func TestReportConsistencyResultsFileAndSkip(t *testing.T) { called := false orig := reportMxCheck t.Cleanup(func() { reportMxCheck = orig }) - reportMxCheck = func(docker.CheckOptions) (docker.CheckReport, error) { called = true; return docker.CheckReport{}, nil } + reportMxCheck = func(docker.CheckOptions, docker.CheckKinds) (docker.CheckReport, error) { + called = true + return docker.CheckReport{}, nil + } - path := filepath.Join(t.TempDir(), "check.txt") - if err := os.WriteFile(path, []byte("Using mx: /x/mx\n"+mxCheckWithErrors), 0644); err != nil { + path := filepath.Join(t.TempDir(), "check.json") + if err := os.WriteFile(path, mxCheckJSON(t, "f01_dupvar.json"), 0644); err != nil { t.Fatal(err) } c, err := reportConsistency(consistencyCmd(t, map[string]string{"mx-check-results": path}), "app.mpr", io.Discard) @@ -117,6 +130,11 @@ func TestReportConsistencyResultsFileAndSkip(t *testing.T) { if _, err := reportConsistency(consistencyCmd(t, map[string]string{"mx-check-results": path + ".missing"}), "app.mpr", io.Discard); err == nil { t.Error("a results file that cannot be read is an error, not a clean check") } + text := filepath.Join(t.TempDir(), "check.txt") + _ = os.WriteFile(text, []byte("[error] [CE0117] \"x\" at Microflow 'A'\nThe app contains: 1 error.\n"), 0644) + if _, err := reportConsistency(consistencyCmd(t, map[string]string{"mx-check-results": text}), "app.mpr", io.Discard); err == nil || !strings.Contains(err.Error(), "-j") { + t.Errorf("printed mx check output is refused, saying which file it takes: %v", err) + } c, _ = reportConsistency(consistencyCmd(t, map[string]string{"no-mx-check": "true"}), "app.mpr", io.Discard) if c.Ran || !strings.Contains(c.NotRun, "--no-mx-check") { t.Errorf("skipped: %+v", c) diff --git a/docs-site/src/tools/mxcli-report.md b/docs-site/src/tools/mxcli-report.md index 546d4e98b..69839e596 100644 --- a/docs-site/src/tools/mxcli-report.md +++ b/docs-site/src/tools/mxcli-report.md @@ -21,13 +21,16 @@ the project and puts its verdict above the score: | `not checked` | no mx for the project's Mendix version, or `--no-mx-check` | | `unknown` | mx's output could not be read in full | -The check's findings are listed under **Consistency Check**, grouped by code (a -stale theme or widget set gives hundreds of one code), and are not part of the -score. They cover the whole project, also with `--modules`. +The check's errors, warnings and deprecations are listed under **Consistency +Check**, grouped by code with the documents they are in (a stale theme or widget +set gives hundreds of one code), and are not part of the score. They cover the +whole project, also with `--modules`. `mx` writes into the project it checks, so +the report runs it on a temporary copy; run `mx check` yourself only on a copy +too. ```bash mxcli setup mxbuild -p app.mpr # mx for the project's version, once -mxcli report -p app.mpr --mx-check-results check.txt # read saved 'mxcli docker check' output +mxcli report -p app.mpr --mx-check-results check.json # the JSON 'mx check -j check.json -w -d' wrote mxcli report -p app.mpr --mxbuild-path /opt/mxbuild # a specific mxbuild mxcli report -p app.mpr --no-mx-check # lint only ``` diff --git a/mdl/linter/report_consistency.go b/mdl/linter/report_consistency.go index 976e33aa1..03d18b966 100644 --- a/mdl/linter/report_consistency.go +++ b/mdl/linter/report_consistency.go @@ -42,7 +42,24 @@ type ConsistencyFinding struct { Severity string `json:"severity"` // error | warning | deprecation Code string `json:"code"` // CE0117, CW…, CD… Message string `json:"message"` - Location string `json:"location,omitempty"` + // Module, Document ("Microflow 'ACT_X'") and Element ("End event") say + // where; a project-level finding has none. + Module string `json:"module,omitempty"` + Document string `json:"document,omitempty"` + Element string `json:"element,omitempty"` +} + +// where is the finding's document, "Module: Microflow 'ACT_X'", or "" for a +// project-level finding. +func (f ConsistencyFinding) where() string { + switch { + case f.Module != "" && f.Document != "": + return f.Module + ": " + f.Document + case f.Document != "": + return f.Document + default: + return f.Module + } } // Verdict is the one-line answer to "does it build?". @@ -93,7 +110,7 @@ type ConsistencyGroup struct { Code string Count int Message string // the first finding's; they differ only in detail - Locations []string // the first few distinct ones + Locations []string // the first few distinct documents More int // distinct locations not listed } @@ -118,10 +135,10 @@ func (c *ConsistencyCheck) Groups() []ConsistencyGroup { } g := &out[i] g.Count++ - if f.Location != "" && !seen[k][f.Location] { - seen[k][f.Location] = true + if loc := f.where(); loc != "" && !seen[k][loc] { + seen[k][loc] = true if len(g.Locations) < maxGroupLocations { - g.Locations = append(g.Locations, f.Location) + g.Locations = append(g.Locations, loc) } else { g.More++ } @@ -190,7 +207,7 @@ func writeConsistencyMarkdown(w io.Writer, c *ConsistencyCheck, modules []string for _, h := range c.hints() { fmt.Fprintf(w, "%s\n\n", h) } - fmt.Fprintf(w, "Every finding is in `--format json`; `mxcli docker check -p ` prints them with mx's own wording.\n\n") + fmt.Fprintf(w, "Every finding, with its element, is in `--format json`; `mxcli docker check -p ` prints them with mx's own wording.\n\n") } func writeConsistencyHTML(w io.Writer, c *ConsistencyCheck, modules []string) { diff --git a/mdl/linter/report_consistency_test.go b/mdl/linter/report_consistency_test.go index 62d46b0f8..76138a304 100644 --- a/mdl/linter/report_consistency_test.go +++ b/mdl/linter/report_consistency_test.go @@ -24,12 +24,12 @@ func renderReport(t *testing.T, format string, c *ConsistencyCheck) string { // rules (none fire here), and the report must still say it does not build. func TestReportSaysAProjectWithConsistencyErrorsDoesNotBuild(t *testing.T) { c := &ConsistencyCheck{Ran: true, Source: "mx check", Errors: 1, Warnings: 1, Findings: []ConsistencyFinding{ - {Severity: "error", Code: "CE0117", Message: "Error(s) in expression.", Location: "Microflow 'ACT_X'"}, - {Severity: "warning", Code: "CW0001", Message: "a warning | with a pipe", Location: "Page 'P'"}, + {Severity: "error", Code: "CE0117", Message: "Error(s) in expression.", Module: "Shop", Document: "Microflow 'ACT_X'", Element: "Change object"}, + {Severity: "warning", Code: "CW0001", Message: "a warning | with a pipe", Module: "Shop", Document: "Page 'P'"}, }} md := renderReport(t, "markdown", c) for _, want := range []string{"**Builds:** no — 1 consistency error(s)", "100/100", "## Consistency Check", - "The project does not build", "| error | CE0117 | 1 | Error(s) in expression. | Microflow 'ACT_X' |", `with a pipe`} { + "The project does not build", "| error | CE0117 | 1 | Error(s) in expression. | Shop: Microflow 'ACT_X' |", `with a pipe`} { if !strings.Contains(md, want) { t.Errorf("markdown lacks %q:\n%s", want, md) } @@ -88,15 +88,15 @@ func TestReportNeverCallsAnUncheckedProjectClean(t *testing.T) { // largest first, with a few locations; JSON keeps every finding. func TestConsistencyFindingsAreGroupedByCode(t *testing.T) { c := &ConsistencyCheck{Ran: true} - add := func(sev, code, loc string, n int) { + add := func(sev, code, doc string, n int) { for i := 0; i < n; i++ { - c.Findings = append(c.Findings, ConsistencyFinding{Severity: sev, Code: code, Message: "m " + code, Location: loc}) + c.Findings = append(c.Findings, ConsistencyFinding{Severity: sev, Code: code, Message: "m " + code, Module: "M", Document: doc, Element: "e"}) } } add("warning", "CW0001", "Page 'A'", 50) - add("error", "CE0463", "Image 'image1'", 3) + add("error", "CE0463", "Page 'Home'", 3) for i := 0; i < 8; i++ { - add("error", "CE6083", "Layout grid row "+string(rune('a'+i)), 2) + add("error", "CE6083", "Page 'P"+string(rune('a'+i))+"'", 2) } c.Errors = 19 gs := c.Groups() From 859c4d0fb21ae16c4a9d564489dd34ed500c2a3e Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 19 Nov 2026 10:07:30 +0000 Subject: [PATCH 05/66] feat(report): report release says what a deploy does to production data A model can be valid, pass mx check and build, and still drop a table on the next deploy. The runtime keys the database on each element's GUID, not its name, and nothing showed what a release does to the data already stored. `mxcli report release -p app.mpr --against ` reads both models' persistent entities, attributes and associations from their stored form. It pairs them by GUID, the way the runtime does, and lists four effects: - destroys: removed, no longer persistent, now calculated, or a new GUID under the same name (table created empty; column recreated with the model default, #1119). A re-mint is one destroy, not a destroy plus an add. An association whose end's table goes is not listed again; - verify: attribute type, shorter string, association type, storage or ends, generalization. What the runtime does with these is not measured, and the page says so; - migrates: renamed or moved under the same GUID, rows kept; - adds: new tables, columns and associations. Persistence follows the generalization chain, into System entities (meta.SystemEntityPersistable). External entities and calculated attributes have no storage and are left out. A git revision is exported with `git archive` (only the .mpr and mprcontents) into a temporary directory; the project may sit in a subdirectory of the repository. An intended drop is annotated release:: with status accepted, and the verdict counts it apart. Docs: a project-reports section and a report-release skill; assess-quality links it. Closes #2424 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82qTgUsRMETmzvjS27w --- .claude/skills/mendix/assess-quality/SKILL.md | 19 +- .claude/skills/mendix/report-release/SKILL.md | 87 ++++ cmd/mxcli/cmd_report.go | 1 + cmd/mxcli/cmd_report_release.go | 71 +++ cmd/mxcli/projectreport/release.go | 485 ++++++++++++++++++ cmd/mxcli/projectreport/release_test.go | 234 +++++++++ cmd/mxcli/release_model.go | 370 +++++++++++++ cmd/mxcli/release_model_test.go | 108 ++++ docs-site/src/tools/project-reports.md | 30 +- 9 files changed, 1395 insertions(+), 10 deletions(-) create mode 100644 .claude/skills/mendix/report-release/SKILL.md create mode 100644 cmd/mxcli/cmd_report_release.go create mode 100644 cmd/mxcli/projectreport/release.go create mode 100644 cmd/mxcli/projectreport/release_test.go create mode 100644 cmd/mxcli/release_model.go create mode 100644 cmd/mxcli/release_model_test.go diff --git a/.claude/skills/mendix/assess-quality/SKILL.md b/.claude/skills/mendix/assess-quality/SKILL.md index 156396d73..0e0f3423b 100644 --- a/.claude/skills/mendix/assess-quality/SKILL.md +++ b/.claude/skills/mendix/assess-quality/SKILL.md @@ -10,15 +10,15 @@ description: "Write a full quality assessment of a Mendix project: the mxcli rep | lines | section | |---|---| -| 31-52 | Step 1: Run the reports | -| 53-207 | Step 2: Manual review | -| 208-214 | Step 3: Write the assessment | -| 215-217 | Summary | -| 218-229 | Measured | -| 230-232 | Reviewed by hand | -| 233-235 | Recommendations | -| 236-240 | Not assessed | -| 241-247 | Sources | +| 31-53 | Step 1: Run the reports | +| 54-208 | Step 2: Manual review | +| 209-215 | Step 3: Write the assessment | +| 216-218 | Summary | +| 219-230 | Measured | +| 231-233 | Reviewed by hand | +| 234-236 | Recommendations | +| 237-241 | Not assessed | +| 242-248 | Sources | # Assess Mendix Project Quality @@ -39,6 +39,7 @@ structure make sense, whether error handling is designed or accidental. | Is it fast enough at production volume? | `mxcli report performance` | `report-performance` | | Was the specification built? | `mxcli report traceability` | `report-traceability` | | What is in it, for someone new? | `mxcli report map` | `report-map` | +| What does the next deploy do to production data? | `mxcli report release --against ` | `report-release` | | Does it build? | `mxcli report` (its **Builds** line), or `mxcli docker check` | `report-quality`, `/validate-project` | Read each report's skill before interpreting it — every one has traps (the quality diff --git a/.claude/skills/mendix/report-release/SKILL.md b/.claude/skills/mendix/report-release/SKILL.md new file mode 100644 index 000000000..53f40b9bf --- /dev/null +++ b/.claude/skills/mendix/report-release/SKILL.md @@ -0,0 +1,87 @@ +--- +name: report-release +description: "Produce and read the release data-impact report (mxcli report release --against ): what deploying this model does to production data — tables and columns dropped, renamed or added, and changes whose effect must be tried on a copy. Use before deploying a release, after changing or moving entities and attributes, after a Marketplace module update, or when asked whether a change is safe for the data in production." +--- + +## Find it fast + +`sed -n ',p' SKILL.md` one section rather than reading the file; +`./mxcli syntax ` answers *how do I write this* for a fraction of either. + +| lines | section | +|---|---| +| 26-40 | Run it | +| 41-58 | How the data is identified | +| 59-75 | Reading it | +| 76-83 | What it does not see | +| 84-87 | Related | + + +# Release Data-Impact Report + +`mxcli report release` answers one question before a deploy: **what happens to the +data already in production?** Nothing else in the toolchain answers it. A model can be +valid, pass `mx check` and build cleanly, and still drop a table on the next deploy. + +## Run it + +```bash +# against the deployed git revision (tag, branch or commit) +mxcli report release -p app.mpr --against v1.4.0 -o docs/release.html +# against a copy of the deployed .mpr +mxcli report release -p app.mpr --against /backups/deployed/App.mpr +# for CI: the changes as data +mxcli report release -p app.mpr --against origin/production --format json -o release.json +jq '[.data.changes[] | select(.effect == "destroys")] | length' release.json +``` + +`--against` must be **what is deployed** — a tag per release makes this cheap. The +project may sit in a subdirectory of the repository. + +## How the data is identified + +The runtime keys every table and column on the element's **GUID**, not its name. The +report pairs the two models the same way: + +| Change | What the deploy does | Listed as | +|---|---|---| +| Rename or move to another module, same GUID | table or column renamed, rows kept | migrates | +| Same name, **new GUID** | table dropped and created empty; a column dropped and recreated with the model default | destroys | +| Removed, made non-persistent, made calculated | dropped with its data | destroys | +| Attribute type, shorter string, association type or ends, generalization | not measured | verify | +| New entity, attribute or association | added; existing rows get the default | adds | + +A new GUID under an old name is the dangerous one, because nothing looks changed: +re-creating an entity instead of altering it, copying a module from another app, or a +tool that rebuilds a domain model. Moving an entity between modules is safe **only** +if its GUID is kept. + +## Reading it + +- **Destroys** is the release blocker. For each one, either undo it (keep the entity or + attribute) or keep its data first (an export, a migration flow that copies it into + the new element before the old one goes), then record the decision: + + ```yaml + items: + "release:entity:Sales.LegacyOrder": {status: accepted, note: "archived to S3 by MF_Archive in 1.4.0"} + ``` + + `--annotations` with that file counts it apart in the verdict. +- **Verify** means deploy to a copy of production first and look at the data. The report + does not guess what the runtime does with a type conversion. +- A change in a **Marketplace module** usually comes from a module update; it destroys + the data all the same. + +## What it does not see + +- A database that drifted from its model: it compares two models, not a model with a + database. +- Removed or renamed enumeration values and changed defaults: rows keep values the + new model may not know. +- Indexes, validation and access rules: no stored data behind them. + +## Related + +- `report-quality` — whether the model builds and how it scores; `manage-security` — + access rules, which a release does not migrate for you. diff --git a/cmd/mxcli/cmd_report.go b/cmd/mxcli/cmd_report.go index a6bd8ef40..9fc41dce2 100644 --- a/cmd/mxcli/cmd_report.go +++ b/cmd/mxcli/cmd_report.go @@ -29,6 +29,7 @@ For reports that answer one question rather than score, use a kind: mxcli report performance journeys at volume: time per step, plans, before/after mxcli report traceability requirements per milestone: built, partial, proven mxcli report map interactive guide: modules, data, flows, page flows, index + mxcli report release what deploying this model does to production data Output formats: - markdown (default): Human-readable Markdown with tables and progress bars diff --git a/cmd/mxcli/cmd_report_release.go b/cmd/mxcli/cmd_report_release.go new file mode 100644 index 000000000..05b3f5f63 --- /dev/null +++ b/cmd/mxcli/cmd_report_release.go @@ -0,0 +1,71 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "fmt" + + "github.com/mendixlabs/mxcli/cmd/mxcli/projectreport" + "github.com/spf13/cobra" +) + +var reportReleaseCmd = newReportKindCmd(reportKindSpec{ + name: "release", + short: "What deploying this model does to production data: tables and columns dropped, renamed, added", + long: `Compare the model with the one that is deployed and say, for every persistent +entity, attribute and association, what the deploy does to the data already +stored: + + destroys a table or column is dropped with its data: removed, no longer + persistent, now calculated, or a GUID changed under the same name + verify handled in a way that has not been measured: an attribute type, a + shorter string, an association's type or ends, a generalization + migrates renamed or moved under the same GUID: the rows are kept + adds a new table, column or association + +The runtime keys the database on each element's GUID, not its name, so the two +models are paired by GUID. A model can be valid, pass mx check and build, and +still drop a table on the next deploy; this report is what shows it. + +--against is what is deployed: a git revision of the project (a tag, a branch, +a commit) or the path of a .mpr. Annotation keys: release::, with +status accepted for a drop that is intended.`, + example: ` mxcli report release -p app.mpr --against v1.4.0 -o release.html + mxcli report release -p app.mpr --against origin/production --format json -o release.json + mxcli report release -p app.mpr --against /backups/deployed/App.mpr`, + flags: func(cmd *cobra.Command) { + cmd.Flags().String("against", "", "What is deployed: a git revision of the project, or a .mpr path (required)") + }, + collect: func(cmd *cobra.Command, projectPath string) (*projectreport.Envelope, error) { + against, _ := cmd.Flags().GetString("against") + if against == "" { + return nil, fmt.Errorf("--against is required: the git revision (or .mpr) that is deployed, e.g. --against v1.4.0") + } + basePath, label, cleanup, err := resolveReleaseBaseline(projectPath, against) + defer cleanup() + if err != nil { + return nil, err + } + deployed, err := loadReleaseModel(basePath) + if err != nil { + return nil, fmt.Errorf("read the deployed model (%s): %w", label, err) + } + next, err := loadReleaseModel(projectPath) + if err != nil { + return nil, fmt.Errorf("read the model: %w", err) + } + data := projectreport.CompareReleases(deployed, next) + data.Against, data.Current = label, projectPath + var warnings []string + if data.Tables == 0 { + warnings = append(warnings, "The deployed model has no persistent entities, so nothing can be destroyed. Check that --against is what is deployed.") + } + return projectreport.NewEnvelope("release", reportProjectName(projectPath), reportNow(), version, data, + projectreport.ReleaseMethod, projectreport.ReleaseLimits, warnings) + }, + render: projectreport.RenderRelease, +}) + +func init() { + reportCmd.AddCommand(reportReleaseCmd) +} diff --git a/cmd/mxcli/projectreport/release.go b/cmd/mxcli/projectreport/release.go new file mode 100644 index 000000000..bdbeb2b28 --- /dev/null +++ b/cmd/mxcli/projectreport/release.go @@ -0,0 +1,485 @@ +// SPDX-License-Identifier: Apache-2.0 + +package projectreport + +import ( + "fmt" + "sort" + "strings" +) + +// The release report (ako/mxcli#2424) says what deploying this model does to +// the data already in production. Nothing else does: the model stays valid, mx +// check is clean and the build passes, and the data goes on the next deploy. +// +// The runtime identifies a table by its entity's GUID and a column by its +// attribute's GUID (mendixsystem$entity.id, mendixsystem$attribute.id; measured +// on Mendix 11.12.1 against PostgreSQL, CLAUDE.md "A GUID Is the Database's +// Identity"). So the report pairs the two models exactly as the runtime will: +// by GUID, never by name. +// +// - Same GUID, other name or module: the runtime renames the table or column, +// and the rows are kept. +// - Same name, other GUID: the runtime sees one entity removed and another +// added, drops the table and creates an empty one. For an attribute the +// column is dropped and recreated, filled with the model default (#1119). +// - GUID only in the deployed model: dropped. +// +// A type change keeps the GUID, and what the runtime does with the column then +// — convert it, or drop and recreate it — has not been measured. The report +// says so ("verify") rather than guess. + +// ReleaseModel is the part of a model that has data behind it. +type ReleaseModel struct { + Entities []RelEntity `json:"entities"` + Associations []RelAssoc `json:"associations"` +} + +// RelEntity is an entity, stored or not: a change of persistence matters too. +type RelEntity struct { + GUID string `json:"guid"` + ID string `json:"id"` + Name string `json:"name"` // Module.Entity + Module string `json:"module"` + Marketplace bool `json:"marketplace,omitempty"` + // Stored: persistable (itself or through its generalization) and not an + // external entity. Only a stored entity has a table. + Stored bool `json:"stored"` + Generalization string `json:"generalization,omitempty"` + Attributes []RelAttr `json:"attributes"` +} + +// RelAttr is an attribute. A calculated one has no column. +type RelAttr struct { + GUID string `json:"guid"` + Name string `json:"name"` + Type string `json:"type"` // String, Integer, Enumeration, … + Length int `json:"length,omitempty"` + Calculated bool `json:"calculated,omitempty"` +} + +// RelAssoc is an association (or a cross-module association). From and To are +// the GUIDs of the entities at its ends. +type RelAssoc struct { + GUID string `json:"guid"` + Name string `json:"name"` + Module string `json:"module"` + From string `json:"from"` + To string `json:"to"` + FromName string `json:"fromName"` + ToName string `json:"toName"` + Type string `json:"type"` // Reference | ReferenceSet + StorageFormat string `json:"storageFormat"` // Table | Column +} + +// Release effects, in the order the page lists them. +const ( + EffectDestroys = "destroys" + EffectVerify = "verify" + EffectMigrates = "migrates" + EffectAdds = "adds" +) + +// ReleaseChange is one thing the deploy does to stored data. +type ReleaseChange struct { + Effect string `json:"effect"` + Kind string `json:"kind"` // entity | attribute | association + Name string `json:"name"` // as it will be after the deploy (as it was, if removed) + Was string `json:"was,omitempty"` + Module string `json:"module"` + What string `json:"what"` + Evidence string `json:"evidence"` + // Marketplace is set when the element is in a Marketplace module: such a + // change usually comes with a module update. + Marketplace bool `json:"marketplace,omitempty"` +} + +// Key is the change's annotation key: release::. +func (c ReleaseChange) Key() string { return "release:" + c.Kind + ":" + c.Name } + +// ReleaseData is the collected release report. +type ReleaseData struct { + Against string `json:"against"` + Current string `json:"current"` + Tables int `json:"tables"` // stored entities in the deployed model + Changes []ReleaseChange `json:"changes"` +} + +// Count returns the changes with the given effect. +func (d *ReleaseData) Count(effect string) int { + n := 0 + for _, c := range d.Changes { + if c.Effect == effect { + n++ + } + } + return n +} + +// CompareReleases lists what deploying next does to data stored under deployed. +func CompareReleases(deployed, next *ReleaseModel) *ReleaseData { + d := &ReleaseData{Changes: []ReleaseChange{}} + oldByGUID, newByGUID := entityIndex(deployed), entityIndex(next) + newByName := map[string]*RelEntity{} + for i := range next.Entities { + newByName[next.Entities[i].Name] = &next.Entities[i] + } + paired := map[string]bool{} // new entity GUIDs accounted for + add := func(c ReleaseChange) { d.Changes = append(d.Changes, c) } + + for i := range deployed.Entities { + o := &deployed.Entities[i] + if !o.Stored { + continue + } + d.Tables++ + n := newByGUID[o.GUID] + switch { + case n != nil && n.Stored: + paired[n.GUID] = true + if n.Name != o.Name { + what := "renamed: the runtime renames the table, and its rows are kept" + if n.Module != o.Module { + what = "moved to module " + n.Module + ": the runtime renames the table, and its rows are kept" + } + add(ReleaseChange{Effect: EffectMigrates, Kind: "entity", Name: n.Name, Was: o.Name, Module: n.Module, Marketplace: n.Marketplace, + What: what, Evidence: "same GUID " + short(o.GUID)}) + } + if n.Generalization != o.Generalization { + add(ReleaseChange{Effect: EffectVerify, Kind: "entity", Name: n.Name, Module: n.Module, Marketplace: n.Marketplace, + What: fmt.Sprintf("generalization changed from %s to %s: the inheritance tables change; not measured, verify on a copy of production", orNone(o.Generalization), orNone(n.Generalization)), + Evidence: "same GUID " + short(o.GUID)}) + } + compareAttributes(o, n, add) + case n != nil: + paired[n.GUID] = true + add(ReleaseChange{Effect: EffectDestroys, Kind: "entity", Name: n.Name, Was: wasIf(o.Name, n.Name), Module: n.Module, Marketplace: n.Marketplace, + What: "no longer persistent (or now external): the table and all its rows are dropped", Evidence: "same GUID " + short(o.GUID)}) + default: + if same := newByName[o.Name]; same != nil && same.Stored && oldByGUID[same.GUID] == nil { + paired[same.GUID] = true + add(ReleaseChange{Effect: EffectDestroys, Kind: "entity", Name: o.Name, Module: o.Module, Marketplace: o.Marketplace || same.Marketplace, + What: "GUID changed under the same name: the runtime treats it as another entity, drops the table with all its rows and creates an empty one", + Evidence: "GUID " + short(o.GUID) + " → " + short(same.GUID)}) + continue + } + add(ReleaseChange{Effect: EffectDestroys, Kind: "entity", Name: o.Name, Module: o.Module, Marketplace: o.Marketplace, + What: "removed: the table and all its rows are dropped", Evidence: "GUID " + short(o.GUID) + " is not in the new model"}) + } + } + for i := range next.Entities { + n := &next.Entities[i] + if !n.Stored || paired[n.GUID] { + continue + } + what := "new entity: a new, empty table" + if o := oldByGUID[n.GUID]; o != nil { + what = "now persistent: a new, empty table" + } + add(ReleaseChange{Effect: EffectAdds, Kind: "entity", Name: n.Name, Module: n.Module, Marketplace: n.Marketplace, What: what, Evidence: "GUID " + short(n.GUID)}) + } + compareAssociations(deployed, next, oldByGUID, newByGUID, add) + + rank := map[string]int{EffectDestroys: 0, EffectVerify: 1, EffectMigrates: 2, EffectAdds: 3} + sort.SliceStable(d.Changes, func(i, j int) bool { + a, b := d.Changes[i], d.Changes[j] + if rank[a.Effect] != rank[b.Effect] { + return rank[a.Effect] < rank[b.Effect] + } + return a.Name < b.Name + }) + return d +} + +func entityIndex(m *ReleaseModel) map[string]*RelEntity { + out := map[string]*RelEntity{} + for i := range m.Entities { + if g := m.Entities[i].GUID; g != "" { + out[g] = &m.Entities[i] + } + } + return out +} + +func compareAttributes(o, n *RelEntity, add func(ReleaseChange)) { + oldBy, newBy, newByName := map[string]*RelAttr{}, map[string]*RelAttr{}, map[string]*RelAttr{} + for i := range o.Attributes { + oldBy[o.Attributes[i].GUID] = &o.Attributes[i] + } + for i := range n.Attributes { + newBy[n.Attributes[i].GUID] = &n.Attributes[i] + newByName[n.Attributes[i].Name] = &n.Attributes[i] + } + paired := map[string]bool{} + ch := func(effect, name, was, what, evidence string) { + add(ReleaseChange{Effect: effect, Kind: "attribute", Name: n.Name + "." + name, Was: was, Module: n.Module, Marketplace: n.Marketplace, What: what, Evidence: evidence}) + } + for i := range o.Attributes { + oa := &o.Attributes[i] + if oa.Calculated { + continue // no column + } + na := newBy[oa.GUID] + switch { + case na != nil && !na.Calculated: + paired[na.GUID] = true + if na.Name != oa.Name { + ch(EffectMigrates, na.Name, o.Name+"."+oa.Name, "renamed: the runtime renames the column, and its values are kept", "same GUID "+short(oa.GUID)) + } + switch { + case na.Type != oa.Type: + ch(EffectVerify, na.Name, "", fmt.Sprintf("type changed from %s to %s: whether the runtime converts the column or drops and recreates it is not measured; verify on a copy of production", + typeLabel(oa), typeLabel(na)), "same GUID "+short(oa.GUID)) + case na.Type == "String" && shorter(na.Length, oa.Length): + ch(EffectVerify, na.Name, "", fmt.Sprintf("length lowered from %s to %s: stored values longer than that may be cut or refused; verify on a copy of production", + lengthLabel(oa.Length), lengthLabel(na.Length)), "same GUID "+short(oa.GUID)) + } + case na != nil: + paired[na.GUID] = true + ch(EffectDestroys, na.Name, "", "now calculated: the column and its values are dropped", "same GUID "+short(oa.GUID)) + default: + if same := newByName[oa.Name]; same != nil && !same.Calculated && oldBy[same.GUID] == nil { + paired[same.GUID] = true + ch(EffectDestroys, oa.Name, "", "GUID changed under the same name: the column is dropped and recreated, and every row gets the model default (ako/mxcli#1119)", + "GUID "+short(oa.GUID)+" → "+short(same.GUID)) + continue + } + ch(EffectDestroys, oa.Name, "", "removed: the column and its values are dropped", "GUID "+short(oa.GUID)+" is not in the new model") + } + } + for i := range n.Attributes { + na := &n.Attributes[i] + if na.Calculated || paired[na.GUID] { + continue + } + ch(EffectAdds, na.Name, "", "new column ("+typeLabel(na)+"): existing rows get the model default", "GUID "+short(na.GUID)) + } +} + +// compareAssociations pairs associations by GUID. An association between two +// stored entities stores references; one whose end is not stored stores nothing. +func compareAssociations(deployed, next *ReleaseModel, oldEnt, newEnt map[string]*RelEntity, add func(ReleaseChange)) { + stored := func(a *RelAssoc, ents map[string]*RelEntity) bool { + f, t := ents[a.From], ents[a.To] + return f != nil && t != nil && f.Stored && t.Stored + } + gone := func(guid string) bool { // the end entity's table is dropped anyway + n := newEnt[guid] + return n == nil || !n.Stored + } + newBy, newByName := map[string]*RelAssoc{}, map[string]*RelAssoc{} + oldBy := map[string]*RelAssoc{} + for i := range next.Associations { + newBy[next.Associations[i].GUID] = &next.Associations[i] + newByName[next.Associations[i].Name] = &next.Associations[i] + } + for i := range deployed.Associations { + oldBy[deployed.Associations[i].GUID] = &deployed.Associations[i] + } + paired := map[string]bool{} + ch := func(effect string, a *RelAssoc, was, what, evidence string) { + mp := false + if e := newEnt[a.From]; e != nil { + mp = e.Marketplace + } else if e := oldEnt[a.From]; e != nil { + mp = e.Marketplace + } + add(ReleaseChange{Effect: effect, Kind: "association", Name: a.Name, Was: was, Module: a.Module, Marketplace: mp, What: what, Evidence: evidence}) + } + for i := range deployed.Associations { + o := &deployed.Associations[i] + if !stored(o, oldEnt) { + continue + } + n := newBy[o.GUID] + switch { + case n != nil && stored(n, newEnt): + paired[n.GUID] = true + if n.Name != o.Name { + ch(EffectMigrates, n, o.Name, "renamed: the references are kept", "same GUID "+short(o.GUID)) + } + if n.From != o.From || n.To != o.To { + ch(EffectVerify, n, "", fmt.Sprintf("now connects %s → %s (was %s → %s): not measured; verify on a copy of production", n.FromName, n.ToName, o.FromName, o.ToName), "same GUID "+short(o.GUID)) + } + if n.Type != o.Type || n.StorageFormat != o.StorageFormat { + ch(EffectVerify, n, "", fmt.Sprintf("changed from %s to %s: the references are migrated or dropped, which is not measured; verify on a copy of production", + assocShape(o), assocShape(n)), "same GUID "+short(o.GUID)) + } + case gone(o.From) || gone(o.To): + // Its table goes with the entity's, which is listed already. + case n != nil: + paired[n.GUID] = true + ch(EffectDestroys, n, "", "an end is no longer persistent: the references are dropped", "same GUID "+short(o.GUID)) + default: + if same := newByName[o.Name]; same != nil && stored(same, newEnt) && oldBy[same.GUID] == nil { + paired[same.GUID] = true + ch(EffectDestroys, o, "", "GUID changed under the same name: the references are dropped and the association starts empty", + "GUID "+short(o.GUID)+" → "+short(same.GUID)) + continue + } + ch(EffectDestroys, o, "", "removed: the references are dropped", "GUID "+short(o.GUID)+" is not in the new model") + } + } + for i := range next.Associations { + n := &next.Associations[i] + if paired[n.GUID] || !stored(n, newEnt) { + continue + } + ch(EffectAdds, n, "", "new association ("+assocShape(n)+"): it starts empty", "GUID "+short(n.GUID)) + } +} + +func assocShape(a *RelAssoc) string { + s := a.Type + if a.StorageFormat != "" { + s += ", stored as " + strings.ToLower(a.StorageFormat) + } + return s +} + +func typeLabel(a *RelAttr) string { + if a.Type == "String" { + return "String(" + lengthLabel(a.Length) + ")" + } + return a.Type +} + +// lengthLabel prints a String length; 0 is Mendix's "unlimited". +func lengthLabel(n int) string { + if n == 0 { + return "unlimited" + } + return fmt.Sprint(n) +} + +// shorter reports whether length a is less than b, where 0 is unlimited. +func shorter(a, b int) bool { + switch { + case a == 0: + return false + case b == 0: + return true + default: + return a < b + } +} + +func short(guid string) string { + if len(guid) > 8 { + return guid[:8] + } + return guid +} + +func orNone(s string) string { + if s == "" { + return "none" + } + return s +} + +func wasIf(was, now string) string { + if was == now { + return "" + } + return was +} + +// ---------------------------------------------------------------- render + +var ReleaseMethod = []string{ + "Both models are read from their stored form: each persistent entity, attribute and association with its GUID. External entities and calculated attributes have no table or column and are left out.", + "The two are paired by GUID, the way the runtime pairs the model with the database (mendixsystem$entity.id and mendixsystem$attribute.id hold the GUID; measured on Mendix 11.12.1 with PostgreSQL). Names are used only to say that an element kept its name but not its GUID.", + "Destroys: a table or column dropped — removed, no longer persistent, now calculated, or a GUID changed under the same name. Measured: an entity whose GUID changed loses its table; an attribute whose GUID changed gets its column dropped and recreated with the model default (#1119).", + "Migrates: the same GUID under another name or module; measured: the runtime renames the table and keeps the rows.", + "Verify: a change the runtime handles in a way that has not been measured — an attribute type, a shorter string, an association's type, storage or ends, a generalization.", +} + +var ReleaseLimits = []string{ + "It compares two models, not a model with a database: --against must be what is deployed. A database that has drifted from its model is not seen.", + "Enumeration values removed or renamed, and changed default values, are not compared: rows keep values the new model may not know.", + "Indexes, validation rules and access rules have no stored data behind them and are not compared.", + "Type conversions are listed as verify because what the runtime does with them has not been measured; try the deploy on a copy of production before trusting one.", +} + +// RenderRelease builds the release page. +func RenderRelease(env *Envelope, ann *Annotations) (Doc, error) { + var d ReleaseData + if err := env.Decode(&d); err != nil { + return Doc{}, err + } + doc := Doc{Title: "Release data impact", Subtitle: "What deploying this model does to the data in production"} + destroys := d.Count(EffectDestroys) + accepted := 0 + for _, c := range d.Changes { + if c.Effect == EffectDestroys { + if n, ok := ann.Get(c.Key()); ok && (n.Status == "accepted" || n.Status == "intended") { + accepted++ + } + } + } + verdict := "No stored data is destroyed." + switch { + case destroys > accepted: + verdict = fmt.Sprintf("This release destroys stored data in %d place(s)", destroys-accepted) + if accepted > 0 { + verdict += fmt.Sprintf(" (and %d more that are accepted)", accepted) + } + verdict += "." + case destroys > 0: + verdict = fmt.Sprintf("This release destroys stored data in %d place(s), all accepted.", destroys) + } + doc.Stats = []Stat{ + {Label: "destroys", Value: fmt.Sprint(destroys)}, + {Label: "verify", Value: fmt.Sprint(d.Count(EffectVerify))}, + {Label: "migrates", Value: fmt.Sprint(d.Count(EffectMigrates))}, + {Label: "adds", Value: fmt.Sprint(d.Count(EffectAdds))}, + {Label: "tables deployed", Value: fmt.Sprint(d.Tables)}, + } + kind := "info" + if destroys > accepted { + kind = "warn" + } + head := []Block{Callout{Kind: kind, Text: verdict + " Compared with " + d.Against + "."}} + doc.Sections = append(doc.Sections, Section{ID: "verdict", Title: "Verdict", Blocks: head}) + + sections := []struct{ effect, id, title, intro string }{ + {EffectDestroys, "destroys", "Destroys data", "Each of these drops a table or a column, with its data, on the next deploy. Annotate `release::` with status accepted when it is intended — and say how the data is kept (an export, a migration flow)."}, + {EffectVerify, "verify", "Verify on a copy of production", "The runtime handles these in a way that has not been measured. Deploy to a copy of production first."}, + {EffectMigrates, "migrates", "Migrates, rows kept", "Renamed or moved under the same GUID: the runtime renames the table or column."}, + {EffectAdds, "adds", "Adds", "New tables, columns and associations. Existing rows get the model default."}, + } + for _, s := range sections { + t := Table{Columns: []string{"Element", "Kind", "What happens", "Evidence", "Status"}} + for _, c := range d.Changes { + if c.Effect != s.effect { + continue + } + name := "`" + c.Name + "`" + if c.Was != "" { + name += " (was `" + c.Was + "`)" + } + if c.Marketplace { + name += " — Marketplace module" + } + status := "" + if n, ok := ann.Get(c.Key()); ok { + status = joinNote(n.Status, n.Note, "") + } + t.Rows = append(t.Rows, []string{name, c.Kind, c.What, c.Evidence, status}) + cls := "" + if c.Effect == EffectDestroys && !(strings.HasPrefix(status, "accepted") || strings.HasPrefix(status, "intended")) { + cls = "bad" + } + t.Classes = append(t.Classes, []string{cls, "", "", "", ""}) + } + var blocks []Block + if len(t.Rows) == 0 { + blocks = []Block{Para{Text: "None."}} + } else { + blocks = []Block{t} + } + doc.Sections = append(doc.Sections, Section{ID: s.id, Title: s.title, Intro: s.intro, Blocks: blocks}) + } + return Frame(env, ann, doc), nil +} diff --git a/cmd/mxcli/projectreport/release_test.go b/cmd/mxcli/projectreport/release_test.go new file mode 100644 index 000000000..e1fb8df96 --- /dev/null +++ b/cmd/mxcli/projectreport/release_test.go @@ -0,0 +1,234 @@ +// SPDX-License-Identifier: Apache-2.0 + +package projectreport + +import ( + "bytes" + "encoding/json" + "strings" + "testing" +) + +func releaseBaseline() *ReleaseModel { + return &ReleaseModel{ + Entities: []RelEntity{ + {GUID: "e-order", Name: "Shop.Order", Module: "Shop", Stored: true, Attributes: []RelAttr{ + {GUID: "a-num", Name: "Number", Type: "Integer"}, + {GUID: "a-note", Name: "Note", Type: "String", Length: 200}, + {GUID: "a-total", Name: "Total", Type: "Decimal"}, + {GUID: "a-code", Name: "Code", Type: "String", Length: 50}, + {GUID: "a-calc", Name: "Label", Type: "String", Calculated: true}, + {GUID: "a-keep", Name: "Kept", Type: "Boolean"}, + }}, + {GUID: "e-cust", Name: "Shop.Customer", Module: "Shop", Stored: true, Attributes: []RelAttr{{GUID: "a-name", Name: "Name", Type: "String", Length: 100}}}, + {GUID: "e-acct", Name: "Admin.Account", Module: "Admin", Marketplace: true, Stored: true, Generalization: "System.User"}, + {GUID: "e-gone", Name: "Shop.Legacy", Module: "Shop", Stored: true}, + {GUID: "e-np", Name: "Shop.Filter", Module: "Shop", Stored: false}, + {GUID: "e-log", Name: "Shop.Log", Module: "Shop", Stored: true}, + }, + Associations: []RelAssoc{ + {GUID: "s-oc", Name: "Shop.Order_Customer", From: "e-order", To: "e-cust", FromName: "Shop.Order", ToName: "Shop.Customer", Type: "Reference", StorageFormat: "Column"}, + {GUID: "s-ol", Name: "Shop.Order_Legacy", From: "e-order", To: "e-gone", FromName: "Shop.Order", ToName: "Shop.Legacy", Type: "Reference"}, + {GUID: "s-lo", Name: "Shop.Log_Order", From: "e-log", To: "e-order", FromName: "Shop.Log", ToName: "Shop.Order", Type: "Reference"}, + }, + } +} + +// clone deep-copies a model through JSON. +func clone(t *testing.T, m *ReleaseModel) *ReleaseModel { + t.Helper() + b, _ := json.Marshal(m) + var out ReleaseModel + if err := json.Unmarshal(b, &out); err != nil { + t.Fatal(err) + } + return &out +} + +func entity(m *ReleaseModel, name string) *RelEntity { + for i := range m.Entities { + if m.Entities[i].Name == name { + return &m.Entities[i] + } + } + return nil +} + +func attr(e *RelEntity, name string) *RelAttr { + for i := range e.Attributes { + if e.Attributes[i].Name == name { + return &e.Attributes[i] + } + } + return nil +} + +func changes(d *ReleaseData) map[string]ReleaseChange { + out := map[string]ReleaseChange{} + for _, c := range d.Changes { + out[c.Kind+" "+c.Name] = c + } + return out +} + +// The control: an unchanged model, and one renamed and moved everywhere under +// the same GUIDs, destroy nothing — names do not identify data, GUIDs do. +func TestReleaseSameGUIDsDestroyNothing(t *testing.T) { + old := releaseBaseline() + if d := CompareReleases(old, clone(t, old)); len(d.Changes) != 0 || d.Tables != 5 { + t.Fatalf("unchanged model: tables %d, changes %+v", d.Tables, d.Changes) + } + next := clone(t, old) + o := entity(next, "Shop.Order") + o.Name, o.Module = "Sales.Order", "Sales" + attr(o, "Total").Name = "Amount" + next.Associations[0].Name = "Sales.Order_Customer" + d := CompareReleases(old, next) + if d.Count(EffectDestroys) != 0 { + t.Fatalf("renames under the same GUID destroy nothing: %+v", d.Changes) + } + cs := changes(d) + if c := cs["entity Sales.Order"]; c.Effect != EffectMigrates || c.Was != "Shop.Order" || !strings.Contains(c.What, "moved to module Sales") { + t.Errorf("moved entity: %+v", c) + } + if c := cs["attribute Sales.Order.Amount"]; c.Effect != EffectMigrates || c.Was != "Shop.Order.Total" { + t.Errorf("renamed attribute: %+v", c) + } + if c := cs["association Sales.Order_Customer"]; c.Effect != EffectMigrates { + t.Errorf("renamed association: %+v", c) + } +} + +// ako/mxcli#2424: each way a deploy drops data, found by GUID. +func TestReleaseFindsWhatDestroysData(t *testing.T) { + old := releaseBaseline() + next := clone(t, old) + entity(next, "Admin.Account").GUID = "e-acct-reminted" // same name, new GUID + o := entity(next, "Shop.Order") + attr(o, "Code").GUID = "a-code-reminted" // same name, new GUID + attr(o, "Kept").Calculated = true // now calculated + var attrs []RelAttr + for _, a := range o.Attributes { + if a.Name != "Note" { + attrs = append(attrs, a) + } + } + o.Attributes = attrs // Note removed + entity(next, "Shop.Log").Stored = false // no longer persistent + var ents []RelEntity + for _, e := range next.Entities { + if e.Name != "Shop.Legacy" { + ents = append(ents, e) + } + } + next.Entities = ents // Legacy removed + next.Associations[0].GUID = "s-oc-reminted" + + d := CompareReleases(old, next) + cs := changes(d) + for key, want := range map[string]string{ + "entity Admin.Account": "GUID changed under the same name", + "entity Shop.Legacy": "removed", + "entity Shop.Log": "no longer persistent", + "attribute Shop.Order.Code": "GUID changed under the same name", + "attribute Shop.Order.Note": "removed", + "attribute Shop.Order.Kept": "now calculated", + "association Shop.Order_Customer": "GUID changed under the same name", + } { + c, ok := cs[key] + if !ok || c.Effect != EffectDestroys || !strings.Contains(c.What, want) { + t.Errorf("%s: want destroys %q, got %+v", key, want, c) + } + } + // The re-minted entity is not also reported as a new table, nor its + // re-minted attribute as a new column. + for _, c := range d.Changes { + if c.Effect == EffectAdds { + t.Errorf("a re-mint is one destroy, not a destroy plus an add: %+v", c) + } + } + // An association whose end's table is dropped is not listed again. + if _, ok := cs["association Shop.Order_Legacy"]; ok { + t.Error("Order_Legacy goes with Legacy's table; listing it again is noise") + } + if c := cs["entity Admin.Account"]; !c.Marketplace || !strings.Contains(c.Evidence, "e-acct") { + t.Errorf("evidence and Marketplace flag: %+v", c) + } + // A calculated attribute and a non-persistent entity never had storage. + for _, key := range []string{"attribute Shop.Order.Label", "entity Shop.Filter"} { + if _, ok := cs[key]; ok { + t.Errorf("%s has no storage and must not be listed", key) + } + } +} + +func TestReleaseVerifyAndAdds(t *testing.T) { + old := releaseBaseline() + next := clone(t, old) + o := entity(next, "Shop.Order") + attr(o, "Number").Type = "String" + attr(o, "Code").Length = 10 + attr(o, "Note").Length = 0 // unlimited: longer, not shorter + o.Attributes = append(o.Attributes, RelAttr{GUID: "a-status", Name: "Status", Type: "String", Length: 30}) + entity(next, "Shop.Customer").Generalization = "Shop.Party" + next.Associations[0].Type = "ReferenceSet" + next.Entities = append(next.Entities, RelEntity{GUID: "e-inv", Name: "Shop.Invoice", Module: "Shop", Stored: true}) + entity(next, "Shop.Filter").Stored = true + + cs := changes(CompareReleases(old, next)) + for key, want := range map[string]string{ + "attribute Shop.Order.Number": "type changed from Integer to String", + "attribute Shop.Order.Code": "length lowered from 50 to 10", + "entity Shop.Customer": "generalization changed", + "association Shop.Order_Customer": "Reference, stored as column to ReferenceSet", + } { + if c := cs[key]; c.Effect != EffectVerify || !strings.Contains(c.What, want) { + t.Errorf("%s: want verify %q, got %+v", key, want, c) + } + } + if _, ok := cs["attribute Shop.Order.Note"]; ok { + t.Error("a longer string limit is not a risk") + } + for key, want := range map[string]string{ + "attribute Shop.Order.Status": "new column", + "entity Shop.Invoice": "new entity", + "entity Shop.Filter": "now persistent", + } { + if c := cs[key]; c.Effect != EffectAdds || !strings.Contains(c.What, want) { + t.Errorf("%s: want adds %q, got %+v", key, want, c) + } + } +} + +// The page leads with the verdict, and an accepted drop is shown as accepted. +func TestRenderReleaseVerdictAndAcceptance(t *testing.T) { + old := releaseBaseline() + next := clone(t, old) + var ents []RelEntity + for _, e := range next.Entities { + if e.Name != "Shop.Legacy" && e.Name != "Shop.Log" { + ents = append(ents, e) + } + } + next.Entities = ents + d := CompareReleases(old, next) + d.Against = "git v1" + env, _ := NewEnvelope("release", "Shop", "now", "t", d, ReleaseMethod, ReleaseLimits, nil) + render := func(ann *Annotations) string { + doc, err := RenderRelease(env, ann) + if err != nil { + t.Fatal(err) + } + var b bytes.Buffer + _ = RenderMarkdown(&b, doc) + return b.String() + } + mustContain(t, render(nil), "This release destroys stored data in 2 place(s). Compared with git v1.", "`Shop.Legacy`", "removed: the table and all its rows are dropped") + md := render(&Annotations{Items: map[string]Note{"release:entity:Shop.Legacy": {Status: "accepted", Note: "archived to S3 first"}}}) + mustContain(t, md, "destroys stored data in 1 place(s) (and 1 more that are accepted)", "accepted: archived to S3 first") + + same := CompareReleases(old, clone(t, old)) + same.Against = "git v1" + env, _ = NewEnvelope("release", "Shop", "now", "t", same, ReleaseMethod, ReleaseLimits, nil) + mustContain(t, render(nil), "No stored data is destroyed.") +} diff --git a/cmd/mxcli/release_model.go b/cmd/mxcli/release_model.go new file mode 100644 index 000000000..2f2ec42c5 --- /dev/null +++ b/cmd/mxcli/release_model.go @@ -0,0 +1,370 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "archive/tar" + "bytes" + "encoding/hex" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "strings" + + "github.com/mendixlabs/mxcli/cmd/mxcli/projectreport" + "github.com/mendixlabs/mxcli/modelsdk/meta" + "github.com/mendixlabs/mxcli/modelsdk/mpr" + "go.mongodb.org/mongo-driver/v2/bson" +) + +// loadReleaseModel reads the persistent part of a model from its stored form: +// every entity, attribute and association with its GUID (ako/mxcli#2424). +// The GUID is read from the stored bytes because it is what the runtime keys +// the database on, and mxcli's own model types do not carry it. +func loadReleaseModel(mprPath string) (*projectreport.ReleaseModel, error) { + r, err := mpr.Open(mprPath) + if err != nil { + return nil, err + } + defer r.Close() + + type module struct { + name string + marketplace bool + } + modules := map[string]module{} + mods, err := r.ListRawUnitsByType("Projects$ModuleImpl") + if err != nil { + return nil, err + } + for _, u := range mods { + var d bson.D + if err := bson.Unmarshal(u.Contents, &d); err != nil { + return nil, fmt.Errorf("module unit %s: %w", u.ID, err) + } + fromStore, _ := field(d, "FromAppStore").(bool) + modules[string(u.ID)] = module{name: str(field(d, "Name")), marketplace: fromStore} + } + + dms, err := r.ListRawUnitsByType("DomainModels$DomainModel") + if err != nil { + return nil, err + } + m := &projectreport.ReleaseModel{} + type rawEntity struct { + idx int + persistable *bool // set by NoGeneralization + generalizes string // Generalization target, qualified + external bool + } + var raws []rawEntity + byID := map[string]int{} // entity $ID -> index in m.Entities + byName := map[string]int{} // qualified name -> index + type rawAssoc struct { + a projectreport.RelAssoc + from, to string // entity $IDs + toName string // cross-association: the target by name + } + var assocs []rawAssoc + + for _, u := range dms { + mod, ok := modules[string(u.ContainerID)] + if !ok { + continue + } + var dm bson.D + if err := bson.Unmarshal(u.Contents, &dm); err != nil { + return nil, fmt.Errorf("domain model of %s: %w", mod.name, err) + } + for _, ev := range elements(field(dm, "Entities")) { + e := projectreport.RelEntity{ + GUID: binHex(field(ev, "GUID")), ID: binHex(field(ev, "$ID")), + Name: mod.name + "." + str(field(ev, "Name")), Module: mod.name, Marketplace: mod.marketplace, + } + re := rawEntity{external: field(ev, "Source") != nil} + if g, ok := field(ev, "MaybeGeneralization").(bson.D); ok { + switch str(field(g, "$Type")) { + case "DomainModels$Generalization": + re.generalizes = str(field(g, "Generalization")) + default: + p, ok := field(g, "Persistable").(bool) + if !ok { + p = true // absent in older formats: persistable was the default + } + re.persistable = &p + } + } + e.Generalization = re.generalizes + for _, av := range elements(field(ev, "Attributes")) { + a := projectreport.RelAttr{GUID: binHex(field(av, "GUID")), Name: str(field(av, "Name"))} + if t, ok := field(av, "NewType").(bson.D); ok { + a.Type = strings.TrimSuffix(strings.TrimPrefix(str(field(t, "$Type")), "DomainModels$"), "AttributeType") + if n, ok := bsonInt(field(t, "Length")); ok { + a.Length = n + } + } + if v, ok := field(av, "Value").(bson.D); ok { + a.Calculated = str(field(v, "$Type")) == "DomainModels$CalculatedValue" + } + e.Attributes = append(e.Attributes, a) + } + re.idx = len(m.Entities) + byID[e.ID] = re.idx + byName[e.Name] = re.idx + m.Entities = append(m.Entities, e) + raws = append(raws, re) + } + for _, av := range elements(field(dm, "Associations")) { + assocs = append(assocs, rawAssoc{ + a: assocOf(av, mod.name), + from: binHex(field(av, "ParentPointer")), to: binHex(field(av, "ChildPointer")), + }) + } + for _, av := range elements(field(dm, "CrossAssociations")) { + assocs = append(assocs, rawAssoc{ + a: assocOf(av, mod.name), + from: binHex(field(av, "ParentPointer")), toName: str(field(av, "Child")), + }) + } + } + + // Persistence follows the generalization chain to its root, which may be + // a System entity. + var persistent func(i int, seen map[int]bool) bool + persistent = func(i int, seen map[int]bool) bool { + re := raws[i] + if re.persistable != nil { + return *re.persistable + } + if re.generalizes == "" || seen[i] { + return true + } + seen[i] = true + if j, ok := byName[re.generalizes]; ok { + return persistent(j, seen) + } + if p, known := meta.SystemEntityPersistable(re.generalizes); known { + return p + } + return true // an unknown parent: assume a table, so a change is reported, not hidden + } + for i := range raws { + m.Entities[i].Stored = !raws[i].external && persistent(i, map[int]bool{}) + } + + for _, ra := range assocs { + a := ra.a + if i, ok := byID[ra.from]; ok { + a.From, a.FromName = m.Entities[i].GUID, m.Entities[i].Name + } + to := -1 + if i, ok := byID[ra.to]; ok { + to = i + } else if i, ok := byName[ra.toName]; ok { + to = i + } + if to >= 0 { + a.To, a.ToName = m.Entities[to].GUID, m.Entities[to].Name + } else if ra.toName != "" { + // A System entity: stored, and the same in every model. + a.To, a.ToName = "system:"+ra.toName, ra.toName + if p, known := meta.SystemEntityPersistable(ra.toName); known && p { + m.Entities = append(m.Entities, projectreport.RelEntity{GUID: a.To, Name: ra.toName, Module: "System", Stored: true}) + } + } + m.Associations = append(m.Associations, a) + } + m.Entities = dedupeSystem(m.Entities) + return m, nil +} + +func assocOf(av bson.D, module string) projectreport.RelAssoc { + return projectreport.RelAssoc{ + GUID: binHex(field(av, "GUID")), Name: module + "." + str(field(av, "Name")), Module: module, + Type: str(field(av, "Type")), StorageFormat: str(field(av, "StorageFormat")), + } +} + +// dedupeSystem keeps one placeholder per System entity an association names. +func dedupeSystem(es []projectreport.RelEntity) []projectreport.RelEntity { + seen := map[string]bool{} + out := es[:0] + for _, e := range es { + if strings.HasPrefix(e.GUID, "system:") { + if seen[e.GUID] { + continue + } + seen[e.GUID] = true + } + out = append(out, e) + } + return out +} + +func field(d bson.D, key string) any { + for _, e := range d { + if e.Key == key { + return e.Value + } + } + return nil +} + +// elements returns the documents of a stored list, whose first item is the +// list-type marker. +func elements(v any) []bson.D { + a, _ := v.(bson.A) + var out []bson.D + for _, x := range a { + if d, ok := x.(bson.D); ok { + out = append(out, d) + } + } + return out +} + +func str(v any) string { + s, _ := v.(string) + return s +} + +func binHex(v any) string { + switch b := v.(type) { + case bson.Binary: + return hex.EncodeToString(b.Data) + case []byte: + return hex.EncodeToString(b) + } + return "" +} + +func bsonInt(v any) (int, bool) { + switch n := v.(type) { + case int32: + return int(n), true + case int64: + return int(n), true + case float64: + return int(n), true + } + return 0, false +} + +// resolveReleaseBaseline turns --against into the path of the deployed model: +// a .mpr on disk as given, or a git revision of the project, exported to a +// temporary directory. cleanup is never nil. +func resolveReleaseBaseline(projectPath, against string) (mprPath, label string, cleanup func(), err error) { + cleanup = func() {} + if fi, statErr := os.Stat(against); statErr == nil && !fi.IsDir() { + return against, against, cleanup, nil + } + if strings.HasSuffix(strings.ToLower(against), ".mpr") { + return "", "", cleanup, fmt.Errorf("--against %s: no such file", against) + } + abs, err := filepath.Abs(projectPath) + if err != nil { + return "", "", cleanup, err + } + dir := filepath.Dir(abs) + top, err := gitOutput(dir, "rev-parse", "--show-toplevel") + if err != nil { + return "", "", cleanup, fmt.Errorf("--against %s is neither a .mpr file nor a git revision of %s: %v", against, dir, err) + } + top = strings.TrimSpace(top) + if real, err := filepath.EvalSymlinks(dir); err == nil { + dir = real + } + if real, err := filepath.EvalSymlinks(top); err == nil { + top = real + } + rel, err := filepath.Rel(top, dir) + if err != nil { + return "", "", cleanup, err + } + commit, err := gitOutput(top, "rev-parse", "--verify", "--quiet", against+"^{commit}") + if err != nil { + return "", "", cleanup, fmt.Errorf("--against %s: not a .mpr file, and not a revision in %s", against, top) + } + commit = strings.TrimSpace(commit) + mprRel := filepath.ToSlash(filepath.Join(rel, filepath.Base(abs))) + paths := []string{mprRel} + contents := filepath.ToSlash(filepath.Join(rel, "mprcontents")) + if out, err := gitOutput(top, "ls-tree", "-d", "--name-only", commit, contents); err == nil && strings.TrimSpace(out) != "" { + paths = append(paths, contents) + } + if out, err := gitOutput(top, "ls-tree", "--name-only", commit, mprRel); err != nil || strings.TrimSpace(out) == "" { + return "", "", cleanup, fmt.Errorf("--against %s: %s is not in that revision", against, mprRel) + } + tmp, err := os.MkdirTemp("", "mxcli-release-*") + if err != nil { + return "", "", cleanup, err + } + cleanup = func() { os.RemoveAll(tmp) } + cmd := exec.Command("git", append([]string{"-C", top, "archive", "--format=tar", commit, "--"}, paths...)...) + var out, errOut bytes.Buffer + cmd.Stdout, cmd.Stderr = &out, &errOut + if err := cmd.Run(); err != nil { + cleanup() + return "", "", func() {}, fmt.Errorf("git archive %s: %v: %s", against, err, strings.TrimSpace(errOut.String())) + } + if err := untar(&out, tmp); err != nil { + cleanup() + return "", "", func() {}, err + } + short := commit + if len(short) > 12 { + short = short[:12] + } + return filepath.Join(tmp, filepath.FromSlash(mprRel)), fmt.Sprintf("git %s (%s)", against, short), cleanup, nil +} + +func gitOutput(dir string, args ...string) (string, error) { + cmd := exec.Command("git", append([]string{"-C", dir}, args...)...) + var out, errOut bytes.Buffer + cmd.Stdout, cmd.Stderr = &out, &errOut + if err := cmd.Run(); err != nil { + return "", fmt.Errorf("%v: %s", err, strings.TrimSpace(errOut.String())) + } + return out.String(), nil +} + +// untar writes a git archive into dir. Paths are checked to stay inside it. +func untar(r io.Reader, dir string) error { + tr := tar.NewReader(r) + for { + h, err := tr.Next() + if err == io.EOF { + return nil + } + if err != nil { + return err + } + target := filepath.Join(dir, filepath.FromSlash(h.Name)) + if !strings.HasPrefix(target, filepath.Clean(dir)+string(os.PathSeparator)) { + return fmt.Errorf("git archive entry %q leaves the target directory", h.Name) + } + switch h.Typeflag { + case tar.TypeDir: + if err := os.MkdirAll(target, 0o755); err != nil { + return err + } + case tar.TypeReg: + if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil { + return err + } + f, err := os.OpenFile(target, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o644) + if err != nil { + return err + } + if _, err := io.Copy(f, tr); err != nil { + f.Close() + return err + } + if err := f.Close(); err != nil { + return err + } + } + } +} diff --git a/cmd/mxcli/release_model_test.go b/cmd/mxcli/release_model_test.go new file mode 100644 index 000000000..b44406c68 --- /dev/null +++ b/cmd/mxcli/release_model_test.go @@ -0,0 +1,108 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/cmd/mxcli/projectreport" +) + +// The loader reads GUIDs from a Studio Pro-authored model, where a GUID is not +// the element's $ID: only such a model can show a re-minted GUID (CLAUDE.md, +// "Before trusting any GUID test, check the subject"). +func TestLoadReleaseModelReadsStudioProGUIDs(t *testing.T) { + m, err := loadReleaseModel(filepath.Join("..", "..", "testdata", "pedapp", "PedApp.mpr")) + if err != nil { + t.Fatal(err) + } + var account *projectreport.RelEntity + stored := 0 + for i, e := range m.Entities { + if e.Stored { + stored++ + } + if e.Name == "Administration.Account" { + account = &m.Entities[i] + } + } + if account == nil || !account.Stored || account.Generalization != "System.User" || !account.Marketplace { + t.Fatalf("Administration.Account is stored through System.User: %+v", account) + } + if account.GUID == "" || account.GUID == account.ID { + t.Fatalf("a Studio Pro entity's GUID is its own, not its $ID: guid %q id %q", account.GUID, account.ID) + } + for _, a := range account.Attributes { + if a.GUID == "" || a.Type == "" { + t.Errorf("attribute %s: guid %q type %q", a.Name, a.GUID, a.Type) + } + } + // show entities lists one persistent entity in this app. + if stored != 1 { + t.Errorf("stored entities = %d, want 1", stored) + } +} + +func git(t *testing.T, dir string, args ...string) { + t.Helper() + cmd := exec.Command("git", append([]string{"-C", dir, "-c", "user.email=t@example.com", "-c", "user.name=t"}, args...)...) + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("git %v: %v\n%s", args, err, out) + } +} + +// --against takes a git revision of the project, in a subdirectory of the +// repository, or a .mpr path; an unchanged model reports nothing either way. +func TestReleaseBaselineFromGitAndPath(t *testing.T) { + if _, err := exec.LookPath("git"); err != nil { + t.Skip("git not installed") + } + repo := t.TempDir() + app := filepath.Join(repo, "apps", "ped") + if err := copyTree(filepath.Join("..", "..", "testdata", "pedapp"), app); err != nil { + t.Fatal(err) + } + git(t, repo, "init", "-q") + git(t, repo, "add", "-A") + git(t, repo, "commit", "-qm", "v1") + git(t, repo, "tag", "v1") + mprPath := filepath.Join(app, "PedApp.mpr") + + base, label, cleanup, err := resolveReleaseBaseline(mprPath, "v1") + defer cleanup() + if err != nil { + t.Fatal(err) + } + if !strings.HasPrefix(label, "git v1 (") || strings.HasPrefix(base, repo) { + t.Errorf("label %q, baseline %q: a revision is exported to a temporary directory", label, base) + } + deployed, err := loadReleaseModel(base) + if err != nil { + t.Fatal(err) + } + next, err := loadReleaseModel(mprPath) + if err != nil { + t.Fatal(err) + } + if d := projectreport.CompareReleases(deployed, next); len(d.Changes) != 0 || d.Tables != 1 { + t.Errorf("unchanged: tables %d, changes %+v", d.Tables, d.Changes) + } + cleanup() + if _, err := os.Stat(base); !os.IsNotExist(err) { + t.Errorf("the export is removed by cleanup: %v", err) + } + + if p, _, _, err := resolveReleaseBaseline(mprPath, mprPath); err != nil || p != mprPath { + t.Errorf("a .mpr path is used as given: %q %v", p, err) + } + if _, _, _, err := resolveReleaseBaseline(mprPath, "no-such-tag"); err == nil || !strings.Contains(err.Error(), "not a revision") { + t.Errorf("an unknown revision is an error naming it: %v", err) + } + if _, _, _, err := resolveReleaseBaseline(mprPath, "/nowhere/App.mpr"); err == nil || !strings.Contains(err.Error(), "no such file") { + t.Errorf("a missing .mpr is an error: %v", err) + } +} diff --git a/docs-site/src/tools/project-reports.md b/docs-site/src/tools/project-reports.md index 7a4b86499..378631c7d 100644 --- a/docs-site/src/tools/project-reports.md +++ b/docs-site/src/tools/project-reports.md @@ -12,6 +12,7 @@ the kinds below answer specific questions instead of scoring. | `performance` | Where does time go in the main journeys at production volume, and did a fix help? | | `traceability` | Is each requirement in the specification built, and what proves it? | | `map` | What does the app contain, and how does it fit together? (an interactive guide) | +| `release` | What does deploying this model do to the data already in production? | ## Shared shape @@ -171,10 +172,37 @@ Entry points include two kinds of reference the catalog records for this: `by_na the flow a call placed in it). Both also keep such flows off the architecture report's unused list. +## release + +Compares the model with the one that is deployed and lists, for every persistent +entity, attribute and association, what the deploy does to stored data: **destroys** +(a table or column dropped), **verify** (handled in a way that has not been measured), +**migrates** (renamed under the same GUID, rows kept) and **adds**. + +```bash +mxcli report release -p app.mpr --against v1.4.0 -o release.html +mxcli report release -p app.mpr --against /backups/deployed/App.mpr --format json -o release.json +``` + +The runtime keys the database on each element's `GUID`, not its name, so the models +are paired by GUID. A model can be valid, pass `mx check` and build, and still drop a +table on the next deploy: an entity removed, made non-persistent, or given a new GUID +under its old name; an attribute removed, made calculated, or given a new GUID (its +column is recreated and every row gets the model default). A rename under the same +GUID keeps the data. Type changes, shorter strings, changed association types or +ends and changed generalizations are listed as **verify**: what the runtime does +with them has not been measured. + +`--against` is a git revision of the project (a tag, a branch, a commit; the project +may sit in a subdirectory of the repository) or the path of a `.mpr`. A drop that is +intended is annotated `release::` with status `accepted`, and the +verdict then counts it apart. + ## Skills Each kind has a skill with the judgement the command cannot make and the traps found building the reports by hand: `report-architecture`, `report-tests`, -`report-security`, `report-performance`, `report-traceability`, `report-map`. +`report-security`, `report-performance`, `report-traceability`, `report-map`, +`report-release`. The scored report has `report-quality`, and `assess-quality` combines all of them with the manual review no rule can do. From cfc20dc5011783c8bef2ce15a528deb88428a9a2 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 19 Nov 2026 11:51:22 +0000 Subject: [PATCH 06/66] feat(verify): make verify the default gate in generated CLAUDE.md, bootstrap and docs verify replaces the separate lint and docker check gates (it runs both against baselines). bootstrap-app records the baselines on the blank app. The default baseline directory is verify-baseline/ next to the project, not .mxcli/, which mxcli init git-ignores. Adds the mxcli verify docs page. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_011oK82stfDEB1TGfJLVwUxc --- .claude/skills/mendix/bootstrap-app/SKILL.md | 23 ++++---- cmd/mxcli/cmd_verify.go | 15 ++++- cmd/mxcli/cmd_verify_test.go | 20 +++++++ cmd/mxcli/init_claudemd.go | 9 +-- docs-site/src/SUMMARY.md | 1 + docs-site/src/tools/bootstrap-prompt.md | 13 ++-- docs-site/src/tools/mxcli-verify.md | 62 ++++++++++++++++++++ docs/11-proposals/PROPOSAL_verify_gate.md | 35 +++++++---- 8 files changed, 141 insertions(+), 37 deletions(-) create mode 100644 docs-site/src/tools/mxcli-verify.md diff --git a/.claude/skills/mendix/bootstrap-app/SKILL.md b/.claude/skills/mendix/bootstrap-app/SKILL.md index aad333a40..658d1bf62 100644 --- a/.claude/skills/mendix/bootstrap-app/SKILL.md +++ b/.claude/skills/mendix/bootstrap-app/SKILL.md @@ -12,10 +12,10 @@ description: "Provision a Mendix project in a repo that has none — interview, |---|---| | 23-43 | When to Use This Skill | | 44-134 | Step 0 — interview, and WAIT for the answers | -| 135-268 | Provision | -| 269-338 | If this is a solution (several apps in one repo) | -| 339-398 | Then propose the model — do not build it yet | -| 399-462 | After bootstrap — the inner loop | +| 135-270 | Provision | +| 271-340 | If this is a solution (several apps in one repo) | +| 341-400 | Then propose the model — do not build it yet | +| 401-463 | After bootstrap — the inner loop | # Bootstrap a Mendix App in an Empty Repo @@ -218,16 +218,18 @@ drop the `./` if it came pre-installed on `PATH`. 8. **Boot and verify:** `./mxcli run --local --watch --detach -p .mpr` returns once the app answers (or prints why the boot failed, exit 1) — report its one line. Manage it with `./mxcli run status|wait|stop`, never `nohup`, poll loops or `pkill`. -9. **Take the quality baseline** — run the two gates that score the project, on the - blank app, before any of your own work is in it: +9. **Take the quality baseline** — on the blank app, before any of your own work is + in it, record what the template ships with and score it: ```bash - ./mxcli lint -p .mpr + ./mxcli verify -p .mpr --update-baseline # lint + mx check baselines ./mxcli report -p .mpr --format markdown ``` - Report the warning count and the six category scores, and put them in - `FINDINGS.md` with the date. This is the only moment the numbers mean "what the + `verify --update-baseline` writes `verify-baseline/` next to the `.mpr`. **Commit + it** — it is what every later `verify` compares against, and a fresh clone or a + reaped session without it reports every finding as new. Report the warning count + and the six category scores, and put them in `FINDINGS.md` with the date. This is the only moment the numbers mean "what the template ships with" — afterwards every figure is yours plus the template's, and there is nothing to subtract. A blank app is **not** expected to score zero warnings; knowing which ones it starts with is what stops you chasing them later. @@ -417,9 +419,8 @@ right, then run the gates once over the result. The whole list after every edit ```bash ./mxcli check change.mdl -p .mpr --references # syntax + references (~2s) ./mxcli exec change.mdl -p .mpr # apply -./mxcli lint -p .mpr # rules (~3s) +./mxcli verify -p .mpr # lint + mx check, new findings only (~10s) ./mxcli report -p .mpr # scored quality report -./mxcli docker check -p .mpr # mxbuild, the slow one (~25s) ./mxcli test tests/ -p .mpr --local # microflow tests (~30s cold, ~2s warm) ./mxcli run --local --watch --detach -p .mpr # the app, in the background; then `run wait` ``` diff --git a/cmd/mxcli/cmd_verify.go b/cmd/mxcli/cmd_verify.go index e5880bc17..b7bc76a2f 100644 --- a/cmd/mxcli/cmd_verify.go +++ b/cmd/mxcli/cmd_verify.go @@ -34,7 +34,7 @@ Every selected stage runs, so one result carries every stage's findings; use mxcli check cannot yet predict everything it rejects, so a project that passes check and lint can still fail mxcheck. -Baselines live in --baseline-dir (default .mxcli/verify next to the project): +Baselines live in --baseline-dir (default verify-baseline/ next to the project): lint.json is a ` + "`mxcli lint --baseline`" + ` file, mxcheck.json records mx check's findings by their element IDs. Without one, every finding is reported as new and the stage says so. Record the current state with --update-baseline (commit the @@ -72,7 +72,7 @@ func init() { verifyCmd.Flags().Bool("fail-fast", false, "Stop at the first stage that fails instead of running every stage") verifyCmd.Flags().String("fail-on", "error", "Lowest severity of a new finding that fails the gate: error or warning") verifyCmd.Flags().Bool("strict", false, "Also fail on unmasked mx check findings (errors an earlier baseline error was hiding)") - verifyCmd.Flags().String("baseline-dir", "", "Where the baselines live (default .mxcli/verify next to the project)") + verifyCmd.Flags().String("baseline-dir", "", "Where the baselines live (default verify-baseline/ next to the project)") verifyCmd.Flags().Bool("update-baseline", false, "Record the current lint and mx check findings as the baseline instead of comparing") verifyCmd.Flags().String("mxbuild-path", "", "Path to MxBuild/Mendix installation (used to find mx)") verifyCmd.Flags().StringP("format", "f", "text", "Output format: text, json") @@ -108,7 +108,7 @@ func runVerify(cmd *cobra.Command, scripts []string) int { return usage("%v", err) } if baselineDir == "" { - baselineDir = filepath.Join(filepath.Dir(projectPath), ".mxcli", "verify") + baselineDir = defaultVerifyBaselineDir(projectPath) } skip := map[string]string{} @@ -138,6 +138,15 @@ func runVerify(cmd *cobra.Command, scripts []string) int { return res.Exit } +// defaultVerifyBaselineDir is where the baselines live unless --baseline-dir says +// otherwise. It is deliberately NOT under .mxcli/: `mxcli init` git-ignores that +// directory (it holds machine-local state such as constants), and a baseline that +// is not committed is gone with the next fresh clone or reaped session, after +// which every run reports every finding as new. +func defaultVerifyBaselineDir(projectPath string) string { + return filepath.Join(filepath.Dir(projectPath), "verify-baseline") +} + type verifyRunner struct { self, project, baselineDir, mxbuildPath string update bool diff --git a/cmd/mxcli/cmd_verify_test.go b/cmd/mxcli/cmd_verify_test.go index 3534af53f..04bca9e5e 100644 --- a/cmd/mxcli/cmd_verify_test.go +++ b/cmd/mxcli/cmd_verify_test.go @@ -137,3 +137,23 @@ func TestVerifyOutputContract(t *testing.T) { } } } + +// The baselines have to be committed to be any use: a fresh clone, or a reaped +// cloud session, starts with none, and every run then reports every finding as +// new. So the default location must not be one the generated .gitignore hides. +func TestDefaultVerifyBaselineDirIsNotGitIgnored(t *testing.T) { + dir := defaultVerifyBaselineDir(filepath.Join("proj", "App.mpr")) + if got := filepath.Base(dir); got != "verify-baseline" || filepath.Base(filepath.Dir(dir)) != "proj" { + t.Fatalf("default dir = %q", dir) + } + for _, line := range strings.Split(mendixGitignore, "\n") { + line = strings.TrimSpace(line) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + pattern := strings.Trim(line, "/") + if pattern == "verify-baseline" || pattern == "*" || strings.HasPrefix(pattern, "verify") { + t.Errorf(".gitignore pattern %q hides the default baseline directory", line) + } + } +} diff --git a/cmd/mxcli/init_claudemd.go b/cmd/mxcli/init_claudemd.go index 7a536d07d..070553a2f 100644 --- a/cmd/mxcli/init_claudemd.go +++ b/cmd/mxcli/init_claudemd.go @@ -105,9 +105,10 @@ type projectGate struct { var projectGates = []projectGate{ {"check script.mdl -p %s", "syntax + references, no apply (~2s)", "mxcli check", "check"}, {"exec script.mdl -p %s", "apply", "mxcli exec", "exec"}, - {"lint -p %s", "rules (~3s)", "mxcli lint", "lint"}, + // verify is lint and mx check in one run, against the baselines recorded at + // bootstrap, so it reports what the change made worse and not the backlog. + {"verify -p %s", "lint + mx check, new findings only (~10s)", "mxcli verify", "verify"}, {"report -p %s", "scored quality report", "mxcli report", "report"}, - {"docker check -p %s", "mxbuild, the slow one (~25s)", "mxcli docker check", "docker check"}, {"test tests/ -p %s --local", "microflow tests (~30s cold, ~2s warm)", "mxcli test", "test"}, // --detach because an agent's tool call must return: without it the run was // started with nohup and then managed by sleep/grep/pkill loops, a quarter @@ -244,8 +245,8 @@ func generateClaudeMD(projectName, mprFile string) string { w(bt3 + "bash\n") w(renderProjectGates(mprPath)) w(bt3 + "\n\n") - w("**" + bt + "lint" + bt + " printing no errors is not a pass** — read the warning count, and read\n") - w(bt + "report" + bt + "'s score. A green " + bt + "check" + bt + " proves nothing about behaviour or about how\n") + w("**" + bt + "verify" + bt + " reports what is new, not what is good** — read the pre-existing count it\n") + w("prints and " + bt + "report" + bt + "'s score. A green " + bt + "check" + bt + " proves nothing about behaviour or about how\n") w("a page renders: logic needs " + bt + "test" + bt + ", and anything visual needs the app actually running.\n\n") w("Set " + bt + "mx" + bt + " up once with " + bt + "./mxcli setup mxbuild -p " + mprPath + bt + ". To call it directly,\n") w("name the version — " + bt + "~/.mxcli/mxbuild//modeler/mx" + bt + " — because a " + bt + "*" + bt + " glob\n") diff --git a/docs-site/src/SUMMARY.md b/docs-site/src/SUMMARY.md index 43dcf1678..2b2a68935 100644 --- a/docs-site/src/SUMMARY.md +++ b/docs-site/src/SUMMARY.md @@ -141,6 +141,7 @@ - [Starlark Rules](tools/starlark-rules.md) - [Writing Custom Rules](tools/custom-rules.md) - [mxcli lint](tools/mxcli-lint.md) + - [mxcli verify](tools/mxcli-verify.md) - [mxcli report](tools/mxcli-report.md) - [Project Reports](tools/project-reports.md) - [Claude Code Companion](tools/companion.md) diff --git a/docs-site/src/tools/bootstrap-prompt.md b/docs-site/src/tools/bootstrap-prompt.md index 6523b5762..ce3bf60d3 100644 --- a/docs-site/src/tools/bootstrap-prompt.md +++ b/docs-site/src/tools/bootstrap-prompt.md @@ -75,8 +75,9 @@ a longer prompt. is no status column to maintain. 5. **Commits, then boots and verifies** — HTTP 200 at `http://localhost:8080/`, plus an optional `run --hub` preview URL. -6. **Takes the quality baseline** — `mxcli lint` and the scored `mxcli report` on the - blank app, recorded in `FINDINGS.md`. This is the only moment those numbers mean +6. **Takes the quality baseline** — `mxcli verify --update-baseline` (the lint and + `mx check` baselines, committed as `verify-baseline/`) and the scored + `mxcli report` on the blank app, recorded in `FINDINGS.md`. This is the only moment those numbers mean "what the template ships with"; afterwards every figure is yours plus the template's, with nothing to subtract. See [mxcli report](mxcli-report.md) for how to read the six category scores. @@ -201,15 +202,15 @@ time, and was the dominant cost in a measured agent session. ```bash ./mxcli check change.mdl -p .mpr --references # syntax + references (~2s) ./mxcli exec change.mdl -p .mpr # apply -./mxcli lint -p .mpr # rules (~3s) +./mxcli verify -p .mpr # lint + mx check, new findings only (~10s) ./mxcli report -p .mpr # scored quality report -./mxcli docker check -p .mpr # mxbuild, the slow one (~25s) ./mxcli test tests/ -p .mpr --local # microflow tests (~30s cold, ~2s warm) ./mxcli run --local --watch --detach -p .mpr # the app, in the background; then `run wait` ``` -Each buys something the one above it cannot: [`lint`](mxcli-lint.md) and -[`report`](mxcli-report.md) score the model that `check` only proved was well-formed, +Each buys something the one above it cannot: [`verify`](mxcli-verify.md) runs +[`lint`](mxcli-lint.md) and `mx check` against the baselines taken at bootstrap and reports +only what the change made worse, [`report`](mxcli-report.md) scores the model that `check` only proved was well-formed, [`test`](testing.md) proves behaviour that a clean build does not, and the running app is the only thing that proves how a page renders. `report`'s six category scores are comparable against the baseline taken at bootstrap — a score that fell is a finding. diff --git a/docs-site/src/tools/mxcli-verify.md b/docs-site/src/tools/mxcli-verify.md new file mode 100644 index 000000000..e0445bb16 --- /dev/null +++ b/docs-site/src/tools/mxcli-verify.md @@ -0,0 +1,62 @@ +# mxcli verify + +`mxcli verify` is the one command that answers "did my change break anything?". It runs the +checks in cost order and reports only the findings that are **new** against a baseline, so on +a mature project the result is about the change just made, not the backlog. + +| Stage | What it runs | Against | +|---|---|---| +| `check` | `mxcli check` on the script files you pass | nothing: every finding is the script's | +| `lint` | `mxcli lint` on the project | `verify-baseline/lint.json` | +| `mxcheck` | `mx check` on a copy of the project | `verify-baseline/mxcheck.json` | + +`mx check` is the authority. `mxcli check` cannot yet predict everything it rejects, so a project +that passes `check` and `lint` can still fail `mxcheck`; that is why the stage is part of the gate. + +## Usage + +```bash +# Record the current state as the baseline (do this once, on a project you accept as it is) +mxcli verify -p app.mpr --update-baseline + +# After a change +mxcli verify -p app.mpr +mxcli verify -p app.mpr change.mdl # also checks the script +mxcli verify -p app.mpr --stages lint --format json +``` + +Commit `verify-baseline/`. It lives next to the `.mpr`, not under `.mxcli/` (which `mxcli init` +git-ignores): a baseline that is not committed is gone with the next fresh clone, and every run +then reports every finding as new. Without a baseline a stage reports every finding as new and +says so. + +Every selected stage runs, so one result carries every stage's findings. `--fail-fast` stops at +the first stage that fails. `--fail-on warning` makes a new warning fail a stage (the default is +errors). `--format json` prints one document with each stage's status, duration and new findings. + +## Exit codes + +| Code | Meaning | +|---|---| +| 0 | nothing new | +| 1 | usage error | +| 2 | a stage could not run (no `mx`, ...) and nothing failed | +| 16 + bits | a stage failed: 1 `check`, 2 `lint`, 4 `mxcheck` (18 = lint, 22 = lint and mxcheck) | + +A stage that could not run is never reported as passed. + +## Why mx check needs special handling + +`mx check` reports errors in stages: an earlier-stage error hides the later ones. A plain diff +would blame a change for an error it only uncovered, or call an error resolved because a new +earlier one hides it. So `mxcheck` sorts findings into: + +- **new** — not in the baseline. This includes a later error uncovered in a microflow whose + earlier baseline error is gone, because the change edited that flow and must leave it clean. +- **unmasked** — not in the baseline, but a baseline unresolved-reference error was hiding the + whole project. Reported, and does not fail the gate unless `--strict`. +- **resolved** — in the baseline, gone, and nothing hides it. +- **hidden** — in the baseline, gone only because a new earlier-stage error hides it. Not resolved. + +See the [proposal](https://github.com/mendixlabs/mxcli/blob/main/docs/11-proposals/PROPOSAL_verify_gate.md) +for the measurements behind this. diff --git a/docs/11-proposals/PROPOSAL_verify_gate.md b/docs/11-proposals/PROPOSAL_verify_gate.md index 23d776845..33ad29c5c 100644 --- a/docs/11-proposals/PROPOSAL_verify_gate.md +++ b/docs/11-proposals/PROPOSAL_verify_gate.md @@ -328,7 +328,7 @@ type Stage interface { 2. **`mxcli verify`, first release** (implemented): stages `check`, `lint`, `mxcheck`; `--format text|json`, the exit-code table, `--fail-fast`, `--fail-on`, `--strict`; and a **`lint` baseline from day one**. Baselines live - in `--baseline-dir` (default `.mxcli/verify` next to the project): `lint.json` + in `--baseline-dir` (default `verify-baseline/` next to the project, not `.mxcli/`, which `mxcli init` git-ignores: a baseline that is not committed is gone with the next fresh clone): `lint.json` is a plain `mxcli lint --baseline` file (so existing ones stay valid) and `mxcheck.json` records mx check's findings by element ID, with the checker and mx version, which a run must match. `lint` and `check` run as subprocesses of @@ -336,16 +336,20 @@ type Stage interface { refactored. Checked on real MxBuild 11.16.1: a new defect fails (exit 20), an accepted one does not, a stage 0 fix is unmasked, a same-flow uncovered error is new. -3. **Make `verify` the thing everything points at**: skills, `mxcli init` output - and the project `CLAUDE.md` guidance name `verify` as the single "did my change - break anything?" command. **Not in the first release, and a decision is - needed:** `projectGates` is the definition of done in every generated - `CLAUDE.md`. Adding `verify` as a seventh gate runs lint and `mx check` twice - and puts the file 77 bytes over its test-enforced budget; the alternative is - for `verify` to *replace* the `lint` and `docker check` gates, which changes - every project's default and needs `bootstrap-app` to record the baselines at - provisioning. Either way the gate tests (`init_claudemd_gates_test.go`) already - keep the three descriptions in step. +3. **Make `verify` the thing everything points at** (implemented). `projectGates`, + the definition of done in every generated `CLAUDE.md`, now lists `verify` **in + place of** the separate `lint` and `docker check` gates, which it runs. Adding it + as a seventh gate would have run both twice and put the file over its + test-enforced byte budget. `report` stays: `verify` reports what is new, not how + good the project is. The bootstrap skill (step 9) records the baselines with + `verify --update-baseline` on the blank app, next to the existing quality + baseline, and says to commit `verify-baseline/`; the docs page and a new + `mxcli verify` page say the same, and `init_claudemd_gates_test.go` keeps the + three descriptions in step. The generated `CLAUDE.md` does not explain how to + record a baseline: a stage without one says so in its own output. The default + baseline directory moved from `.mxcli/verify` to `verify-baseline/` because + `mxcli init` git-ignores `.mxcli/`; an ignored baseline is lost on the next fresh + clone or reaped session. 4. **`--base `** via scratch export. 5. **Opt-in `build`, `test`, `smoke` stages.** `smoke` is also the only stage that can reach the runtime-only findings listed under Evidence. @@ -391,5 +395,10 @@ type Stage interface { 9. **`mendix check` pre-stage:** worth a flag now, or defer until its false-positive rate is measured on real projects? 10. **Default stage set** — `mxcheck` costs ~8 s once per run and is on by default. -11. **Step 3:** add `verify` as a seventh gate, or replace `lint` and `docker check` - with it (see Phasing)? +11. ~~Step 3: add `verify` as a seventh gate, or replace `lint` and `docker check`?~~ + **Settled:** it replaces them (see Phasing). +12. **Existing projects.** A project initialised before this change has the old gate + list in its `CLAUDE.md` and no `verify-baseline/`. Does `mxcli init` re-run (or + a new command) offer to update the list, and should the first `verify` on a + project with no baseline refuse with a pointer instead of reporting everything + as new? From 4e7d26d325145c904b61aff3cab3b1863201c7d5 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 19 Nov 2026 14:30:43 +0000 Subject: [PATCH 07/66] feat(lint): MPR018 reports a model element named inside a string literal Closes ako/mxcli#2405. A flow that passes 'Module.SomeFlow' as text leaves no reference for any checker to follow, so renaming or moving that element passes Studio Pro, mxbuild and `mxcli check --references`, and the app fails at runtime. From the benchmark that raised the issue: a Java action received the text "Simulation.SIM_Tick", and renaming the microflow stopped the simulation with an ERROR in the server log every second. mxcli has a particular stake in it. It makes renaming cheap and scriptable (`rename microflow`, `rename entity`), and `check --references` is sold as the thing that catches a stale reference; a string literal is the one case where that promise silently does not hold. Two verdicts: a literal that RESOLVES to a real element is a warning, the rename trap; a literal whose module exists but whose name resolves to nothing is info, because a string may legitimately look like a qualified name. Two narrowings carry the precision, each the inverse of the obvious choice: - String LITERALS, not whole expressions. An expression's own references are dotted too -- an association path $Var/Mod.Assoc/Attr, an enumeration value Mod.Enum.Value -- and those ARE real references Studio Pro resolves and renames. Scanning expressions would report every association path in the project. - Only the fields that hold a value: Expression, Argument, Value, matched by NAME through reflection, so a new action type is covered the day it is added. Reflection over every string field is comprehensive and wrong, because JavaActionCallAction.JavaAction, MicroflowCall.Microflow and JavaActionParameterMapping.Parameter all hold qualified names and are all genuine references: that version flags every call in the project. Missing an expression field costs a finding; missing a reference field invents one. One further test removes nearly all remaining noise: the dotted prefix must be a module the project actually has. That excludes prose, file paths, URLs, CSS selectors and Java packages (com.mendix.core.Core) without enumerating any. Measured - 7 tests, including that a Java action call's reference fields are NOT scanned while its argument IS, member values inside loops, and the doubled-quote escape ('it''s here'). - Positive control, run before polishing: a planted literal naming a real microflow is reported; ordinary prose, a Java class name and a 4-segment name are not; and after `rename microflow` the warning becomes the dangling info. - The PREMISE measured, not asserted: mxbuild reported 28 errors before the rename and 28 after, naming the renamed microflow in neither. (`mxcli check