diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-19-mapping-handler-member-type-ce0282.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-19-mapping-handler-member-type-ce0282.json new file mode 100644 index 0000000000..03b5b4fbf4 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-19-mapping-handler-member-type-ce0282.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-11-19", "refs": ["ako/mxcli#1258"], "ce": ["CE0282"], "symptom": "An import mapping's custom object handler `find … by M.MF ( P: member )` that fed a JSON member to a microflow parameter of an incompatible type (a String member into an Integer parameter) passed `check` and `exec` and was written; `mx check` reported CE0282 \"The microflow parameter 'P' is of type 'Integer'. It does not match the specified type 'String'\".", "file": "mdl/executor/validate_mapping_handler_types.go ValidateMappingHandlerTypes (MDL107), called from validate_program.go with the lazy project opener. Tests: validate_mapping_handler_types_test.go; fixtures mdl-examples/bug-tests/mapping-handler-param-type-mismatch.fail.mdl and the clean control mapping-handler-types.mdl", "cause": "buildCustomHandler binds each parameter to its member and never compares the member's JSON type with the parameter's declared type.", "insight": "**The compatibility matrix is small enough to measure whole, which made a name-based lookup safe where a path-following one was not.** 25 pairings of {String, Integer, Long, Decimal, Boolean, DateTime} x {string, integer, decimal, boolean, date-time} members, one project each on MxBuild 11.16.1: a String parameter takes everything, Decimal also takes Integer, Long takes Integer, the rest only their own type. The rule finds the member by its NAME in the whole structure and judges only when every occurrence is one known value type, so it never has to follow the handler's element path through arrays and two-level items (the duplication that made a CE0281 mirror a drift risk); a name that is absent or typed differently in two places is left to the build. The cost is silence on that ambiguous case, which is the right direction to be wrong in."} diff --git a/.claude/skills/fix-issue/findings/mdl-other/2026-11-19-memory-catalog-per-connection-database.json b/.claude/skills/fix-issue/findings/mdl-other/2026-11-19-memory-catalog-per-connection-database.json new file mode 100644 index 0000000000..bb272eeb3c --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-other/2026-11-19-memory-catalog-per-connection-database.json @@ -0,0 +1 @@ +{"area": "mdl/catalog", "date": "2026-11-19", "symptom": "On a run where lint BUILDS the catalog, one iterator failed while others succeeded on the same handle: `lint could not read the catalog for AttributesFor: SQL logic error: no such table: attributes` with ctx.Entities() returning rows in the same run, and the next identical run (cached catalog) correct. Reproduced on testdata/testapp-views and not on testdata/pedapp. Also: a fresh catalog's snapshot ObjectCount was counted as 0.", "cause": "catalog.New() opened `sql.Open(\"sqlite\", \":memory:\")`. Every CONNECTION to \":memory:\" is its own separate, empty database, and sql.Open returns a POOL — so createTables' schema landed on one connection and whatever query the pool later served on another saw nothing. Which iterator broke was pool timing, hence the per-project difference. Fixed with a uniquely named shared-cache DSN (`file:mxcli-catalog--?mode=memory&cache=shared&_pragma=foreign_keys(1)`) plus one pinned connection so the shared cache is not freed. Two rejected fixes mattered: db.SetMaxOpenConns(1) gives one database but DEADLOCKS, because the readers nest (ctx.Entities() yields from an open cursor while the body calls ctx.AttributesFor()) and because Builder.updateSnapshotCount queried the POOL while the build transaction held a connection; shared cache alone still hung on that same builder read until it was routed through b.txQuery.", "file": "`mdl/catalog/catalogdb_sqlite.go` (NewSqliteCatalogDB, Close, NewSqliteCatalogDBFromFile); `mdl/catalog/builder.go` (updateSnapshotCount); tests `mdl/catalog/catalogdb_memory_pool_2417_test.go`", "insight": "\":memory:\" plus database/sql is never one database — it is one per connection, so any pooled access is a coin flip, and the failure presents as a MISSING TABLE on a schema that provably exists (sqlite_master listed all 88 views in the failing run). That signature — one table unreadable while its neighbours read fine, same handle, same run — means separate databases, not a broken view; stop comparing view DDL and look at the DSN. The per-project reproduction (one project yes, one no) is the tell that it is pool timing rather than data. Pinning the pool is the obvious fix and is wrong here: a goroutine dump taken with kill -QUIT named the exact blocker (Builder.updateSnapshotCount -> QueryRow -> database/sql.(*DB).conn), and it only showed up because the hang was investigated instead of the pin reverted. Reading a count on the pool while the build transaction is open was also returning 0, because a pool query cannot see the transaction's uncommitted rows: the deadlock and the wrong count were one bug. Finally, a claim about CI behaviour needs the exit code, not the printed text: lint already prints \"Results are INCOMPLETE\" and exits 1 on any recorded query error (cmd_lint.go), so the \"CI sees a pass\" worry in the issue was unfounded."} diff --git a/.claude/skills/fix-issue/findings/other/2026-11-19-dotted-object-name-dropped-silently.json b/.claude/skills/fix-issue/findings/other/2026-11-19-dotted-object-name-dropped-silently.json new file mode 100644 index 0000000000..2010f04973 --- /dev/null +++ b/.claude/skills/fix-issue/findings/other/2026-11-19-dotted-object-name-dropped-silently.json @@ -0,0 +1 @@ +{"area":"sql","date":"2026-11-19","symptom":"`SQL GENERATE CONNECTOR INTO ` silently omitted every object whose own NAME contains a dot. On a DuckGraphLake lake the raw topic tables are named after the Kafka topic (`main.\"local__cdc.CdcPublish.Customer.1.local\"`), so 4 of the 9 listed objects were missing from the output, with no warning and no error -- the only warnings printed were about unmappable columns (ako/mxcli#2422).","cause":"the address was flattened to one string by qualifiedTableName and split again by splitTableName, which took the last two dot-separated parts: schema `1`, name `local`. DescribeTable then found nothing and readListed's `continue` discarded the error. Two more faults sat behind that one, each of which would have turned the silent drop into silently invalid output: TableSchema.QualifiedName joined the parts unquoted, so the emitted SQL would have been `FROM main.local__cdc.CdcPublish.Customer.1.local` (`syntax error at or near \".1.\"`), and toPascalCase treated only `_ - space` as separators, so TableToEntityName gave `LocalCdc.cdcpublish.customer.1.local` -- dots and all -- which is not a Mendix entity name.","file":"sql/meta.go (splitQualifiedIdent, quoteIdent/needsQuoting, splitTableName); sql/schema.go (QualifiedName, qualifiedTableName, readListed returns what it skipped); sql/naming.go (toPascalCase: `.` is a separator, in both the early return and the loop); sql/generate.go (GenerateResult.SkippedObjects); mdl/executor/cmd_sql.go (reports them); test sql/generate_dotted_names_2422_test.go","insight":"Fixing the addressing ALONE would have been worse than the bug: the object would have been generated with an entity name containing dots and a FROM clause that is a syntax error, where before it was merely absent. Check what the name becomes, not just whether the object is found -- `TableToEntityName` had a second, easily missed trap, an early return for names without `_` or `-` that only capitalises the first letter, so `my.table` became `My.table` and never reached the separator loop. Quote only what needs it (anything outside [A-Za-z0-9_], or a leading digit): quoting every identifier would change every query the tool has ever generated, and a bare reserved word needs no quoting once it is schema-qualified, which ako/mxcli#2414 measured. Three one-line probes settle the whole diagnosis on any such database: `select count(*) from information_schema.columns where table_schema='main' and table_name=''` (7 -- the metadata side is fine), the same select quoted (5 rows), and unquoted (syntax error). Four controls, each failing a different test: split on every dot -> SplitTableName_QuotedNameKeepsItsDots; quoteIdent never quotes -> QualifiedName_QuotesANameThatNeedsIt; `.` not a separator -> TableToEntityName_DotsAreWordSeparators; skipped list dropped -> UnreadableObjectIsReported. Live result: 5 of 9 objects generated before, 9 of 9 after, all 9 queries running.","refs":["ako/mxcli#2422","ako/mxcli#2414","ako/mxcli#2415"]} diff --git a/.claude/skills/fix-issue/findings/other/2026-11-19-views-clause-generated-every-table-too.json b/.claude/skills/fix-issue/findings/other/2026-11-19-views-clause-generated-every-table-too.json new file mode 100644 index 0000000000..aac978d562 --- /dev/null +++ b/.claude/skills/fix-issue/findings/other/2026-11-19-views-clause-generated-every-table-too.json @@ -0,0 +1 @@ +{"area":"sql","date":"2026-11-19","symptom":"`SQL GENERATE CONNECTOR INTO VIEWS (customer)` generated every table as well and then the named view on top, so one connector held two entities and two queries called LakeSql.Customer / GetAllCustomers -- not a model Mendix can load. Measured on a DuckGraphLake lake (all objects are views): `views (customer)` gave 6 queries with one duplicate, `views (customer, order)` 7 with two, `tables (customer) views (customer)` 2 with one. `tables (customer)` alone was correct, which is what made it look like a views-only quirk (ako/mxcli#2415).","cause":"two things compounded in sql/generate.go: `cfg.Tables == nil` meant \"every table\" with nothing suppressing it when Views was set (there was no views-only mode), and the table reader ignored information_schema's table_type -- the third column ShowTables already returns -- so a view was in the tables set before the view pass added it again. A third made the dedupe impossible: ReadTableSchema recorded no schema for a name the user wrote bare, so `TABLES (customer)` and `VIEWS (cdcpublish.customer)` had different addresses and could not be recognised as one object. That last one is also why the ako/mxcli#2414 fix did not reach the explicit clauses: it qualified the discovery path only.","file":"sql/generate.go (clause scoping, dedupeByAddress); sql/schema.go (discoveredObjects, readListed, ReadAllSchemas, ReadAllTableSchemas now excludes views, ReadTableSchema resolves a bare name); sql/meta.go (schemasHolding, factored out of refuseAmbiguousTable); test sql/generate_clause_scope_2415_test.go","insight":"Do not filter views out of the table reader without giving the no-clause path somewhere else to get them: on a CDC lake every object is a view, so filtering alone turns `generate connector` into `no tables or views found`. Enumerating views from the driver's view listing is the wrong somewhere -- pg_views on the lake returns 23 rows, 14 of them internal temp views (sqlite_master, pragma_database_list), where information_schema.tables returns the 9 real ones. So read the one listing and partition it on table_type. Resolving a bare name needs no new query shape: refuseAmbiguousTable already asked which schemas hold a name, for all three drivers, and discards the answer when there is exactly one -- that answer is the schema the generated query needs. Four controls, each failing a different test: clause scoping reverted -> ViewsClauseDoesNotPullInEveryTable; dedupe removed -> ObjectInBothClausesEmittedOnce; table_type ignored -> NoClauseGeneratesEverythingOnce + both reader tests; bare-name resolution removed -> BareClauseNameIsQualified + ObjectInBothClausesEmittedOnce. Adjacent and NOT fixed here: splitTableName takes the last two dot-separated parts, so an object whose own name contains dots (`main.\"local__cdc.App.Entity.1.local\"`, which is what a Kafka topic table is called) resolves to schema `1`, name `local`, fails describe, and is dropped by the reader's `continue` with no warning -- four of the lake's nine objects vanish from the no-clause output, before and after this fix.","refs":["ako/mxcli#2415","ako/mxcli#2414"]} diff --git a/.claude/skills/mendix/assess-quality/SKILL.md b/.claude/skills/mendix/assess-quality/SKILL.md index 4b120f44fe..23d47b86a2 100644 --- a/.claude/skills/mendix/assess-quality/SKILL.md +++ b/.claude/skills/mendix/assess-quality/SKILL.md @@ -1,6 +1,6 @@ --- name: assess-quality -description: "Audit a Mendix project against best practices for naming, security, performance, maintainability and architecture, and report a scored result. Use when asked to evaluate project quality, run a pre-go-live health check, or get oriented in an unfamiliar app." +description: "Write a full quality assessment of a Mendix project: the mxcli reports for what can be measured, plus a manual review of what no rule can judge (naming that says what a flow does, folder structure, error-handling design, connector conventions, UI styling, complexity hotspots). Use when asked for a quality assessment, a pre-go-live health check or an audit, or to get oriented in an unfamiliar app." --- ## Find it fast @@ -10,108 +10,59 @@ description: "Audit a Mendix project against best practices for naming, security | lines | section | |---|---| -| 29-36 | When to Use This Skill | -| 37-96 | Assessment Workflow | -| 97-356 | Quality Guidelines Reference | -| 357-367 | Assessment Report Template | -| 368-380 | Executive Summary | -| 381-391 | project Metrics | -| 392-404 | Automated Findings | -| 405-424 | Manual Review Findings | -| 425-440 | Recommendations | -| 441-451 | Sources | +| 31-52 | Step 1: Run the reports | +| 53-207 | Step 2: Manual review | +| 208-214 | Step 3: Write the assessment | +| 215-217 | Summary | +| 218-229 | Measured | +| 230-232 | Reviewed by hand | +| 233-235 | Recommendations | +| 236-240 | Not assessed | +| 241-247 | Sources | # Assess Mendix Project Quality -This skill guides a comprehensive quality assessment of a Mendix project, evaluating it against established best practices across naming, security, performance, maintainability, architecture, and more. +An assessment has two halves. **What can be measured** comes from the mxcli reports — +run them, do not re-derive their findings by hand. **What needs judgement** is the +manual review below: whether names say what a flow does, whether the folder and module +structure make sense, whether error handling is designed or accidental. -## When to Use This Skill +## Step 1: Run the reports -Use this skill when: -- A user asks to evaluate the quality of a Mendix project -- Reviewing a project against best practices before go-live -- Performing a health check or audit on an existing application -- Onboarding onto a project and wanting to understand its quality posture +| Question | Report | Skill | +|---|---|---| +| How does it score against the lint rules? | `mxcli report` | `report-quality` | +| How is it layered; cycles, coupling, dead code? | `mxcli report architecture` | `report-architecture` | +| Who can read what; which flows trust the caller? | `mxcli report security` | `report-security` | +| Do the tests pass, and what do they reach? | `mxcli report tests` | `report-tests` | +| Is it fast enough at production volume? | `mxcli report performance` | `report-performance` | +| Was the specification built? | `mxcli report traceability` | `report-traceability` | +| What is in it, for someone new? | `mxcli report map` | `report-map` | +| Does it build? | `mxcli docker check -p app.mpr` | `/validate-project` command | -## Assessment Workflow +Read each report's skill before interpreting it — every one has traps (the quality +score counts Marketplace modules unless scoped; the security report treats intended +row-level rules differently from real exposure). Findings that are intended belong in +the project brain as accepted lint findings (`report-quality`), so the next run does +not report them again. -### Step 1: Run Automated Analysis +The rules behind these reports are listed by `mxcli lint -p app.mpr --list-rules`. +This skill does not repeat them: the guidelines below are the ones no rule checks. -Run the automated linter and report to get a baseline score: +## Step 2: Manual review -```bash -# connect to the project and build a full catalog (needed for deep analysis) -mxcli -p app.mpr -c "refresh catalog full" +### A. Naming -# run the linter (41 rules: MDL, SEC, QUAL, ARCH, design, CONV series) -mxcli lint -p app.mpr +The rules check form (prefixes, suffixes, Boolean names). Judge meaning: -# generate the scored best practices report -mxcli report -p app.mpr --format markdown +- A flow's name says what it does beyond its prefix: `ACT_Customer_Save`, not + `ACT_Save`. +- Entities are singular and name a business concept, not a screen. +- One module uses one naming style; a mix shows a module with several authors and no + convention. -# score only the app's own modules (not Marketplace / platform modules) -mxcli report -p app.mpr --modules MyModule,OtherModule --format markdown -``` - -The report covers 6 categories with scores: **Naming**, **Security**, **Quality**, **Architecture**, **Performance**, **Design**. - -### Step 2: Explore Project Structure - -Use catalog queries to understand the project's shape: - -```sql --- Overview of project size and structure -describe structure depth 2; - --- Key metrics -select module_name, count(*) from CATALOG.entities GROUP by module_name; -select module_name, count(*) from CATALOG.microflows GROUP by module_name; -select module_name, count(*) from CATALOG.pages GROUP by module_name; - --- Complexity metrics (McCabe/Sigrid-style) -select qualified_name, cyclomatic_complexity -from CATALOG.microflows -where cyclomatic_complexity > 10 -ORDER by cyclomatic_complexity desc; - --- Large microflows -select qualified_name, activity_count -from CATALOG.microflows -where activity_count > 15 -ORDER by activity_count desc; -``` - -### Step 2b: Architecture & Dependency-Graph Analysis - -mxcli models the project as a dependency graph and exposes topological analyses on top of it — god nodes, coupling, cohesion, communities, centrality, cycles, and layering. These answer questions catalog metrics can't: what's central, what's tangled, what belongs together. - -**Read [`graph-analysis`](../graph-analysis/SKILL.md) before running graph analysis.** It covers setup (two-step refresh required), all eight use cases (impact assessment, dead code, cohesion, communities, bridges, hotspots, cycles, monolith splits), column schemas, false-positive caveats for dead assets, and a quick-reference table mapping questions to tables. - -### Step 3: Manual Review Against Guidelines - -After reviewing automated results, assess the following areas manually. The guidelines below are organized by category. - ---- - -## Quality Guidelines Reference - -### A. Naming Conventions - -**Enforced by rules:** MPR001, CONV001, CONV003, CONV004, CONV005 - -| Element | Convention | Example | -|---------|-----------|---------| -| **Entities** | Singular, PascalCase, no underscores | `Customer`, `OrderLine` (not `Customers`, `Order_Line`) | -| **Attributes** | PascalCase | `FirstName`, `OrderDate` | -| **Boolean attributes** | Prefix with Is/Has/Can/Should/Was/Will | `IsActive`, `HasChildren` | -| **Microflows** | Prefix indicating purpose | `ACT_Customer_Save`, `DS_Customer_GetAll` | -| **Pages** | Entity + suffix | `Customer_NewEdit`, `Order_Overview`, `Product_View` | -| **Enumerations** | Prefix with ENUM_ | `ENUM_OrderStatus`, `ENUM_Gender` | -| **Snippets** | Prefix with SNIPPET_ | `SNIPPET_CustomerDetails` | -| **Java actions** | Prefix with JA_ | `JA_SendEmail` | - -**Microflow prefix reference** (21 recognized prefixes): +Prefix reference for microflows: | Prefix | Purpose | Layer | |--------|---------|-------| @@ -121,200 +72,112 @@ After reviewing automated results, assess the following areas manually. The guid | `LOGIN_` | Login/authentication flow | UI | | `SUB_` | Reusable submicroflow | Logic | | `VAL_` | Validation logic | Logic | -| `BCO_` | Before commit handler | Event | -| `ACO_` | After commit handler | Event | -| `BCR_` | Before create handler | Event | -| `ACR_` | After create handler | Event | -| `BDE_` | Before delete handler | Event | -| `ADE_` | After delete handler | Event | -| `BRO_` | Before rollback handler | Event | -| `ARO_` | After rollback handler | Event | +| `BCO_` / `ACO_` | Before / after commit handler | Event | +| `BCR_` / `ACR_` | Before / after create handler | Event | +| `BDE_` / `ADE_` | Before / after delete handler | Event | +| `BRO_` / `ARO_` | Before / after rollback handler | Event | | `OCH_` | On-change handler | Event | -| `SCH_` | Scheduled event | Integration | +| `SCH_` / `SE_` | Scheduled event | Integration | | `IVK_` | Invocable (called by external systems) | Integration | -| `SE_` | Scheduled event (alternative) | Integration | | `DL_` | Deep link handler | Integration | | `PWS_` | Published web service | Integration | | `ASU_` | After startup microflow | System | -**Manual check:** Verify that microflow names are descriptive beyond just the prefix. For example, `ACT_Customer_Save` is better than `ACT_Save`. - -### B. Folder Structure +### B. Folder structure -**Not automatically enforced** (folder hierarchy not fully captured in catalog) +Not in the catalog in enough detail to check automatically. -| Convention | What to Check | +| Convention | What to check | |-----------|---------------| -| Each module has `[objects]` and `[UI]` top-level folders | Open in Studio Pro or check folder data | -| `ACT_` and `DS_` microflows live in the `[UI]` folder | These are page-related, belong in UI | -| Entity-specific microflows grouped in entity subfolders under `[objects]` | e.g., `[objects]/Customer/BCO_Customer_Validate` | -| Pages organized by entity or feature | Not scattered randomly | - -### C. Security - -**Enforced by rules:** SEC001-SEC011, CONV006, CONV007, CONV008, MPR007 - -#### C.1 Entity Access - -| Guideline | Rule | Priority | -|-----------|------|----------| -| All persistent entities must have access rules | SEC001 | Critical | -| Every role that opens a page can read the entities its grids, galleries and combo boxes list (else it renders empty at Production) | SEC010 | High | -| Every association a page data source's XPath crosses is readable by the roles that open the page, on the entity the step starts from (else the page fails to open at Production) | SEC011 | High | -| Every page a user role can reach shows only attributes and entities that role may read (else mxbuild fails with CE2729) | SEC012 | High | -| Do not grant Create or Delete rights on entity access rules | CONV006 | High | -| All entity access rules should have XPath constraints | CONV007 | High | -| Set default access rights to None | Manual check | High | -| Do not use entity validation rules (use microflow validation instead) | CONV015 | Medium | -| Do not use entity event handlers (use explicit microflow calls) | CONV016 | Medium | - -#### C.2 Role Management - -| Guideline | Rule | Priority | -|-----------|------|----------| -| 1:1 mapping between module roles and user roles | CONV008 | High | -| Pages in navigation must have allowed roles | MPR007 | High | -| Every user role can open its home page (else CE2729 at Production security) | MPR007 | High | -| No guest/anonymous access to sensitive data | SEC004 | Critical | -| Strict security mode enabled | SEC005 | High | -| No demo users in production | SEC003 | Critical | -| Password policy >= 8 characters | SEC002 | High | - -#### C.3 DIVD Findings (Common Misconfigurations) - -The Dutch Institute for Vulnerability Disclosure (DIVD) found widespread authorization misconfigurations in Mendix apps. Key checks: - -| Check | What to Verify | -|-------|---------------| -| Anonymous user permissions | Anonymous role should have minimal entity access; no access to business data | -| Newly registered user permissions | Default user role should not grant broad data access | -| XPath constraints completeness | Every READ/WRITE access rule on business entities needs XPath constraints | -| Published REST/OData security | All published services require authentication | -| Entity access audit | Run `select * from CATALOG.permissions where is_constrained = 0` | - -#### C.4 Additional Security Practices - -- Do not store sensitive information (passwords, API keys) in constants (values are visible in version control) -- Sanitize user input to prevent XSS (use `HTMLSanitize()` from Community Commons) -- Be cautious with nanoflow security — nanoflows execute client-side and can be inspected -- Use HTTPS for all external integrations -- Validate and `urlEncode()` all values used in REST URLs +| Each module has `[objects]` and `[UI]` top-level folders | Folder data, or Studio Pro | +| `ACT_` and `DS_` microflows live in `[UI]` | They belong to pages | +| Entity-specific microflows grouped per entity under `[objects]` | e.g. `[objects]/Customer/BCO_Customer_Validate` | +| Pages organised by entity or feature | Not scattered | -### D. Maintainability +### C. Security beyond the rules -**Enforced by rules:** QUAL001-QUAL004, CONV009, CONV012, CONV014, CONV020, CONV021, MPR002, MPR004, MPR006 - -| Guideline | Rule/Check | Threshold | -|-----------|-----------|-----------| -| Microflow complexity (McCabe) | QUAL001 | Cyclomatic complexity <= 10 | -| Microflow size | CONV009 | Max 15 activities | -| Documentation on public microflows | QUAL002 | All ACT_, DS_, IVK_ should be documented | -| No orphaned/unused elements | QUAL004 | Remove unused microflows, pages, entities | -| No empty microflows | MPR002 | Every microflow should have at least one activity | -| Caption on exclusive splits | CONV012 | All decision points must have captions | -| No "Continue" error handling | CONV014 | Never swallow errors silently | -| No created object silently discarded | CONV020 | A persistent object created without commit is committed, returned or passed on | -| Commit on the create/change, not after it | CONV021 | No Create/Change object activity directly followed by a Commit of the same object | -| ACT_ microflows: thin controllers | CONV010 | Only page activities + submicroflow calls | - -**Manual checks:** - -- Don't change activity captions (they should match the default for readability) -- Don't modify imported/marketplace modules (fork or extend instead) -- Generalize repeated snippets — if the same layout appears 3+ times, extract to a snippet -- Remove all Studio Pro warnings (yellow markers) -- Limit use of non-Mendix languages (Java, JavaScript) to what's necessary +`report security` gives the access grid, the client-callable flows that trust the +caller, unsafe defaults and a verdict per security rule. On top of it, check: -### E. Performance +- Anonymous and newly registered users get no business data. The DIVD (Dutch + Institute for Vulnerability Disclosure) found this misconfigured in many Mendix + apps. +- Every published REST and OData service requires authentication. +- User input shown as HTML is sanitised (`HTMLSanitize()` from Community Commons). +- Values in REST URLs are checked for empty and `urlEncode()`d. +- Nanoflows hold no logic that must be enforced: they run on the client. +- External integrations use HTTPS. + +### D. Maintainability -**Enforced by rules:** CONV011, CONV016, CONV017 +- Marketplace modules are not modified (fork or extend instead); a changed one cannot + be upgraded without losing the change. +- A layout repeated three or more times is a snippet. +- Activity captions keep their defaults, so a flow reads the same in every module. +- Java and JavaScript are limited to what Mendix cannot do. +- Studio Pro warnings are cleared, not ignored (`mxcli docker check` lists them). -| Guideline | Rule/Check | Impact | -|-----------|-----------|--------| -| No calculated attributes | CONV017 | Calculated attributes execute on every retrieve | -| No entity event handlers | CONV016 | Event handlers fire on every create/commit/delete, often unnecessarily | -| No commit inside loops | CONV011 | Each commit is a database round-trip; batch instead | -| No Export to Excel widget | Manual | Use CSV export for large datasets | -| Minimize in-use session objects | Manual | Large session caches increase memory pressure | +### E. Performance -**XPath performance tips:** -- Avoid negation in XPath (`not()`, `!=`) — these cannot use indexes efficiently -- Use indexed attributes in XPath constraints -- Prefer `[Assoc/entity/attribute = value]` over retrieving + filtering in microflow -- For large datasets, consider using a data source microflow with pagination +`report performance` measures journeys at volume; the rules catch commits in loops and +calculated attributes. Review by reading: -**Microflow performance tips:** -- Use batching for operations on 100+ objects (retrieve in batches of 1000, process, commit batch) -- Avoid deep nesting of microflow calls (call stack overhead) -- Use list operations (`find`, `filter`, `head`) instead of loops when possible +- XPath avoids negation (`not()`, `!=`), which cannot use an index, and filters on + indexed attributes. +- Filtering happens in XPath (`[Assoc/Entity/Attribute = value]`), not by retrieving + everything and filtering in the flow. +- Operations on 100+ objects are batched (retrieve 1000, process, commit the batch). +- List operations (`find`, `filter`, `head`) replace loops where they can. +- Large exports use CSV, not the Export to Excel widget. +- Few objects are kept in the session. ### F. Architecture -**Enforced by rules:** ARCH001-ARCH003, CONV010, MPR003 +`report architecture` shows layers, coupling, bounded contexts and cycles. Judge: -| Guideline | Rule/Check | Details | -|-----------|-----------|---------| -| No cross-module direct data access | ARCH001 | Access data through microflows, not direct entity references | -| Data changes through microflows only | ARCH002 | Don't change entities from pages directly | -| Business key on entities | ARCH003 | Persistent entities should have a business key attribute | -| Domain model size | MPR003 | Max 15 persistent entities per module | -| Entity attribute count | DESIGN001 | Max 10 attributes per entity (consider splitting) | +- Each module is one bounded context with a responsibility you can say in a sentence + (and its domain-model documentation says it: `report-map`). +- Non-persistent entities are used for view models and DTOs. +- Page data sources stay simple; complex retrieval is a `DS_` microflow. -**Manual checks:** +### G. Error handling -- Modules should represent bounded contexts (clear responsibilities) -- Avoid circular dependencies between modules -- Use non-persistent entities for view models / DTOs -- Keep page data sources simple — use DS_ microflows for complex data retrieval +The rules flag service calls without custom error handling and "Continue" handling. +Check the design: -### G. Error Handling +- The default is Stop + log + roll back + show the user a message. +- A REST call's error handler checks `$latestHttpResponse`: if present, log status + code, reason and response body, then rethrow; if absent (timeout, connection + refused), the default is enough. +- Logs carry context: the inputs and the response, not only "it failed". +- Errors are rethrown after logging, so the caller can handle them. -**Enforced by rules:** CONV013, CONV014 +### H. Connectors (consumed services) | Guideline | Details | |-----------|---------| -| Default error handling | All microflows should use Stop + Log + Roll back + Show user message | -| Custom error handling on service calls | REST calls, web service calls, and Java actions need custom error handling | -| Never use "Continue" error handling | This silently swallows errors — always handle explicitly | -| Log with context | Include status code, response body, and input parameters in error logs | -| Rethrow after logging | After logging the error details, rethrow so the caller can handle it | - -**Pattern for REST call error handling:** -1. Call REST service (with custom error handling) -2. On error: check if `$latestHttpResponse` exists -3. If yes: log status code + reason + response content, then rethrow -4. If no: default error handling is sufficient (timeout/connection refused) - -### H. Connectors (Consumed Services) +| Simplify mappings | Remove empty and optional entities from import mappings | +| Domain model as a tree | Parents above children, generalisation where it fits, comments between sections | +| String attributes unlimited | Connector strings should not be length-limited | +| No default 0 on numbers | Leave connector numbers empty | +| Document the model | Each entity and attribute's purpose | +| One place for connection details | Endpoint, credentials and timeout in constants with empty defaults, or a ConnectionDetails entity | +| Operation naming | `METHOD_VERSION_OPERATION` | +| Folders | `Connection Details`, `Operations`, `Private` | -| Guideline | Details | -|-----------|---------| -| Simplify mappings | Remove empty/optional entities from import mappings | -| Tree structure for domain model | Parent entities above children, use inheritance, add comments between sections | -| String attributes: set to Unlimited | Connector strings should not have length limits | -| Remove default value 0 on numbers | Default 0 is confusing for connector data — leave empty | -| Add documentation to model | Document each entity and attribute purpose | -| Use ConnectionDetails entity | Centralize endpoint, username, password, timeout in one entity | -| Use constants for connection settings | Avoid dependency on Encryption module; use constants with empty defaults | -| Operations naming | Follow `METHOD_VERSION_OPERATION` pattern | -| File structure | Separate `connection Details`, `Operations`, `Private` folders | -| Validate input before urlEncode | Check for empty values before constructing URLs | - -### I. UI Best Practices +### I. UI -| Guideline | Details | -|-----------|---------| -| Use SASS, not inline CSS | All styling in `_custom-variables.scss` or theme files | -| Avoid nested styling rules | Keep CSS flat for maintainability | -| No inline `style` attributes | Use CSS classes instead | -| Consistent page layout | Use layout grids, not absolute positioning | +- Styling is in the theme's SCSS, not inline `style` attributes (`theme-styling` says + which file compiles). +- SCSS rules are kept flat. +- Layout uses layout grids, not absolute positioning. -### J. Quality Metrics (SIG/Sigrid Model) +### J. Complexity hotspots (SIG/Sigrid model) -The catalog includes McCabe cyclomatic complexity metrics aligned with the ISO 25010 maintainability standard used by Sigrid. Use these queries to assess code quality: +The catalog has McCabe complexity per microflow, aligned with the ISO 25010 +maintainability model Sigrid uses: ```sql --- Complexity distribution (Sigrid-style) select case when cyclomatic_complexity <= 5 then 'Low (1-5)' @@ -326,7 +189,6 @@ select from CATALOG.microflows GROUP by complexity_band; --- Top complex microflows select qualified_name, cyclomatic_complexity, activity_count from CATALOG.microflows where cyclomatic_complexity > 10 @@ -334,118 +196,52 @@ ORDER by cyclomatic_complexity desc limit 20; ``` -**Sigrid benchmarks** (5-star system): -- 5 stars: 90%+ of microflows have complexity <= 5 -- 4 stars: 80%+ of microflows have complexity <= 10 -- 3 stars: 70%+ of microflows have complexity <= 15 -- Below 3 stars: significant maintainability risk +Sigrid benchmarks: 5 stars when 90%+ of microflows are at complexity ≤ 5; 4 stars at +80%+ ≤ 10; 3 stars at 70%+ ≤ 15; below that is a maintainability risk. -### K. View Entities +### K. View entities -View entities are powerful for specific scenarios but not universal replacements for standard logic: +Use one where a microflow data source is too slow, where OQL replaces loops and +filters, or to flatten data for reports. They add OQL to the skills the team needs, so +only where the benefit is clear. -| Use Case | Recommendation | -|----------|---------------| -| Large dataset performance | View entity is recommended when microflow data source is too slow | -| Complex OQL queries | View entity can replace complex microflow logic with loops/filters | -| Flattening data for reports | More efficient than non-persistent entities filled by microflows | - -**Caution:** View entities increase technical complexity and require OQL knowledge. Only use when there is a clear performance or simplification benefit. - ---- - -## Assessment Report Template - -After running the automated tools and performing manual review, produce a report using this structure: +## Step 3: Write the assessment ```markdown -# Mendix project Quality Assessment - -**project:** [Name] -**date:** [date] -**Assessor:** Claude Code - -## Executive Summary +# Quality assessment: [project] -Overall Score: [X]/100 (from `mxcli report`) +**Date:** [date] · **Mendix:** [version] · **Assessed with:** mxcli [version] -| Category | Score | key Finding | -|----------|-------|-------------| -| Naming | X% | [one-line summary] | -| security | X% | [one-line summary] | -| Quality | X% | [one-line summary] | -| Architecture | X% | [one-line summary] | -| Performance | X% | [one-line summary] | -| design | X% | [one-line summary] | +## Summary +[Three sentences: is it in good shape, what are the biggest risks, what to do first.] -## project Metrics +## Measured +| Area | Result | Source | +|---|---|---| +| Builds | [clean / N errors, M warnings] | mxcli docker check | +| Quality score | [X/100; lowest category and why] | mxcli report | +| Architecture | [cycles, coupling hotspots, dead code] | report architecture | +| Security | [open high findings; flows that trust the caller] | report security | +| Tests | [pass/fail; coverage of the app's modules] | report tests | +| Performance | [slowest step at volume] | report performance | +| Requirements | [built / partial / not built] | report traceability | +| Accepted findings | [count, with the brain decision ids] | mxcli lint --show-accepted | -| Metric | value | -|--------|-------| -| modules | X | -| entities (persistent) | X | -| microflows | X | -| pages | X | -| avg complexity | X | -| max complexity | X | - -## Automated Findings - -[Summary of lint violations by category, grouped by severity] - -### critical Issues (must fix) -- [list] - -### High Priority (should fix) -- [list] - -### Medium Priority (consider fixing) -- [list] - -## Manual Review Findings - -### Naming Conventions -[Assessment against Section A guidelines] - -### security Posture -[Assessment against Section C guidelines, including DIVD checklist] - -### Maintainability -[Assessment against Section D guidelines] - -### Performance Risks -[Assessment against Section E guidelines] - -### Architecture -[Assessment against Section F guidelines] - -### error Handling -[Assessment against Section G guidelines] +## Reviewed by hand +[One short section per area from Step 2 with something to say; skip the rest.] ## Recommendations +1. [Highest impact first; each one specific and actionable] -### Top 5 actions (Highest impact) -1. [specific, actionable recommendation] -2. [specific, actionable recommendation] -3. [specific, actionable recommendation] -4. [specific, actionable recommendation] -5. [specific, actionable recommendation] - -### Quick Wins (Low Effort, High value) -- [list of easy fixes] - -### long-term Improvements -- [list of architectural/structural improvements] +## Not assessed +[What this assessment did not look at: anything a report was not run for, and what +the model cannot show — infrastructure, hosting, backups, licences, a penetration test.] ``` ## Sources -This skill incorporates guidelines from: -- **Conventions.pdf** — Squad Apps internal best practices (14 categories, 80+ guidelines) -- **CONV001-CONV017 lint rules** — Automated checks derived from Conventions.pdf -- **MPR001-MPR007, SEC001-SEC011** — Built-in linter rules -- **ARCH, DESIGN, QUAL series** — Starlark architecture/quality rules -- **Mendix Performance Best Practices** — Official Mendix documentation on performance optimization -- **Mendix Security Best Practices** — Official Mendix documentation on security configuration -- **DIVD Mendix Findings** — Dutch Institute for Vulnerability Disclosure (entity access misconfigurations) -- **SIG/Sigrid Quality Model** — ISO 25010 maintainability metrics, McCabe complexity benchmarks +- **Conventions.pdf** — Squad Apps internal best practices; the CONV lint rules + automate part of it. +- **Mendix performance and security best practices** — official Mendix documentation. +- **DIVD Mendix findings** — entity access misconfigurations found in the wild. +- **SIG/Sigrid quality model** — ISO 25010 maintainability, McCabe benchmarks. diff --git a/.claude/skills/mendix/bootstrap-app/SKILL.md b/.claude/skills/mendix/bootstrap-app/SKILL.md index 0c40780d10..aad333a40b 100644 --- a/.claude/skills/mendix/bootstrap-app/SKILL.md +++ b/.claude/skills/mendix/bootstrap-app/SKILL.md @@ -12,10 +12,10 @@ description: "Provision a Mendix project in a repo that has none — interview, |---|---| | 23-43 | When to Use This Skill | | 44-134 | Step 0 — interview, and WAIT for the answers | -| 135-267 | Provision | -| 268-337 | If this is a solution (several apps in one repo) | -| 338-397 | Then propose the model — do not build it yet | -| 398-461 | After bootstrap — the inner loop | +| 135-268 | Provision | +| 269-338 | If this is a solution (several apps in one repo) | +| 339-398 | Then propose the model — do not build it yet | +| 399-462 | After bootstrap — the inner loop | # Bootstrap a Mendix App in an Empty Repo @@ -231,8 +231,9 @@ drop the `./` if it came pre-installed on `PATH`. template ships with" — afterwards every figure is yours plus the template's, and there is nothing to subtract. A blank app is **not** expected to score zero warnings; knowing which ones it starts with is what stops you chasing them later. - Read `.ai-context/skills/assess-quality/SKILL.md` before interpreting the report — - it covers what each category means and which findings are worth acting on. + Read `.ai-context/skills/report-quality/SKILL.md` before interpreting the report — + it covers how the score is computed, what it leaves out, and how to accept a + finding that is intended. 10. **(Optional) browser preview from a cloud session:** `./mxcli run --hub https://hub.mxcli.org -p .mpr`, and report the preview URL it prints. Needs `MXCLI_HUB_KEY` on the environment; without it, continue as a @@ -427,7 +428,7 @@ Two of them are easy to mistake for optional and are not: - **`report` is the quality report**, and its six category scores are what the baseline in step 9 exists to be compared against. A score that fell is a finding, - not a detail. `assess-quality` covers how to read it. + not a detail. `report-quality` covers how to read it. - **`test` needs a suite to run.** Write the first one with the **first microflow you build** — not "later", because later is after the code is written and the expected values have stopped being obvious. One `tests/.test.mdl` per slice is the diff --git a/.claude/skills/mendix/graph-analysis/SKILL.md b/.claude/skills/mendix/graph-analysis/SKILL.md index 914b1e7b80..18ec37c9a0 100644 --- a/.claude/skills/mendix/graph-analysis/SKILL.md +++ b/.claude/skills/mendix/graph-analysis/SKILL.md @@ -22,7 +22,7 @@ description: "Query the project as a dependency graph to assess the blast radius | 300-331 | Use Case 8: Monolith-to-Multi-App Planning | | 332-347 | Topological Layer Reference | | 348-367 | Quick Reference: Which Table Answers What | -| 368-373 | Related Skills | +| 368-374 | Related Skills | # Graph Analysis: Impact Assessment & Architecture Evaluation @@ -367,7 +367,8 @@ Elements at the same layer can be parallelised safely. A microflow calling somet ## Related Skills -- [assess-quality](../assess-quality/SKILL.md) — Full quality assessment including lint, report scores, and manual review guidelines +- [assess-quality](../assess-quality/SKILL.md) — Full quality assessment: the reports plus the manual review no rule can do +- [report-quality](../report-quality/SKILL.md) — The scored quality report and how to read it - [organize-project](../organize-project/SKILL.md) — MOVE command and folder structure - [write-lint-rules](../write-lint-rules/SKILL.md) — Encoding a layering policy as an enforced Starlark rule - [odata-data-sharing](../odata-data-sharing/SKILL.md) — Publishing entities for cross-app access diff --git a/.claude/skills/mendix/json-structures-and-mappings/SKILL.md b/.claude/skills/mendix/json-structures-and-mappings/SKILL.md index 7da0b26423..6d4b715bdb 100644 --- a/.claude/skills/mendix/json-structures-and-mappings/SKILL.md +++ b/.claude/skills/mendix/json-structures-and-mappings/SKILL.md @@ -14,16 +14,16 @@ description: "Create and manage JSON structures, import mappings and export mapp | 111-122 | Export Mappings | | 123-187 | JSON Structures | | 188-234 | Message Definitions | -| 235-398 | Import Mappings | -| 399-496 | Export Mappings | -| 497-514 | Starting a Mapping Below the Payload Root | -| 515-598 | Microflow Actions | -| 599-611 | Browse | -| 612-659 | Export Workflow: PE → NPE → JSON | -| 660-666 | Realistic Example: Countries REST API | -| 667-690 | Placing Documents in Folders | -| 691-706 | Documentation | -| 707-721 | Common Mistakes | +| 235-400 | Import Mappings | +| 401-498 | Export Mappings | +| 499-516 | Starting a Mapping Below the Payload Root | +| 517-600 | Microflow Actions | +| 601-613 | Browse | +| 614-661 | Export Workflow: PE → NPE → JSON | +| 662-668 | Realistic Example: Countries REST API | +| 669-692 | Placing Documents in Folders | +| 693-708 | Documentation | +| 709-723 | Common Mistakes | # JSON Structures, Import Mappings & Export Mappings @@ -389,7 +389,9 @@ create import mapping Module.IMM_Embed Using `parameter` without declaring one is refused — the build reports it as CE0279. The declared entity must match the microflow's parameter type, which the -build checks as CE0282. +build checks as CE0282. A handler's `P: member` is checked the same way: a +String parameter takes any member, a Decimal parameter also an Integer, a Long +parameter an Integer, and every other type only its own (CE0282, `check` MDL107). The **activity** that calls such a mapping passes the object in its settings list, `(Parameter: $Obj)` — see "Settings" under Import from Mapping below. diff --git a/.claude/skills/mendix/report-quality/SKILL.md b/.claude/skills/mendix/report-quality/SKILL.md new file mode 100644 index 0000000000..8c4e4f0232 --- /dev/null +++ b/.claude/skills/mendix/report-quality/SKILL.md @@ -0,0 +1,104 @@ +--- +name: report-quality +description: "Produce and read the scored quality report (mxcli report): every lint rule run over the project, six category scores and an overall score out of 100, recommendations and findings. Use when asked for a project's quality score, to compare it with a baseline, or to decide which findings matter, and before accepting a finding as intended." +--- + +## Find it fast + +`sed -n ',p' SKILL.md` one section rather than reading the file; +`./mxcli syntax ` answers *how do I write this* for a fraction of either. + +| lines | section | +|---|---| +| 29-40 | Run it | +| 41-56 | Before trusting the number | +| 57-76 | Reading the score | +| 77-97 | Accepting a finding | +| 98-104 | Related | + + +# Quality Report + +`mxcli report` with no kind is the **scored** report: it runs every lint rule, groups +the findings into categories (Security, Quality, Architecture, Performance, Naming, +Design, and Other for unmapped rules), scores each, and weighs them into one number +out of 100. The other reports each answer one question +(`report-architecture`, `report-security`, `report-tests`, `report-performance`, +`report-traceability`, `report-map`); this one is the summary across all rules. + +## Run it + +```bash +mxcli report -p app.mpr -o docs/quality.md +mxcli report -p app.mpr --format html -o docs/quality.html +mxcli report -p app.mpr --format json -o docs/quality.json # for CI, or to compare runs +# the app's own modules only, without Marketplace and platform modules +mxcli report -p app.mpr --modules Sales,Billing +``` + +It builds a full catalog first, so the first run on a project takes a while. + +## Before trusting the number + +- **Are all the rules running?** Without a `.claude/lint-rules/` directory beside the + project (or above it, up to the repository root), only the built-in rules run and the + bundled Starlark rules do not. The report warns on stderr, and the score looks + *better*, because fewer rules can find anything. `mxcli init` installs them. + `mxcli lint -p app.mpr --list-rules` shows what runs. +- **Which modules?** Without `--modules`, Marketplace and platform modules count too, + and their findings are not yours to fix. With `--modules`, project-wide findings — + security level, demo users, user-role mappings — are left out (they are about no + one module). Use both: the full run for the project's settings, the scoped run + for the code the team owns. +- **Does it build?** The score does not include Mendix's own consistency check. + A project with CE errors can score well. Run `mxcli docker check -p app.mpr` as + well (ako/mxcli#2423 tracks putting it in the report). + +## Reading the score + +Each category starts at 100 and loses a fixed amount **per finding** (an error costs +far more than a warning, a warning more than an info), floored at 0. The overall +score is a weighted average with Security weighted highest. The amounts and weights +live in `mdl/linter/report.go`. + +Two consequences: + +- **The penalty does not scale with the app.** A large app with many intended + findings of one rule — row-level access rules, say — can sit at 0 in a category for + reasons that are all deliberate. Read the findings, not the number. +- **The number is for comparison, not for judgement.** Record it at a known point + (the template's first build, a release) and compare later runs with it; a score + that fell is a finding. `bootstrap-app` records this baseline. + +The **Detailed Findings** table is what to work through: the rule, its severity, where, +and why. `mxcli lint -p app.mpr` gives the same findings for one module or one +document (`--modules`, `--documents`) while fixing them. + +## Accepting a finding + +When a finding is intended, record that decision once, in the project brain, rather +than ignoring it: + +```bash +mxcli brain capture "Beperking has no access rules on purpose" \ + --accepts "SEC001:read only through DS_Beperkingen_Inzien, which logs every view" \ + -a @Dossier.Beperking -p app.mpr +``` + +The report then counts it apart ("Accepted (brain decisions, not scored)") and does not +score it; `lint` leaves it out of the open findings, and `report security` shows the +rule as accepted. A decision with no anchor accepts a project-wide finding (SEC013, +SEC003). When the rule stops firing there, `mxcli lint` reports the acceptance as +stale. `mxcli lint --show-accepted` lists what is accepted, with the deciding entry. + +Accept only what is decided. Code still being written belongs in a lint baseline +(`mxcli lint --baseline lint-baseline.json --update-baseline`), which tracks new +findings without saying anything is intended. + +## Related + +- `assess-quality` — the manual review the rules cannot do, and how to write a full + assessment that combines it with these reports. +- `write-lint-rules` — adding a project's own rules. +- `report-security` — the security findings with a verdict per rule, and what + protects the data. diff --git a/CHANGELOG.md b/CHANGELOG.md index 335de69685..a4aae40112 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +### Fixed + +- **A freshly built catalog is one database again, so an iterator no longer fails while its neighbours work** (ako/mxcli#2417) — on a run that built the catalog rather than loading it, `lint` could report `could not read the catalog for AttributesFor: no such table: attributes` while `Entities` returned rows in the same run on the same handle, and the next identical run was correct. `catalog.New()` opened `:memory:`, where **every connection is its own separate, empty database**, and `sql.Open` hands back a pool — so the schema landed on one connection and a query the pool served on another saw nothing. Which iterator broke was pool timing, which is why it reproduced on one project and not another. The in-memory catalog now uses a uniquely named shared-cache DSN with one pinned connection, so every pooled connection sees the same database. Measured: the first `lint` of a fresh project reports what the second always did, and reverting only the DSN brings the old error back. + + Two knock-on fixes. A fresh catalog's snapshot `ObjectCount` was counted as **0**, because `updateSnapshotCount` read through the pool while the build transaction was open and so could not see the rows that transaction had just written; it now reads through the transaction. And `foreign_keys` was set with `Exec`, which reaches only the one pooled connection it runs on, so the pragma was off for most queries — on both the in-memory and the file path. It now rides in the DSN, as the busy timeout already did for exactly this reason. + + Not the fix, recorded because it looks like one: pinning the pool to a single connection also makes the catalog one database and **deadlocks**. The readers nest — `Entities()` yields from an open cursor while the loop body calls `AttributesFor()` — so the inner read waits forever for the connection the outer cursor holds, and a lint run that took seconds hung past 600s. + ### Added - **Lint `MPR017`: a Boolean flag a decision tests but which can only ever hold one value** (ako/mxcli#2403) — the branch it guards is unreachable, and the flow reads as though it has a safety interlock when it has none. Nothing else reports it: the model is valid and the build is clean. @@ -29,6 +37,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **An enumeration value written without a caption gets its name as the caption** (ako/mxcli#2397, mendixlabs/mxcli#1893) — `create enumeration … (Red)` and `alter enumeration … add value Blue` stored an empty caption, so the value showed as a blank option and nothing reported it. The caption now defaults to the value's name, as Studio Pro fills it; `''` written explicitly stays empty. - **`create translations` matches a source whatever its line endings** (ako/mxcli#2383, mendixlabs/mxcli#1890) — Marketplace texts carry CRLF, and a `describe translations` file whose CRs were lost when it was edited reported each of them as unknown, leaving confirmation questions without a default-language text (CE4899). A source now matches with CRLF and LF treated alike, and its translation is stored with the source's line breaks. - **Lint `parse_xpath` parses `true()` / `false()` and no longer returns a partial tree** (ako/mxcli#2398, mendixlabs/mxcli#1894) — everything after an XPath `true()` or `false()` was dropped without a word, so `[IsConfidential = false() or Owner = '[%CurrentUser%]']` reached a rule as `IsConfidential = false`. Both forms now parse whole, and a constraint that does not parse to its end comes back as a `recovered` node, as the lint-rules skill documents. +- **`check` rejects an import mapping's custom handler that feeds a parameter a member of the wrong type** (ako/mxcli#1258) — `find … by M.MF ( P: member )` with a String member into an Integer parameter, or any member into a Boolean or DateTime parameter of another type, passed `check` and `exec`, and the build reported CE0282 "The microflow parameter … does not match the specified type". New MDL107 over the measured matrix: a String parameter takes every member type, a Decimal parameter also an Integer, a Long parameter an Integer, the rest only their own type. The member is found by name in the whole structure and judged only when every occurrence has one type, so an ambiguous member is left to the build. Measured on MxBuild 11.16.1, one mapping per project. - **`check` judges the values of an enumeration split** (ako/mxcli#1258) — a `case $S/C when … end case` that leaves an enumeration value without a branch, names a value the enumeration does not have, or names one value in two branches passed `check` and `exec`, and the build reported CE0079, CE1613 (values are case-sensitive) and CE0771. Two new rules: MDL105 (a value named twice, no project needed) and MDL106 (an unknown or missing value, under `check -p`, which reads the enumeration from the script or the project). An unknown value is reported alone, as the build does. A split whose subject does not resolve — a rule call, an association path, an enumeration the script alters or drops — is not judged. Measured on MxBuild 11.16.1, microflows and nanoflows. CE0773, which older records list beside CE0079, does not appear for any of these; it is the follow-on of a branch with no configured value, which the grammar already refuses for `else`. - **`marketplace diff` and `update` no longer report untouched pluggable-widget snippets as edits** (ako/mxcli#2367, mendixlabs/mxcli#1882) — right after an install, Workflow Commons showed 2 changed snippets on the first `diff` and 8 on every later one, so `update` needed `--force`, which also dropped the guard for real edits. The reference cache now keeps the widget definitions, and a reference gets the same widget and design-property repairs install runs on the app. Measured: 0 false edits on both runs, and a real edit is still reported. Cached references from earlier versions are rebuilt once. - **An imported navigation association that would clash is named `_`** (ako/mxcli#2354) — `create external entities` named Product's `Category` navigation `Category_2`, because the name collides with the entity Category. It is now `Product_Category`, with a number only if that is taken too. Names that do not collide are unchanged, and a re-import still recognises existing associations by their navigation property. diff --git a/cmd/mxcli/docker/check_findings.go b/cmd/mxcli/docker/check_findings.go index 97f127ce3e..09c542149b 100644 --- a/cmd/mxcli/docker/check_findings.go +++ b/cmd/mxcli/docker/check_findings.go @@ -4,136 +4,59 @@ package docker import ( "bytes" + "encoding/json" "fmt" - "regexp" - "strconv" + "os" + "sort" "strings" ) -// A CheckFinding is one diagnostic line of `mx check`, parsed. It is the unit a -// baseline compares (PROPOSAL_verify_gate.md): the text of `mx check` is not -// stable between runs, so a caller compares Keys and counts, never lines. -// -// mx check prints one line per finding: -// -// [error] [CE0117] "Error(s) in expression." at Microflow 'ACT_X' +// A CheckFinding is one problem `mx check` reports. It is the unit a baseline +// compares (PROPOSAL_verify_gate.md): the text of `mx check` is not stable +// between runs and does not identify the document (two microflows with the same +// defect print the same line), so findings are read from `mx check --json`, +// which names the module, document, element and unit by name and by ID. type CheckFinding struct { - Severity string // "error", "warning" or "deprecation", as mx prints it - Code string // "CE0117" - Message string // the quoted text; display only, never part of the Key - Location string // text after `" at `, or "" when the line names none - Kind string // "Microflow", "Data grid 2", ... (the Location before its quoted name) - Name string // the quoted name in Location; "" when empty or absent - Class LocationClass - Stage int // MxBuild report stage 0..3, see StageOf - Line string + Severity string // "error", "warning" or "deprecation": the JSON list it came from + Code string // "CE0109" + Message string // display only, never part of the Key + Locations []CheckLocation + Stage int // MxBuild report stage 0..3, see StageOf } -// LocationClass says how much of "which element" a finding's text carries, -// which decides how a baseline can match it (PROPOSAL_verify_gate.md, -// "Finding identity"). -type LocationClass int - -const ( - // Located: the line names the kind and name of the element. - Located LocationClass = iota - // DocumentOnly: the line names a document, not the element in it. - DocumentOnly - // Unlocated: the line names nothing that identifies it. Matched as a count - // per distinct line, as recheckCE0463 already does. - Unlocated -) - -func (c LocationClass) String() string { - switch c { - case Located: - return "located" - case DocumentOnly: - return "document-only" - default: - return "unlocated" - } +// CheckLocation is where a finding is. IDs are what a baseline matches on; the +// names are for a reader and change when an element is renamed. +type CheckLocation struct { + ModuleName string `json:"module-name"` + DocumentName string `json:"document-name"` // "Microflow 'F05'" + ElementName string `json:"element-name"` // "End event" + ElementID string `json:"element-id"` + UnitID string `json:"unit-id"` + ModuleID string `json:"module-id"` } -// Key is what a baseline matches on: severity, code and location, never the -// message. For an Unlocated finding the location is the whole line's text, so -// the same line repeated is the same key and the caller counts it. -func (f CheckFinding) Key() string { - return f.Severity + "\x00" + f.Code + "\x00" + f.Class.String() + "\x00" + f.Location -} - -var findingRe = regexp.MustCompile(`^\[(error|warning|deprecation)\]\s+\[(C[EWD]\d+)\]\s+"`) - -// locationRe splits "Microflow 'ACT_X'" into kind and name. -var locationRe = regexp.MustCompile(`^(.*?)\s*'(.*)'$`) - -// ParseCheckOutput reads the diagnostics out of `mx check` output. Lines that -// are not findings (banners, the closing "The app contains: N errors.") are -// skipped. A finding the parser cannot split still counts: its Message is the -// rest of the line. -func ParseCheckOutput(output string) []CheckFinding { - var out []CheckFinding - for _, raw := range strings.Split(output, "\n") { - line := strings.TrimSpace(strings.TrimRight(raw, "\r")) - m := findingRe.FindStringSubmatchIndex(line) - if m == nil { - continue - } - f := CheckFinding{Severity: line[m[2]:m[3]], Code: line[m[4]:m[5]], Line: line} - rest := line[m[1]:] // after the opening quote - if i := strings.LastIndex(rest, `" at `); i >= 0 { - f.Message, f.Location = rest[:i], strings.TrimSpace(rest[i+len(`" at `):]) - } else { - f.Message = strings.TrimSuffix(rest, `"`) +// Identified says whether the finding names an element by ID. One without a +// location (a project-level finding, such as the runtime settings) has nothing +// to key on but its code, so a baseline matches it by count per Key. +func (f CheckFinding) Identified() bool { + for _, l := range f.Locations { + if l.ElementID != "" || l.UnitID != "" { + return true } - if lm := locationRe.FindStringSubmatch(f.Location); lm != nil { - f.Kind, f.Name = strings.TrimSpace(lm[1]), lm[2] - } else { - f.Kind = f.Location - } - f.Class = classify(f) - f.Stage = StageOf(f.Code) - out = append(out, f) } - return out + return false } -// unlocatedCodes are codes whose text does not name the element, so two -// same-named elements on different documents print the same line. CE0463 does -// not name its page (docker/check.go, recheckCE0463). -var unlocatedCodes = map[string]bool{"CE0463": true} - -// documentKinds are the kinds whose name is a document, so the line says which -// document but not what in it. A kind not listed names an element (a widget, -// an activity) and counts as Located. The list is the measured set, not a -// claim of completeness: an unlisted document kind errs towards Located. -var documentKinds = map[string]bool{ - "Microflow": true, - "Nanoflow": true, - "Page": true, - "Snippet": true, - "Layout": true, - "Domain model of module": true, - "Enumeration": true, - "Workflow": true, - "Java action": true, - "Rule": true, - "Constant": true, - "Import mapping": true, - "Export mapping": true, - "Published REST service": true, - "Consumed REST service": true, -} - -func classify(f CheckFinding) LocationClass { - switch { - case unlocatedCodes[f.Code], f.Location == "", f.Name == "": - return Unlocated - case documentKinds[f.Kind]: - return DocumentOnly - default: - return Located +// Key is what a baseline matches on: severity, code and the IDs of every +// location, never the message or a name. Two findings with the same Key are the +// same finding repeated, and a baseline compares how many of them there are. +func (f CheckFinding) Key() string { + var ids []string + for _, l := range f.Locations { + ids = append(ids, l.UnitID+"/"+l.ElementID) } + sort.Strings(ids) + return f.Severity + "\x00" + f.Code + "\x00" + strings.Join(ids, ",") } // mxStages is MxBuild's report stage per code, as measured in @@ -160,59 +83,111 @@ func StageOf(code string) int { return 3 } -var appErrorsRe = regexp.MustCompile(`The app contains: (\d+) errors?\.`) +// checkJSONVersion is the `serialization_version` this reader understands. +const checkJSONVersion = 1 + +type checkJSONProblem struct { + Code string `json:"code"` + Message string `json:"message"` + Locations []CheckLocation `json:"locations"` +} + +type checkJSON struct { + Version int `json:"serialization_version"` + Errors []checkJSONProblem `json:"errors"` + Warnings []checkJSONProblem `json:"warnings"` + Deprecations []checkJSONProblem `json:"deprecations"` + Total *int `json:"total-problems"` +} -// CheckReport is the parsed result of one `mx check` run. +// CheckReport is the parsed result of one `mx check --json` run. type CheckReport struct { Findings []CheckFinding - // Declared is the error count mx printed on its closing line; HasDeclared - // is false when that line is absent (mx did not complete). + // Declared is the total mx printed, HasDeclared false when it printed none. Declared int HasDeclared bool } -// Errors counts the findings of severity error. -func (r CheckReport) Errors() int { +// Count returns the findings of one severity. +func (r CheckReport) Count(severity string) int { n := 0 for _, f := range r.Findings { - if f.Severity == "error" { + if f.Severity == severity { n++ } } return n } -// Verify cross-checks the parse against the count mx printed. A mismatch means -// the parser lost lines (or mx changed its format), and a baseline built from it -// would silently hide findings, so a caller must not treat the report as -// complete. +// Verify cross-checks the findings against the total mx declared. A mismatch +// means a list this reader does not know about (or a format change), and a +// baseline built from it would silently omit findings, so a caller must not +// treat the report as complete. func (r CheckReport) Verify() error { if !r.HasDeclared { - return fmt.Errorf("mx check did not print its closing error count; the report may be incomplete") + return fmt.Errorf("mx check --json did not declare a total; the report may be incomplete") } - if got := r.Errors(); got != r.Declared { - return fmt.Errorf("parsed %d errors but mx check reported %d; the output format may have changed", got, r.Declared) + if len(r.Findings) != r.Declared { + return fmt.Errorf("read %d findings but mx check declared %d; the output format may have changed", + len(r.Findings), r.Declared) } return nil } -// ParseCheckReport parses output and reads mx's own error count. -func ParseCheckReport(output string) CheckReport { - r := CheckReport{Findings: ParseCheckOutput(output)} - if m := appErrorsRe.FindStringSubmatch(output); m != nil { - if n, err := strconv.Atoi(m[1]); err == nil { - r.Declared, r.HasDeclared = n, true +// ParseCheckJSON reads the file `mx check --json ` writes. It refuses a +// serialization_version it has not seen rather than guess at its fields. +func ParseCheckJSON(data []byte) (CheckReport, error) { + var j checkJSON + if err := json.Unmarshal(data, &j); err != nil { + return CheckReport{}, fmt.Errorf("read mx check json: %w", err) + } + if j.Version != checkJSONVersion { + return CheckReport{}, fmt.Errorf("mx check json has serialization_version %d; this reader understands %d", + j.Version, checkJSONVersion) + } + var r CheckReport + add := func(severity string, ps []checkJSONProblem) { + for _, p := range ps { + r.Findings = append(r.Findings, CheckFinding{ + Severity: severity, Code: p.Code, Message: p.Message, + Locations: p.Locations, Stage: StageOf(p.Code), + }) } } - return r + add("error", j.Errors) + add("warning", j.Warnings) + add("deprecation", j.Deprecations) + if j.Total != nil { + r.Declared, r.HasDeclared = *j.Total, true + } + return r, nil +} + +// CheckKinds selects the findings beyond errors, which mx always reports. +type CheckKinds struct { + Warnings, Deprecations, BestPractice bool } -// CheckFindings runs `mx check` on a temporary copy of the project and returns -// its parsed report. It checks the project as stored (no update-widgets), never -// modifies it, and returns the report together with mx's own exit error, which -// is non-zero whenever the project has errors: a non-nil error is not a failure -// to run unless the report is empty and Verify fails. -func CheckFindings(opts CheckOptions) (CheckReport, error) { +func (k CheckKinds) args(jsonPath string) []string { + args := []string{"-j", jsonPath} + if k.Warnings { + args = append(args, "-w") + } + if k.Deprecations { + args = append(args, "-d") + } + if k.BestPractice { + args = append(args, "-b") + } + return args +} + +// CheckFindings runs `mx check --json` on a temporary copy of the project and +// returns the parsed report. It checks the project as stored (no +// update-widgets) and never modifies it. mx exits non-zero whenever it finds +// anything, so a run error is returned alongside the report; it is a failure to +// run only when the report is empty and Verify fails. +func CheckFindings(opts CheckOptions, kinds CheckKinds) (CheckReport, error) { projectVersion := "" if opts.ProjectPath != "" { if reader, err := openReadOnly(opts.ProjectPath); err == nil { @@ -224,7 +199,27 @@ func CheckFindings(opts CheckOptions) (CheckReport, error) { if err != nil { return CheckReport{}, err } + // The report goes outside the temporary copy, which MxCheckOnCopy removes. + f, err := os.CreateTemp("", "mxcheck-*.json") + if err != nil { + return CheckReport{}, err + } + jsonPath := f.Name() + f.Close() + defer os.Remove(jsonPath) + var out, errOut bytes.Buffer - runErr := MxCheckOnCopy(mxPath, opts.ProjectPath, nil, &out, &errOut) - return ParseCheckReport(out.String() + "\n" + errOut.String()), runErr + runErr := MxCheckOnCopy(mxPath, opts.ProjectPath, kinds.args(jsonPath), &out, &errOut) + data, readErr := os.ReadFile(jsonPath) + if readErr != nil || len(bytes.TrimSpace(data)) == 0 { + if runErr != nil { + return CheckReport{}, fmt.Errorf("mx check wrote no json report: %w\n%s", runErr, strings.TrimSpace(errOut.String())) + } + return CheckReport{}, fmt.Errorf("mx check wrote no json report") + } + report, err := ParseCheckJSON(data) + if err != nil { + return CheckReport{}, err + } + return report, runErr } diff --git a/cmd/mxcli/docker/check_findings_integration_test.go b/cmd/mxcli/docker/check_findings_integration_test.go new file mode 100644 index 0000000000..a4e7b57183 --- /dev/null +++ b/cmd/mxcli/docker/check_findings_integration_test.go @@ -0,0 +1,48 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build integration + +package docker + +import ( + "os/exec" + "path/filepath" + "testing" +) + +// TestCheckFindings_RealMx runs CheckFindings against a real mx on a scaffolded +// project: the report must read, its total must match the findings, and asking +// for warnings must add findings, not change how errors read. It skips when no +// mx is resolvable (the CI integration job installs one). +func TestCheckFindings_RealMx(t *testing.T) { + mxPath, err := ResolveMx("") + if err != nil { + t.Skipf("mx not resolvable: %v", err) + } + dir := t.TempDir() + scaffold := exec.Command(mxPath, "create-project") + scaffold.Dir = dir + if out, err := scaffold.CombinedOutput(); err != nil { + t.Skipf("mx create-project failed: %v\n%s", err, out) + } + opts := CheckOptions{ProjectPath: filepath.Join(dir, "App.mpr")} + + errorsOnly, _ := CheckFindings(opts, CheckKinds{}) + if err := errorsOnly.Verify(); err != nil { + t.Fatalf("errors-only report: %v", err) + } + if n := errorsOnly.Count("warning") + errorsOnly.Count("deprecation"); n != 0 { + t.Errorf("%d warnings/deprecations without -w -d", n) + } + + all, _ := CheckFindings(opts, CheckKinds{Warnings: true, Deprecations: true}) + if err := all.Verify(); err != nil { + t.Fatalf("report with warnings: %v", err) + } + if all.Count("error") != errorsOnly.Count("error") { + t.Errorf("asking for warnings changed the error count: %d vs %d", all.Count("error"), errorsOnly.Count("error")) + } + if len(all.Findings) <= len(errorsOnly.Findings) { + t.Errorf("-w -d added no findings (%d vs %d); a fresh project has some", len(all.Findings), len(errorsOnly.Findings)) + } +} diff --git a/cmd/mxcli/docker/check_findings_test.go b/cmd/mxcli/docker/check_findings_test.go index 23b17b5af5..01a8bb95c5 100644 --- a/cmd/mxcli/docker/check_findings_test.go +++ b/cmd/mxcli/docker/check_findings_test.go @@ -3,110 +3,155 @@ package docker import ( + "os" + "path/filepath" "strings" "testing" ) -// The finding lines are the ones mx check printed in the repository's own -// records (docker/check.go's CE0463 comment, the fix-issue findings); the -// closing line is mx's. -const sampleCheck = `Checking project... -[error] [CE0463] "The definition of this widget has changed. ..." at Data grid 2 'dgRequests' -[error] [CE0463] "The definition of this widget has changed. ..." at Data grid 2 'dgRequests' -[error] [CE0117] "Error(s) in expression." at Microflow 'ACT_X' -[error] [CE0066] "Entity access is out of date…" at Domain model of module 'Administration' -[error] [CE0090] "The 'I913.Animal' value should be configured for an outgoing flow." at Object type decision '' -[error] [CE0109] "Variable has no value." -The app contains: 6 errors. -` +// The files in testdata/mxcheck are `mx check --json` output from MxBuild +// 11.16.1 run on a clean `mxcli new` project with one +// mdl-examples/check-gap fixture applied (f10 is the control, which has no +// defect), and two_microflows_same_defect.json is the same defect written in two +// microflows. They are real output, not hand-written. -func TestParseCheckOutput(t *testing.T) { - fs := ParseCheckOutput(sampleCheck) - if len(fs) != 6 { - t.Fatalf("parsed %d findings, want 6: %+v", len(fs), fs) +func load(t *testing.T, name string) CheckReport { + t.Helper() + data, err := os.ReadFile(filepath.Join("testdata", "mxcheck", name)) + if err != nil { + t.Fatal(err) } + r, err := ParseCheckJSON(data) + if err != nil { + t.Fatalf("%s: %v", name, err) + } + if err := r.Verify(); err != nil { + t.Fatalf("%s: %v", name, err) + } + return r +} + +func TestParseCheckJSONFixtures(t *testing.T) { for _, c := range []struct { - i int - code, kind string - name string - class LocationClass - stage int - messagePart string + file, code, document, element string + stage int }{ - {0, "CE0463", "Data grid 2", "dgRequests", Unlocated, 3, "widget has changed"}, - {2, "CE0117", "Microflow", "ACT_X", DocumentOnly, 3, "Error(s) in expression."}, - {3, "CE0066", "Domain model of module", "Administration", DocumentOnly, 3, "Entity access"}, - {4, "CE0090", "Object type decision", "", Unlocated, 3, "'I913.Animal'"}, - {5, "CE0109", "", "", Unlocated, 2, "Variable has no value."}, + {"f01_dupvar.json", "CE0111", "Microflow 'F01'", "Create variable activity 'Create Integer/Long variable'", 1}, + {"f02_badparam.json", "CE1613", "Microflow 'F02'", "Property 'Parameter' of call microflow activity 'Helper'", 0}, + {"f05_undef.json", "CE0109", "Microflow 'F05'", "End event", 2}, + {"f06_assigntype.json", "CE0117", "Microflow 'F06'", "Create variable activity 'Create Integer/Long variable'", 3}, + {"f07_noreturn.json", "CE0109", "Microflow 'F07'", "End event", 2}, + {"f08_rettype.json", "CE0117", "Microflow 'F08'", "End event", 3}, } { - f := fs[c.i] - if f.Code != c.code || f.Kind != c.kind || f.Name != c.name || f.Class != c.class || f.Stage != c.stage || - !strings.Contains(f.Message, c.messagePart) { - t.Errorf("finding %d = %+v, want code %s kind %q name %q class %v stage %d message ~%q", - c.i, f, c.code, c.kind, c.name, c.class, c.stage, c.messagePart) + r := load(t, c.file) + if len(r.Findings) != 1 { + t.Fatalf("%s: %d findings, want 1", c.file, len(r.Findings)) + } + f := r.Findings[0] + l := f.Locations[0] + if f.Severity != "error" || f.Code != c.code || l.DocumentName != c.document || l.ElementName != c.element || + l.ModuleName != "Sp" || f.Stage != c.stage || !f.Identified() { + t.Errorf("%s: got %+v, want %s in %s at %s, stage %d", c.file, f, c.code, c.document, c.element, c.stage) } } } -// A finding on a widget with a code that does name its element is Located; the -// CE0463 override is what makes the same text Unlocated. This is the control for -// the table: without it every widget finding would pass as Unlocated. -func TestClassifyLocatedNeedsAnElementName(t *testing.T) { - f := ParseCheckOutput(`[error] [CE0001] "msg" at Data grid 2 'dgRequests'`)[0] - if f.Class != Located { - t.Errorf("class = %v, want located", f.Class) - } - f = ParseCheckOutput(`[error] [CE0463] "msg" at Data grid 2 'dgRequests'`)[0] - if f.Class != Unlocated { - t.Errorf("CE0463 class = %v, want unlocated", f.Class) +// The control: a project with no defect reports nothing and still verifies. +func TestParseCheckJSONClean(t *testing.T) { + if r := load(t, "f10_clean.json"); len(r.Findings) != 0 || !r.HasDeclared || r.Declared != 0 { + t.Errorf("clean report = %+v", r) } } -// The key ignores the message: the same finding reworded or on another line is -// one finding, which is the point of keying (the control: a different location -// is a different key). -func TestKeyIgnoresMessage(t *testing.T) { - a := ParseCheckOutput(`[error] [CE0117] "one wording" at Microflow 'ACT_X'`)[0] - b := ParseCheckOutput(`[error] [CE0117] "another wording" at Microflow 'ACT_X'`)[0] - c := ParseCheckOutput(`[error] [CE0117] "one wording" at Microflow 'ACT_Y'`)[0] - if a.Key() != b.Key() { - t.Error("the same finding with a different message must have the same key") - } - if a.Key() == c.Key() { - t.Error("a different location must have a different key") +// The reason findings are read from the JSON: two microflows with the same defect +// print the same text line, and only the IDs tell them apart. +func TestSameDefectInTwoMicroflowsHasTwoKeys(t *testing.T) { + r := load(t, "two_microflows_same_defect.json") + if len(r.Findings) != 2 { + t.Fatalf("%d findings, want 2", len(r.Findings)) + } + a, b := r.Findings[0], r.Findings[1] + if a.Code != b.Code || a.Message != b.Message || a.Locations[0].ElementName != b.Locations[0].ElementName { + t.Fatalf("the two findings are not textually identical: %+v / %+v", a, b) + } + if a.Key() == b.Key() { + t.Error("two findings in different microflows must have different keys") + } + if a.Locations[0].DocumentName == b.Locations[0].DocumentName { + t.Error("the documents must differ") } } -func TestStageOf(t *testing.T) { - for code, want := range map[string]int{"CE1613": 0, "CE0111": 1, "CE0109": 2, "CE0067": 3, "CE9999": 3} { - if got := StageOf(code); got != want { - t.Errorf("StageOf(%s) = %d, want %d", code, got, want) - } +// The key ignores the message and the names: the same finding reworded or in a +// renamed document is one finding. A different element is a different key. +func TestKeyIsIDsNotText(t *testing.T) { + base := CheckFinding{Severity: "error", Code: "CE0109", Message: "one", Locations: []CheckLocation{ + {DocumentName: "Microflow 'A'", ElementName: "End event", UnitID: "u1", ElementID: "e1"}}} + reworded := base + reworded.Message = "another" + reworded.Locations = []CheckLocation{{DocumentName: "Microflow 'Renamed'", ElementName: "End", UnitID: "u1", ElementID: "e1"}} + other := base + other.Locations = []CheckLocation{{UnitID: "u1", ElementID: "e2"}} + if base.Key() != reworded.Key() { + t.Error("message and names must not be part of the key") + } + if base.Key() == other.Key() { + t.Error("a different element must be a different key") } } -func TestParseCheckReportVerify(t *testing.T) { - r := ParseCheckReport(sampleCheck) - if !r.HasDeclared || r.Declared != 6 { - t.Fatalf("declared = %d (%v), want 6", r.Declared, r.HasDeclared) +// Warnings and deprecations come with -w -d: the file has 16 warnings and one +// deprecation on an empty project. Some have no location at all. +func TestParseCheckJSONAllKinds(t *testing.T) { + r := load(t, "f05_undef_all_kinds.json") + if r.Count("error") != 1 || r.Count("warning") != 16 || r.Count("deprecation") != 1 { + t.Errorf("counts = %d/%d/%d", r.Count("error"), r.Count("warning"), r.Count("deprecation")) } - if err := r.Verify(); err != nil { - t.Errorf("Verify() = %v on a complete report", err) + unidentified := 0 + for _, f := range r.Findings { + if !f.Identified() { + unidentified++ + } } + if unidentified != 1 { + t.Errorf("%d findings without an ID, want 1", unidentified) + } +} - // Control: a format change that loses lines must be reported, not absorbed. - lost := strings.Replace(sampleCheck, "[error] [CE0117]", "[err] [CE0117]", 1) - if err := ParseCheckReport(lost).Verify(); err == nil { - t.Error("Verify() = nil although a finding line was not parsed") +// The control for Verify: a list the reader does not know about, or a lost +// finding, must be reported rather than absorbed. +func TestVerifyCatchesAMismatch(t *testing.T) { + data, err := os.ReadFile(filepath.Join("testdata", "mxcheck", "f05_undef.json")) + if err != nil { + t.Fatal(err) + } + lost := strings.Replace(string(data), `"total-problems": 1`, `"total-problems": 2`, 1) + r, err := ParseCheckJSON([]byte(lost)) + if err != nil { + t.Fatal(err) + } + if r.Verify() == nil { + t.Error("Verify() = nil although mx declared a problem the report does not have") + } + if _, err := ParseCheckJSON([]byte(`{"serialization_version": 2}`)); err == nil { + t.Error("an unknown serialization_version must be refused") + } + if _, err := ParseCheckJSON([]byte(`not json`)); err == nil { + t.Error("invalid json must be an error") } - if err := ParseCheckReport("[error] [CE0117] \"x\" at Microflow 'M'\n").Verify(); err == nil { - t.Error("Verify() = nil without mx's closing count") +} + +func TestStageOf(t *testing.T) { + for code, want := range map[string]int{"CE1613": 0, "CE0111": 1, "CE0109": 2, "CE0067": 3, "CE9999": 3} { + if got := StageOf(code); got != want { + t.Errorf("StageOf(%s) = %d, want %d", code, got, want) + } } } -func TestParseCheckOutputKeepsSeverityAndCRLF(t *testing.T) { - fs := ParseCheckOutput("[warning] [CW0001] \"w\" at Page 'P'\r\n[deprecation] [CD0001] \"d\"\r\n") - if len(fs) != 2 || fs[0].Severity != "warning" || fs[1].Severity != "deprecation" { - t.Fatalf("got %+v", fs) +func TestCheckKindsArgs(t *testing.T) { + got := strings.Join(CheckKinds{Warnings: true, BestPractice: true}.args("/x.json"), " ") + if got != "-j /x.json -w -b" { + t.Errorf("args = %q", got) } } diff --git a/cmd/mxcli/docker/testdata/mxcheck/f01_dupvar.json b/cmd/mxcli/docker/testdata/mxcheck/f01_dupvar.json new file mode 100644 index 0000000000..7429d557b6 --- /dev/null +++ b/cmd/mxcli/docker/testdata/mxcheck/f01_dupvar.json @@ -0,0 +1,22 @@ +{ + "serialization_version": 1, + "errors": [ + { + "code": "CE0111", + "message": "Duplicate variable name 'X'.", + "locations": [ + { + "module-id": "a7b26067-8ce2-4841-b1fb-4180ab9ee886", + "module-name": "Sp", + "document-name": "Microflow 'F01'", + "element-name": "Create variable activity 'Create Integer/Long variable'", + "element-id": "82de161a-0a36-4fc1-89e0-d3f74f7ef044", + "unit-id": "7c6de221-9a5a-4fba-b952-149302ae85b7" + } + ] + } + ], + "warnings": [], + "deprecations": [], + "total-problems": 1 +} \ No newline at end of file diff --git a/cmd/mxcli/docker/testdata/mxcheck/f02_badparam.json b/cmd/mxcli/docker/testdata/mxcheck/f02_badparam.json new file mode 100644 index 0000000000..142a2f4382 --- /dev/null +++ b/cmd/mxcli/docker/testdata/mxcheck/f02_badparam.json @@ -0,0 +1,22 @@ +{ + "serialization_version": 1, + "errors": [ + { + "code": "CE1613", + "message": "The selected parameter 'Sp.Helper.Wrong' no longer exists.", + "locations": [ + { + "module-id": "9aed1f7d-e017-4aa8-b6c7-0eb988a9d8c6", + "module-name": "Sp", + "document-name": "Microflow 'F02'", + "element-name": "Property 'Parameter' of call microflow activity 'Helper'", + "element-id": "b83b2d08-143f-4e80-befb-556f0640cd4f", + "unit-id": "c457ea06-b527-485e-85be-b04d5ee30490" + } + ] + } + ], + "warnings": [], + "deprecations": [], + "total-problems": 1 +} \ No newline at end of file diff --git a/cmd/mxcli/docker/testdata/mxcheck/f05_undef.json b/cmd/mxcli/docker/testdata/mxcheck/f05_undef.json new file mode 100644 index 0000000000..5408fc9930 --- /dev/null +++ b/cmd/mxcli/docker/testdata/mxcheck/f05_undef.json @@ -0,0 +1,22 @@ +{ + "serialization_version": 1, + "errors": [ + { + "code": "CE0109", + "message": "Undefined variable 'Nope'.", + "locations": [ + { + "module-id": "eb9312a5-53fc-4290-9527-dcb445b6a024", + "module-name": "Sp", + "document-name": "Microflow 'F05'", + "element-name": "End event", + "element-id": "389d91af-39cb-4e5c-8dd1-4e4864cc05c8", + "unit-id": "edda7090-416b-4db5-ba81-aada9115570c" + } + ] + } + ], + "warnings": [], + "deprecations": [], + "total-problems": 1 +} \ No newline at end of file diff --git a/cmd/mxcli/docker/testdata/mxcheck/f05_undef_all_kinds.json b/cmd/mxcli/docker/testdata/mxcheck/f05_undef_all_kinds.json new file mode 100644 index 0000000000..622eb6e528 --- /dev/null +++ b/cmd/mxcli/docker/testdata/mxcheck/f05_undef_all_kinds.json @@ -0,0 +1,253 @@ +{ + "serialization_version": 1, + "errors": [ + { + "code": "CE0109", + "message": "Undefined variable 'Nope'.", + "locations": [ + { + "module-id": "eb9312a5-53fc-4290-9527-dcb445b6a024", + "module-name": "Sp", + "document-name": "Microflow 'F05'", + "element-name": "End event", + "element-id": "389d91af-39cb-4e5c-8dd1-4e4864cc05c8", + "unit-id": "edda7090-416b-4db5-ba81-aada9115570c" + } + ] + } + ], + "warnings": [ + { + "code": "CW0055", + "message": "No 'On click' action specified.", + "locations": [ + { + "module-id": "0e80ce99-3bb6-4e88-bd3e-a1dafd3a33bf", + "module-name": "Atlas_Core", + "document-name": "Menu 'Tablet_Menu'", + "element-name": "Menu item", + "element-id": "59cfb12a-0e84-4389-b89d-f48e154b8711", + "unit-id": "139b69e0-8cad-4068-810c-a51a1a118c5e" + } + ] + }, + { + "code": "CW0055", + "message": "No 'On click' action specified.", + "locations": [ + { + "module-id": "0e80ce99-3bb6-4e88-bd3e-a1dafd3a33bf", + "module-name": "Atlas_Core", + "document-name": "Menu 'Tablet_Menu'", + "element-name": "Menu item", + "element-id": "00803845-4eaf-429a-8dfc-a191dbedf1c1", + "unit-id": "139b69e0-8cad-4068-810c-a51a1a118c5e" + } + ] + }, + { + "code": "CW0055", + "message": "No 'On click' action specified.", + "locations": [ + { + "module-id": "0e80ce99-3bb6-4e88-bd3e-a1dafd3a33bf", + "module-name": "Atlas_Core", + "document-name": "Menu 'Tablet_Menu'", + "element-name": "Menu item", + "element-id": "c56a8f99-7b88-4f61-a77f-4ad031a6228b", + "unit-id": "139b69e0-8cad-4068-810c-a51a1a118c5e" + } + ] + }, + { + "code": "CW0055", + "message": "No 'On click' action specified.", + "locations": [ + { + "module-id": "0e80ce99-3bb6-4e88-bd3e-a1dafd3a33bf", + "module-name": "Atlas_Core", + "document-name": "Menu 'Tablet_Menu'", + "element-name": "Menu item", + "element-id": "4493978e-f5e5-4122-922c-831e1c4606f5", + "unit-id": "139b69e0-8cad-4068-810c-a51a1a118c5e" + } + ] + }, + { + "code": "CE0635", + "message": "Widgets used in a page template should not contain references to other documents", + "locations": [ + { + "module-id": "eb218b30-f187-4700-bcb6-2150e51c222c", + "module-name": "Atlas_Web_Content", + "document-name": "Page template 'Tablet_Login'", + "element-name": "Nanoflow of action button 'actionButton3'", + "element-id": "14731960-063c-4fac-a3e9-6130783a69ea", + "unit-id": "ec6cea43-013e-4e12-b582-eea26db381be" + } + ] + }, + { + "code": "CE0635", + "message": "Widgets used in a page template should not contain references to other documents", + "locations": [ + { + "module-id": "eb218b30-f187-4700-bcb6-2150e51c222c", + "module-name": "Atlas_Web_Content", + "document-name": "Page template 'Tablet_Login'", + "element-name": "Attribute of text 'text2'", + "element-id": "d5bf11f9-f38d-47e9-b4f1-1675e1c3ef8a", + "unit-id": "ec6cea43-013e-4e12-b582-eea26db381be" + } + ] + }, + { + "code": "CE0635", + "message": "Widgets used in a page template should not contain references to other documents", + "locations": [ + { + "module-id": "eb218b30-f187-4700-bcb6-2150e51c222c", + "module-name": "Atlas_Web_Content", + "document-name": "Page template 'Tablet_Login'", + "element-name": "Expression of text 'text2'", + "element-id": "d5bf11f9-f38d-47e9-b4f1-1675e1c3ef8a", + "unit-id": "ec6cea43-013e-4e12-b582-eea26db381be" + } + ] + }, + { + "code": "CE0635", + "message": "Widgets used in a page template should not contain references to other documents", + "locations": [ + { + "module-id": "eb218b30-f187-4700-bcb6-2150e51c222c", + "module-name": "Atlas_Web_Content", + "document-name": "Page template 'Tablet_Login'", + "element-name": "Nanoflow of data view 'dataView1'", + "element-id": "576328bf-fdf7-4018-9e9f-1e50a7d65a22", + "unit-id": "ec6cea43-013e-4e12-b582-eea26db381be" + } + ] + }, + { + "code": "CW0263", + "message": "Empty template. [English, United States]", + "locations": [ + { + "module-id": "eb218b30-f187-4700-bcb6-2150e51c222c", + "module-name": "Atlas_Web_Content", + "document-name": "Nanoflow 'ACT_Login'", + "element-name": "Validation feedback activity 'Show validation message on member Username of LoginContext'", + "element-id": "534cd30f-f1e1-461e-b614-8d04ce6d74bb", + "unit-id": "e011257a-9a48-4eb1-9784-01eb32254977" + } + ] + }, + { + "code": "CW0263", + "message": "Empty template. [English, United States]", + "locations": [ + { + "module-id": "eb218b30-f187-4700-bcb6-2150e51c222c", + "module-name": "Atlas_Web_Content", + "document-name": "Nanoflow 'ACT_Login'", + "element-name": "Validation feedback activity 'Show validation message on member Password of LoginContext'", + "element-id": "908179c7-775d-4125-816a-95d73e5fe8aa", + "unit-id": "e011257a-9a48-4eb1-9784-01eb32254977" + } + ] + }, + { + "code": "CW0055", + "message": "No 'On click' action specified.", + "locations": [ + { + "module-id": "0e80ce99-3bb6-4e88-bd3e-a1dafd3a33bf", + "module-name": "Atlas_Core", + "document-name": "Menu 'Phone_Menu'", + "element-name": "Menu item", + "element-id": "7c8453cd-e738-4a81-a3ed-58af75d1b67f", + "unit-id": "05b4a901-e7fc-40a5-9380-7a27c8924f68" + } + ] + }, + { + "code": "CW0055", + "message": "No 'On click' action specified.", + "locations": [ + { + "module-id": "0e80ce99-3bb6-4e88-bd3e-a1dafd3a33bf", + "module-name": "Atlas_Core", + "document-name": "Menu 'Phone_Menu'", + "element-name": "Menu item", + "element-id": "ce19400e-df53-41ec-b3f2-92edf09a74d3", + "unit-id": "05b4a901-e7fc-40a5-9380-7a27c8924f68" + } + ] + }, + { + "code": "CW0055", + "message": "No 'On click' action specified.", + "locations": [ + { + "module-id": "0e80ce99-3bb6-4e88-bd3e-a1dafd3a33bf", + "module-name": "Atlas_Core", + "document-name": "Menu 'Phone_Menu'", + "element-name": "Menu item", + "element-id": "ac9672c9-e4f8-4d88-a4e1-1fb860b4187b", + "unit-id": "05b4a901-e7fc-40a5-9380-7a27c8924f68" + } + ] + }, + { + "code": "CW0055", + "message": "No 'On click' action specified.", + "locations": [ + { + "module-id": "0e80ce99-3bb6-4e88-bd3e-a1dafd3a33bf", + "module-name": "Atlas_Core", + "document-name": "Menu 'Phone_Menu'", + "element-name": "Menu item", + "element-id": "40cd400c-7e6d-4fd4-9c0f-19f3b1fffc37", + "unit-id": "05b4a901-e7fc-40a5-9380-7a27c8924f68" + } + ] + }, + { + "code": "CE0582", + "message": "Widget static image is not supported in React client. Right-click this error to convert it to an alternative widget type.", + "locations": [ + { + "module-id": "eb218b30-f187-4700-bcb6-2150e51c222c", + "module-name": "Atlas_Web_Content", + "document-name": "Page template 'Detail_Timeline'", + "element-name": "Static image 'image6'", + "element-id": "87954738-a1ca-4983-9236-92d25dd4a7ec", + "unit-id": "23a822d9-94fd-44ed-98bc-7f7b9f7bd425" + } + ] + }, + { + "code": "CE4271", + "message": "Version '6.0.0' of the module 'NanoflowCommons' is not compatible with this Mendix version. Visit the Marketplace to check for updates of 'Nanoflow Commons'.", + "locations": [] + } + ], + "deprecations": [ + { + "code": "CW2512", + "message": "Java versions below 25 are deprecated for deployment.", + "locations": [ + { + "module-id": "", + "module-name": "-", + "document-name": "Settings", + "element-name": "Runtime settings", + "element-id": "239a8165-5256-533c-b055-300e409a5001", + "unit-id": "4e70063d-53e1-477a-ab92-6b763330c293" + } + ] + } + ], + "total-problems": 18 +} \ No newline at end of file diff --git a/cmd/mxcli/docker/testdata/mxcheck/f06_assigntype.json b/cmd/mxcli/docker/testdata/mxcheck/f06_assigntype.json new file mode 100644 index 0000000000..e94a2e5ba0 --- /dev/null +++ b/cmd/mxcli/docker/testdata/mxcheck/f06_assigntype.json @@ -0,0 +1,22 @@ +{ + "serialization_version": 1, + "errors": [ + { + "code": "CE0117", + "message": "Error(s) in expression.", + "locations": [ + { + "module-id": "2245e1df-c521-4097-8c6f-6e4338b89ee0", + "module-name": "Sp", + "document-name": "Microflow 'F06'", + "element-name": "Create variable activity 'Create Integer/Long variable'", + "element-id": "d2ca8f45-a346-4389-971c-192978cdab16", + "unit-id": "0ddb128d-2278-4bb6-ad13-3d7ead1b9ac3" + } + ] + } + ], + "warnings": [], + "deprecations": [], + "total-problems": 1 +} \ No newline at end of file diff --git a/cmd/mxcli/docker/testdata/mxcheck/f07_noreturn.json b/cmd/mxcli/docker/testdata/mxcheck/f07_noreturn.json new file mode 100644 index 0000000000..a0ce8ffc3f --- /dev/null +++ b/cmd/mxcli/docker/testdata/mxcheck/f07_noreturn.json @@ -0,0 +1,22 @@ +{ + "serialization_version": 1, + "errors": [ + { + "code": "CE0109", + "message": "Undefined variable 'R'.", + "locations": [ + { + "module-id": "97100ad9-1b61-4b02-bb15-8529f694f829", + "module-name": "Sp", + "document-name": "Microflow 'F07'", + "element-name": "End event", + "element-id": "54894d79-d85d-4c2b-9e3c-07e4bdc8de89", + "unit-id": "d6cab120-5978-4190-8756-8e54d5ea41e5" + } + ] + } + ], + "warnings": [], + "deprecations": [], + "total-problems": 1 +} \ No newline at end of file diff --git a/cmd/mxcli/docker/testdata/mxcheck/f08_rettype.json b/cmd/mxcli/docker/testdata/mxcheck/f08_rettype.json new file mode 100644 index 0000000000..dbd8b1928c --- /dev/null +++ b/cmd/mxcli/docker/testdata/mxcheck/f08_rettype.json @@ -0,0 +1,22 @@ +{ + "serialization_version": 1, + "errors": [ + { + "code": "CE0117", + "message": "Error(s) in expression.", + "locations": [ + { + "module-id": "1b5e1ab0-ec94-411f-8f4c-dd5e0febbcd9", + "module-name": "Sp", + "document-name": "Microflow 'F08'", + "element-name": "End event", + "element-id": "6ea0946e-4b62-48ea-a9b5-ffb2f73cec7b", + "unit-id": "c6917576-1558-4e91-9f6e-9a37b9c41d66" + } + ] + } + ], + "warnings": [], + "deprecations": [], + "total-problems": 1 +} \ No newline at end of file diff --git a/cmd/mxcli/docker/testdata/mxcheck/f10_clean.json b/cmd/mxcli/docker/testdata/mxcheck/f10_clean.json new file mode 100644 index 0000000000..ee571bca47 --- /dev/null +++ b/cmd/mxcli/docker/testdata/mxcheck/f10_clean.json @@ -0,0 +1,7 @@ +{ + "serialization_version": 1, + "errors": [], + "warnings": [], + "deprecations": [], + "total-problems": 0 +} \ No newline at end of file diff --git a/cmd/mxcli/docker/testdata/mxcheck/two_microflows_same_defect.json b/cmd/mxcli/docker/testdata/mxcheck/two_microflows_same_defect.json new file mode 100644 index 0000000000..4c6b02763f --- /dev/null +++ b/cmd/mxcli/docker/testdata/mxcheck/two_microflows_same_defect.json @@ -0,0 +1,36 @@ +{ + "serialization_version": 1, + "errors": [ + { + "code": "CE0109", + "message": "Undefined variable 'Nope'.", + "locations": [ + { + "module-id": "fec11de2-45bc-4f34-8747-6c2876c30c31", + "module-name": "Sp", + "document-name": "Microflow 'A'", + "element-name": "End event", + "element-id": "2f4214f6-fdf0-487f-bef2-b6f53397892a", + "unit-id": "e3936f55-e0a5-4925-837d-01703b9aee19" + } + ] + }, + { + "code": "CE0109", + "message": "Undefined variable 'Nope'.", + "locations": [ + { + "module-id": "fec11de2-45bc-4f34-8747-6c2876c30c31", + "module-name": "Sp", + "document-name": "Microflow 'B'", + "element-name": "End event", + "element-id": "7bce67dc-c225-42b7-b280-09b26004f9b0", + "unit-id": "8f2fd632-74fe-4d5c-99fd-abe7d482ff03" + } + ] + } + ], + "warnings": [], + "deprecations": [], + "total-problems": 2 +} \ No newline at end of file diff --git a/cmd/mxcli/syntax/features_integration.go b/cmd/mxcli/syntax/features_integration.go index b4b10b47ea..5e6a8b7cbc 100644 --- a/cmd/mxcli/syntax/features_integration.go +++ b/cmd/mxcli/syntax/features_integration.go @@ -500,7 +500,10 @@ func init() { " -- A name may be schema-qualified, and must be where two\n" + " -- schemas hold the same one. Each generated QUERY selects\n" + " -- FROM the schema the object was discovered in, so it runs\n" + - " -- wherever the connection's search path points.", + " -- wherever the connection's search path points.\n" + + " -- Either clause scopes the whole command: with neither, every\n" + + " -- table and view is generated; with either, only what it names.\n" + + " -- An object named twice is generated once.", Example: "SQL source GENERATE CONNECTOR INTO HRModule;\nSQL source GENERATE CONNECTOR INTO HRModule TABLES (employees, hr.departments) EXEC;", SeeAlso: []string{"sql", "sql.connect"}, }) diff --git a/docs-site/src/migration/README.md b/docs-site/src/migration/README.md index 85783f9114..0d920d6f01 100644 --- a/docs-site/src/migration/README.md +++ b/docs-site/src/migration/README.md @@ -30,4 +30,4 @@ Traditional migrations require deep Mendix expertise and manual work in Studio P | [Transformation](transformation.md) | Map source elements to Mendix, plan module structure | Assessment output as input | | [Generation](generation.md) | Generate domain model, microflows, pages, security | `generate-domain-model`, `write-microflows`, `create-page`, `manage-security` | | [Data Migration](data-migration.md) | Import data from source databases | `demo-data`, `database-connections` | -| [Validation and Handoff](validation.md) | Test, lint, and hand off to Studio Pro | `check-syntax`, `assess-quality` | +| [Validation and Handoff](validation.md) | Test, lint, and hand off to Studio Pro | `check-syntax`, `report-quality`, `assess-quality` | diff --git a/docs-site/src/migration/validation.md b/docs-site/src/migration/validation.md index c475b5f1bc..b150784eba 100644 --- a/docs-site/src/migration/validation.md +++ b/docs-site/src/migration/validation.md @@ -39,7 +39,7 @@ The linter covers 6 categories: ## Quality Reports -The `assess-quality` skill guides the agent to run `mxcli report` and interpret the scores. A migration is ready for handoff when: +The `report-quality` skill guides the agent to run `mxcli report` and interpret the scores; `assess-quality` adds the manual review no rule can do. A migration is ready for handoff when: - No critical lint issues (SEC, QUAL categories) - Quality score above 70 across all categories diff --git a/docs-site/src/tools/mxcli-report.md b/docs-site/src/tools/mxcli-report.md index 0cbff7864d..0d99d30b51 100644 --- a/docs-site/src/tools/mxcli-report.md +++ b/docs-site/src/tools/mxcli-report.md @@ -100,13 +100,19 @@ mxcli report -p app.mpr --format json --output report.json Use the JSON format to fail CI pipelines when scores drop below a threshold: ```bash -# Check if overall score is above 70 -SCORE=$(mxcli report -p app.mpr --format json | jq '.overallScore') -if [ "$SCORE" -lt 70 ]; then - echo "Quality score $SCORE is below threshold 70" +# Fail when the overall score is below 70. The score has one decimal (e.g. 82.5), +# so compare in jq: the shell's -lt only takes integers. +mxcli report -p app.mpr --format json -o quality.json +jq -e '.overallScore >= 70' quality.json > /dev/null || { + echo "Quality score $(jq .overallScore quality.json) is below threshold 70" exit 1 -fi +} ``` +The `report-quality` skill covers how the score is computed, what it leaves out +(Mendix's own consistency check; the bundled Starlark rules when the project has no +`.claude/lint-rules/`), and how to accept an intended finding in the project brain. + For reports that answer one question instead of scoring — architecture, tests, -security, performance and traceability — see [Project Reports](project-reports.md). +security, performance, traceability and the software map — see +[Project Reports](project-reports.md). diff --git a/docs-site/src/tools/project-reports.md b/docs-site/src/tools/project-reports.md index 2c109cdc99..7a4b864993 100644 --- a/docs-site/src/tools/project-reports.md +++ b/docs-site/src/tools/project-reports.md @@ -176,3 +176,5 @@ report's unused list. Each kind has a skill with the judgement the command cannot make and the traps found building the reports by hand: `report-architecture`, `report-tests`, `report-security`, `report-performance`, `report-traceability`, `report-map`. +The scored report has `report-quality`, and `assess-quality` combines all of them +with the manual review no rule can do. diff --git a/docs/11-proposals/PROPOSAL_verify_gate.md b/docs/11-proposals/PROPOSAL_verify_gate.md index fdd6255a50..16286433ab 100644 --- a/docs/11-proposals/PROPOSAL_verify_gate.md +++ b/docs/11-proposals/PROPOSAL_verify_gate.md @@ -137,11 +137,12 @@ mxcli verify -p app.mpr [--base | --baseline ] **Update 2026-11-19.** Since this section was written, `main` gained about twenty `check` validations (step 0: MDL097/MDL098, the `replace()` fix, DECLARE/SET/RETURN type checks, and others), the `mdl-examples/check-gap/` parity harness, -`scripts/diagcoverage` and `scripts/maskmatrix`. None of it parses `mx check` -output into findings: the harness compares codes in shell and `diagcoverage` reads -only the code catalog, so **step 1 is still needed**. What the work adds is -fixtures to build the parser's golden files from, and two measurements that change -the design (next two sections). +`scripts/diagcoverage` and `scripts/maskmatrix`. None of it read `mx check` +output into findings (the harness compares codes in shell, `diagcoverage` reads +only the code catalog); step 1 (`docker.ParseCheckJSON`) now does, from +`mx check --json`. The work also supplied the fixtures for its golden files and +two measurements that change the design (next sections). The gaps listed below +predate step 1 and the `docker.Check` one is closed for the stage's purposes. - `mdl/linter/baseline.go` has `WriteBaseline`/`LoadBaseline`/`Compare`, but it is **not reusable as is** against the `Finding` below: @@ -211,8 +212,7 @@ Rules that follow: findings the staging may still be holding back. - A new stage 0 or earlier-stage finding always fails the gate, and the `hidden` count is printed next to it so a reader does not read "400 resolved". -- Each finding carries its **stage** (the classification table of step 1 gets a - stage column). A code not in the measured set is treated as stage 3 (hides +- Each finding carries its **stage** (`StageOf`, from the measured table). A code not in the measured set is treated as stage 3 (hides nothing, hidden by earlier stages): the conservative reading for `unmasked`, and recorded as a limit until `maskmatrix` covers more codes. The measurement covers microflows and pages only. @@ -240,26 +240,37 @@ authoritative stage. Design consequences: - Off by default; revisit once there is evidence about its false-positive rate on real projects. -## Finding identity: for which codes can the key be built at all? - -The key `(code, document, element path)` is only as good as `mx check`'s text. In -`recheckCE0463` (`docker/check.go`): *"mx check's text output does not name a -CE0463's page, so two same-named widgets on different pages print the same line; -the lines are compared as counts."* For CE0463 the element identity is **not in -the output**. Step 1 must therefore classify each code the parser sees: - -1. **Located** — the line names document and element (e.g. CE0109 on a microflow - activity). Key = `(code, document, path)`; `resolved` is a list. -2. **Document-only** — names the document but not the element. Key = - `(code, document)` with multiplicity; matching is by count per key. -3. **Unlocated** — names neither (CE0463). Fallback: **count-based matching per - distinct line**, exactly as `recheckCE0463` does today; `resolved` is a count - delta, never a list, and `--format json` says so. - -The CE0463 stale/persisting labelling from the re-check must survive unchanged as -a label on class 3 findings. The first question for step 1 is the classification -table (code → class), built from real `mx check` output; stability under rename -or move (formerly the only question) applies only to class 1. +## Finding identity: read `mx check --json`, not its text + +An earlier draft keyed findings on `(code, document, element path)` parsed from +the text, and asked for which codes that could be built. Run against real MxBuild +11.16.1 (the `mdl-examples/check-gap/` fixtures; the output is committed as +`cmd/mxcli/docker/testdata/mxcheck/`), the premise was wrong: + +- **The text never names the document of a flow element.** A finding reads + `[error] [CE0109] "Undefined variable 'Nope'." at End event`. The same defect in + two microflows prints two identical lines. A default caption such as `Create + variable activity 'Create Integer/Long variable'` is not unique either. No + classification table over the text can fix that. +- **`mx check --json ` carries the identity.** Each problem has + `module-name`, `document-name` (`Microflow 'F05'`), `element-name`, + `element-id` and `unit-id`, and the two microflows above differ in both IDs. The + IDs were identical across two runs and a copy of the project. `-w -d -b` add + warnings, deprecations and best-practice findings (a fresh project already has + 16 warnings and 1 deprecation); the file declares `total-problems`, which the + reader checks against what it read. + +So the `mxcheck` stage **reads the JSON** and keys a finding on severity, code and +the `(unit-id, element-id)` of its locations; names and messages are display only +and may change without making a finding new. A finding with no location (the +runtime-settings deprecation) has nothing to key on but its code and is matched by +count, as `recheckCE0463` does today. + +Not yet measured: a CE0463 (the widget case that motivated the count fallback) +could not be produced in the spike, so whether its JSON carries IDs is **open**. +`recheckCE0463` keeps its text-based comparison until it is. Stability of the IDs +across a Studio Pro module update or an `mxcli` rewrite is also unmeasured; they +are `$ID`s, which `canon` carries across writes but Studio Pro renumbers. ## Design @@ -276,9 +287,9 @@ type Stage interface { } ``` -- `Finding.Key()` is **per class** (see Finding identity): `Stage + Code + Document + Path` for located findings, down to count-per-line for unlocated ones. The lint stage's key additionally uses `ID` when present, as `baselineKey` does. Message is display only except where a stage's existing matcher (lint `Compare`) uses it, and then that is stated. -- Shared **`mx check` output parser** (`docker.ParseCheckOutput`) producing - `[]Finding` from `mx` lines (`ERROR [CE0109] "…" at Microflow 'M.F', Activity …`). +- `Finding.Key()` is built from IDs (see Finding identity): `Stage + Code + UnitID/ElementID`, by count for a finding with no location. The lint stage's key additionally uses `ID` when present, as `baselineKey` does. Message is display only except where a stage's existing matcher (lint `Compare`) uses it, and then that is stated. +- Shared **`mx check --json` reader** (`docker.ParseCheckJSON`, `docker.CheckFindings`) + producing `[]Finding` with module, document and element IDs. Also unblocks #1313 and #1312; lands as its own PR. - Baseline file is versioned JSON, sorted for stable diffs, one section per stage (lint reuses its existing format inside it or keeps its own file — see questions). @@ -298,17 +309,18 @@ type Stage interface { `PROPOSAL_check_mxbuild_gap_heuristics.md`. Already under way (each validation moved into `mxcli check` removes a class of late failures); independent of, and not blocked by, the steps below. -1. **Structured `mx check` findings**: parser + `docker.CheckFindings()`, tests - against recorded `mx` output, built from the `mdl-examples/check-gap/` - fixtures. Includes the code classification table with its **stage** column. - Standalone PR; unblocks the `mxcheck` baseline, #1313 and #1312. +1. **Structured `mx check` findings** (done): `docker.ParseCheckJSON` and + `docker.CheckFindings`, reading `mx check --json`, with golden files from a real + run of the `mdl-examples/check-gap/` fixtures and a control (no defect). An + earlier text parser was replaced because the text does not identify the + document (see Finding identity). Remaining: a CE0463 golden file, and wiring the + `mxcheck` stage to it in step 2. 2. **`mxcli verify`, first release**: stages `check`, `lint`, `mxcheck`; `--format`, exit-code table, `--fail-fast`; and a **`lint` baseline from day one** (`--baseline`, `--update-baseline`, reusing `mdl/linter/baseline.go` through the adapter described above), so the first run on an existing project - reports only what is new. The `mxcheck` stage baselines once step 1 lands, with the four-state - classification above; until - then it reports all findings and says it is unbaselined. + reports only what is new. The `mxcheck` stage reads `mx check --json` (step 1, done) and baselines with the + four-state classification above. 3. **Make `verify` the thing everything points at**: skills, `mxcli init` output and the project `CLAUDE.md` guidance name `verify` as the single "did my change break anything?" command, instead of listing six commands. Ships with step 2, @@ -320,15 +332,15 @@ type Stage interface { ## Testing -- Unit: finding key normalisation per class (located / document-only / unlocated) (same finding, different line/message → same - key); classification new/resolved/pre-existing; default run-all vs `--fail-fast`. +- Unit: finding keys are IDs, not text (same finding, different message or name → same + key; same text, different element → different key); classification new/resolved/pre-existing; default run-all vs `--fail-fast`. - Staging: for each stage pair from `maskmatrix`, a fixture diff classified unmasked (not new) when the baseline hid it and hidden (not resolved) when the change introduces the hiding finding; **control:** the same finding with no hiding baseline finding must be new and must fail the gate. - Checker identity: a baseline from one checker compared against the other is an error. -- Parser: golden `mx check` outputs (CE0109, CE0844, CE7811, CE0463 labelled). +- Reader: golden `mx check --json` files from real runs (committed under `cmd/mxcli/docker/testdata/mxcheck/`), the same defect in two microflows (two keys), a clean control, a lost-finding control for `Verify`, and an integration test against a real `mx`. - Control (per CLAUDE.md): a "nothing new" test paired with one that injects a finding and must fail. - Output purity row in `json_output_purity_test.go`; exit-code table incl. several failing stages at once and "could not run". @@ -343,9 +355,8 @@ type Stage interface { `verify`, under a namespace.) 2. **One baseline file or one per stage?** One file is simpler for agents; per-stage keeps `lint --baseline` files valid unchanged. -3. **Code classification** (step 1): which `mx check` codes are located, - document-only, or unlocated, and is count-based matching acceptable for the - unlocated ones? +3. **ID stability.** Are `unit-id` / `element-id` stable across a Studio Pro + module update and an `mxcli` rewrite, and does a CE0463's JSON carry them? 4. **`--base` for MPR v1** — extract via `git show` of the single `.mpr`, or require v2? 5. **Should `check` stage fail on warnings,** or only errors? 6. **Exit-code encoding** — bitmask offsets, and whether to align with `check`'s diff --git a/mdl-examples/bug-tests/mapping-handler-param-type-mismatch.fail.mdl b/mdl-examples/bug-tests/mapping-handler-param-type-mismatch.fail.mdl new file mode 100644 index 0000000000..10d09dda6c --- /dev/null +++ b/mdl-examples/bug-tests/mapping-handler-param-type-mismatch.fail.mdl @@ -0,0 +1,39 @@ +mdl 1; +-- NEGATIVE TEST — `mxcli check` must REFUSE this file (MDL107), with no project. +-- +-- A custom object handler (`find … by M.MF ( P: member )`) that feeds a JSON member to a +-- microflow parameter of an incompatible type. `check` and `exec` accepted it, and the +-- build reported +-- +-- [CE0282] "The microflow parameter 'P' is of type 'Integer'. It does not match the +-- specified type 'String'." +-- +-- Measured on MxBuild 11.16.1, one mapping per project: an Integer parameter fed a String, +-- Decimal, Boolean or DateTime member, a Decimal parameter fed a String member, and so on — +-- see mapping-handler-types.mdl for the pairings that build clean. +create persistent entity MyFirstModule.HtRoot (Name: string(50)); +create persistent entity MyFirstModule.HtChild (Code: string(50)); +create association MyFirstModule.HtChild_HtRoot from MyFirstModule.HtChild to MyFirstModule.HtRoot type reference; +create json structure MyFirstModule.JS_Ht sample '{"name": "n", "items": [{"code": "c", "idx": 1, "d": 1.5, "b": true, "dt": "2020-01-01T00:00:00Z"}]}'; +create microflow MyFirstModule.IntParam ($P: integer) returns MyFirstModule.HtChild begin + @start(100, 200) + @position(400, 200) + return empty; +end; +create microflow MyFirstModule.BoolParam ($P: boolean) returns MyFirstModule.HtChild begin + @start(100, 200) + @position(400, 200) + return empty; +end; +create import mapping MyFirstModule.IMM_IntFromString with json structure MyFirstModule.JS_Ht { + create MyFirstModule.HtRoot { + Name = name, + find MyFirstModule.HtChild_HtRoot/MyFirstModule.HtChild by MyFirstModule.IntParam ( P: code ) = items { Code = code } + } +}; +create import mapping MyFirstModule.IMM_BoolFromInteger with json structure MyFirstModule.JS_Ht { + create MyFirstModule.HtRoot { + Name = name, + find MyFirstModule.HtChild_HtRoot/MyFirstModule.HtChild by MyFirstModule.BoolParam ( P: idx ) = items { Code = code } + } +}; diff --git a/mdl-examples/bug-tests/mapping-handler-types.mdl b/mdl-examples/bug-tests/mapping-handler-types.mdl new file mode 100644 index 0000000000..07b068e952 --- /dev/null +++ b/mdl-examples/bug-tests/mapping-handler-types.mdl @@ -0,0 +1,86 @@ +mdl 1; +-- CONTROL for mapping-handler-param-type-mismatch.fail.mdl: every pairing of a handler's +-- JSON member and microflow parameter that MxBuild 11.16.1 builds clean. A String parameter +-- takes any member; a Decimal parameter also takes an Integer; the rest take their own type. +create persistent entity MyFirstModule.HtRoot (Name: string(50)); +create persistent entity MyFirstModule.HtChild (Code: string(50)); +create association MyFirstModule.HtChild_HtRoot from MyFirstModule.HtChild to MyFirstModule.HtRoot type reference; +create json structure MyFirstModule.JS_Ht sample '{"name": "n", "items": [{"code": "c", "idx": 1, "d": 1.5, "b": true, "dt": "2020-01-01T00:00:00Z"}]}'; +create microflow MyFirstModule.StrParam ($P: string) returns MyFirstModule.HtChild begin + @start(100, 200) + @position(400, 200) + return empty; +end; +create microflow MyFirstModule.DecParam ($P: decimal) returns MyFirstModule.HtChild begin + @start(100, 200) + @position(400, 200) + return empty; +end; +create microflow MyFirstModule.IntParam ($P: integer) returns MyFirstModule.HtChild begin + @start(100, 200) + @position(400, 200) + return empty; +end; +create microflow MyFirstModule.LongParam ($P: long) returns MyFirstModule.HtChild begin + @start(100, 200) + @position(400, 200) + return empty; +end; +create microflow MyFirstModule.DtParam ($P: datetime) returns MyFirstModule.HtChild begin + @start(100, 200) + @position(400, 200) + return empty; +end; +create microflow MyFirstModule.BoolParam ($P: boolean) returns MyFirstModule.HtChild begin + @start(100, 200) + @position(400, 200) + return empty; +end; +create import mapping MyFirstModule.IMM_StrFromInt with json structure MyFirstModule.JS_Ht { + create MyFirstModule.HtRoot { + Name = name, + find MyFirstModule.HtChild_HtRoot/MyFirstModule.HtChild by MyFirstModule.StrParam ( P: idx ) = items { Code = code } + } +}; +create import mapping MyFirstModule.IMM_StrFromDt with json structure MyFirstModule.JS_Ht { + create MyFirstModule.HtRoot { + Name = name, + find MyFirstModule.HtChild_HtRoot/MyFirstModule.HtChild by MyFirstModule.StrParam ( P: dt ) = items { Code = code } + } +}; +create import mapping MyFirstModule.IMM_DecFromInt with json structure MyFirstModule.JS_Ht { + create MyFirstModule.HtRoot { + Name = name, + find MyFirstModule.HtChild_HtRoot/MyFirstModule.HtChild by MyFirstModule.DecParam ( P: idx ) = items { Code = code } + } +}; +create import mapping MyFirstModule.IMM_DecFromDec with json structure MyFirstModule.JS_Ht { + create MyFirstModule.HtRoot { + Name = name, + find MyFirstModule.HtChild_HtRoot/MyFirstModule.HtChild by MyFirstModule.DecParam ( P: d ) = items { Code = code } + } +}; +create import mapping MyFirstModule.IMM_IntFromInt with json structure MyFirstModule.JS_Ht { + create MyFirstModule.HtRoot { + Name = name, + find MyFirstModule.HtChild_HtRoot/MyFirstModule.HtChild by MyFirstModule.IntParam ( P: idx ) = items { Code = code } + } +}; +create import mapping MyFirstModule.IMM_LongFromInt with json structure MyFirstModule.JS_Ht { + create MyFirstModule.HtRoot { + Name = name, + find MyFirstModule.HtChild_HtRoot/MyFirstModule.HtChild by MyFirstModule.LongParam ( P: idx ) = items { Code = code } + } +}; +create import mapping MyFirstModule.IMM_DtFromDt with json structure MyFirstModule.JS_Ht { + create MyFirstModule.HtRoot { + Name = name, + find MyFirstModule.HtChild_HtRoot/MyFirstModule.HtChild by MyFirstModule.DtParam ( P: dt ) = items { Code = code } + } +}; +create import mapping MyFirstModule.IMM_BoolFromBool with json structure MyFirstModule.JS_Ht { + create MyFirstModule.HtRoot { + Name = name, + find MyFirstModule.HtChild_HtRoot/MyFirstModule.HtChild by MyFirstModule.BoolParam ( P: b ) = items { Code = code } + } +}; diff --git a/mdl/catalog/builder.go b/mdl/catalog/builder.go index d22dc5f240..f5c5df051e 100644 --- a/mdl/catalog/builder.go +++ b/mdl/catalog/builder.go @@ -832,9 +832,19 @@ func (b *Builder) buildSnapshotRecord() error { } func (b *Builder) updateSnapshotCount() { - // Count total objects + // Counted through the TRANSACTION, not the pool (ako/mxcli#2417). + // + // Two reasons. It runs while the build's transaction is open, so a pool + // query needs a second connection: against an in-memory catalog, where + // every connection is its own database, that read saw an empty one and the + // count came back 0; with the pool pinned to one connection it deadlocked + // outright, waiting for the connection the transaction holds. + // + // And the transaction is the right scope anyway: the rows being counted are + // the ones this build just wrote and has not committed, which a pool query + // cannot see by definition. var count int - row := b.catalog.db.QueryRow(` + rows, err := b.txQuery(` SELECT COUNT(*) FROM ( SELECT Id FROM modules UNION ALL SELECT Id FROM entities @@ -846,7 +856,15 @@ func (b *Builder) updateSnapshotCount() { UNION ALL SELECT Id FROM workflows ) `) - row.Scan(&count) + if err != nil { + return + } + defer rows.Close() + if rows.Next() { + if err := rows.Scan(&count); err != nil { + return + } + } b.snapshot.ObjectCount = count } diff --git a/mdl/catalog/catalogdb_memory_pool_2417_test.go b/mdl/catalog/catalogdb_memory_pool_2417_test.go new file mode 100644 index 0000000000..1c5fa3c3d8 --- /dev/null +++ b/mdl/catalog/catalogdb_memory_pool_2417_test.go @@ -0,0 +1,152 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build !js + +package catalog + +import ( + "sync" + "testing" + "time" +) + +// ako/mxcli#2417: an in-memory catalog's schema must be visible on every +// connection the pool hands out. +// +// sql.Open("sqlite", ":memory:") returns a POOL, and each connection to +// ":memory:" is its own separate, empty database. createTables() therefore +// populated whichever connection it happened to get, and any later query the +// pool served on a different connection saw an empty database and failed with +// "no such table: ". +// +// The symptom was not a clean failure. On a freshly built catalog +// LintContext.AttributesFor failed with "no such table: attributes" while +// ctx.Entities() on the same handle in the same run returned rows, because the +// two queries landed on different connections. Which query failed was down to +// pool timing, so it reproduced on one project and not another, and a lint run +// reported "No issues found." with the error printed after the findings. +// +// Forcing a second concurrent connection is what makes this deterministic: a +// single sequential query reuses the one connection and passes either way. +func TestInMemoryCatalogSchemaIsVisibleOnEveryPooledConnection(t *testing.T) { + c, err := New() + if err != nil { + t.Fatal(err) + } + defer c.Close() + + // Hold one query open so a second must use another connection, and read a + // table on each. Two tables that the linter reads through different code + // paths (entities via Entities, attributes via AttributesFor) are the pair + // that actually diverged. + var wg sync.WaitGroup + errs := make(chan error, 16) + for i := 0; i < 8; i++ { + wg.Add(1) + go func() { + defer wg.Done() + for _, table := range []string{"attributes", "entities", "activities", "widgets"} { + rows, err := c.CatalogDB().Query(`SELECT count(*) FROM ` + table) + if err != nil { + errs <- err + return + } + rows.Close() + } + }() + } + wg.Wait() + close(errs) + for err := range errs { + t.Errorf("query on a pooled connection failed: %v", err) + } +} + +// The pragma set at open time has the same problem: db.Exec runs it on ONE +// connection, so with a pool the others would not have it. Pinning the pool to +// a single connection is what makes the pragma hold for every query too. +func TestInMemoryCatalogForeignKeysPragmaHoldsForEveryQuery(t *testing.T) { + c, err := New() + if err != nil { + t.Fatal(err) + } + defer c.Close() + + var wg sync.WaitGroup + bad := make(chan int, 16) + for i := 0; i < 8; i++ { + wg.Add(1) + go func() { + defer wg.Done() + rows, err := c.CatalogDB().Query(`PRAGMA foreign_keys`) + if err != nil { + bad <- -1 + return + } + defer rows.Close() + if rows.Next() { + var on int + if err := rows.Scan(&on); err == nil && on != 1 { + bad <- on + } + } + }() + } + wg.Wait() + close(bad) + for v := range bad { + t.Errorf("foreign_keys = %d on a pooled connection, want 1", v) + } +} + +// A nested query while a cursor is open must not deadlock. +// +// This is the production pattern, not a contrived one: LintContext.Entities() +// yields rows from an open cursor and its consumers call AttributesFor() inside +// the loop, so a second query runs while the first still holds a connection. +// +// Regression guard for a fix that was tried and rejected. Pinning the pool to a +// single connection (db.SetMaxOpenConns(1)) also gives an in-memory catalog one +// database and looks like the obvious fix, but it makes this nested read wait +// forever for the connection the outer cursor holds — measured as a lint run +// hanging past 600s where it had taken seconds. The shared-cache DSN is what +// gives one database AND the connections these reads need. +// +// The test would hang rather than fail under that regression, so it carries its +// own deadline instead of relying on the package timeout to tell the story. +func TestNestedQueryWhileCursorOpenDoesNotDeadlock(t *testing.T) { + c, err := New() + if err != nil { + t.Fatal(err) + } + defer c.Close() + + done := make(chan error, 1) + go func() { + outer, err := c.CatalogDB().Query(`SELECT Id FROM entities`) + if err != nil { + done <- err + return + } + defer outer.Close() + // Query while `outer` is open, as the linter's iterators do. Reaching + // this at all is the point; an empty catalog yields no rows, so the + // nested read is issued before any Next(). + inner, err := c.CatalogDB().Query(`SELECT count(*) FROM attributes`) + if err != nil { + done <- err + return + } + inner.Close() + done <- nil + }() + + select { + case err := <-done: + if err != nil { + t.Fatalf("nested query failed: %v", err) + } + case <-time.After(20 * time.Second): + t.Fatal("nested query while a cursor is open deadlocked: the pool cannot serve a second connection") + } +} diff --git a/mdl/catalog/catalogdb_sqlite.go b/mdl/catalog/catalogdb_sqlite.go index cb7ec87d6e..b4aef5a95e 100644 --- a/mdl/catalog/catalogdb_sqlite.go +++ b/mdl/catalog/catalogdb_sqlite.go @@ -5,9 +5,12 @@ package catalog import ( + "context" "database/sql" "fmt" + "os" "strings" + "sync/atomic" _ "modernc.org/sqlite" ) @@ -15,19 +18,51 @@ import ( // SqliteCatalogDB wraps *sql.DB from modernc.org/sqlite. type SqliteCatalogDB struct { db *sql.DB + // keepAlive pins one connection open for an in-memory catalog's lifetime. + // A shared-cache in-memory database is destroyed when its LAST connection + // closes, and the pool may close idle ones, so without this a quiet moment + // could discard the catalog. nil for a file-backed catalog. + keepAlive *sql.Conn } +// memCatalogSeq names each in-memory catalog apart from the others. +var memCatalogSeq atomic.Uint64 + // NewSqliteCatalogDB opens an in-memory SQLite database. +// +// The DSN names a SHARED-CACHE database rather than a bare ":memory:" +// (ako/mxcli#2417). Every connection to ":memory:" is its own separate, empty +// database and sql.Open hands back a POOL, so the schema createTables wrote +// landed on one connection while a query the pool served on another saw +// nothing. That surfaced as LintContext.AttributesFor failing with "no such +// table: attributes" on a freshly built catalog while ctx.Entities() succeeded +// in the same run on the same handle. Which query broke came down to pool +// timing, so it reproduced on one project and not another, and lint printed +// "No issues found." with the error after the findings. +// +// Pinning the pool to one connection would also give one database and is NOT +// the fix: the readers query while a cursor is open — ctx.Entities() yields +// rows and the loop body calls ctx.AttributesFor() — so one connection +// deadlocks, the inner read waiting for the connection the outer cursor holds. +// Measured: a lint run that took seconds hung past 600s. A shared cache gives +// one database AND keeps the connections those nested reads need. +// +// The name is unique per catalog so two in one process stay separate, and the +// pragma rides in the DSN because a pragma set through Exec reaches only the +// one pooled connection it ran on. func NewSqliteCatalogDB() (*SqliteCatalogDB, error) { - db, err := sql.Open("sqlite", ":memory:") + dsn := fmt.Sprintf("file:mxcli-catalog-%d-%d?mode=memory&cache=shared&_pragma=foreign_keys(1)", + os.Getpid(), memCatalogSeq.Add(1)) + db, err := sql.Open("sqlite", dsn) if err != nil { return nil, err } - if _, err := db.Exec("PRAGMA foreign_keys = ON"); err != nil { + conn, err := db.Conn(context.Background()) + if err != nil { db.Close() return nil, err } - return &SqliteCatalogDB{db: db}, nil + return &SqliteCatalogDB{db: db, keepAlive: conn}, nil } // fileBusyTimeoutMs is how long a connection to an on-disk catalog waits for a @@ -73,6 +108,12 @@ func (s *SqliteCatalogDB) Begin() (CatalogTx, error) { } func (s *SqliteCatalogDB) Close() error { + // The pinned connection goes first: it is what keeps an in-memory + // catalog's shared cache alive. + if s.keepAlive != nil { + _ = s.keepAlive.Close() + s.keepAlive = nil + } return s.db.Close() } diff --git a/mdl/executor/cmd_sql.go b/mdl/executor/cmd_sql.go index 5f00f1be3b..f3353eb99d 100644 --- a/mdl/executor/cmd_sql.go +++ b/mdl/executor/cmd_sql.go @@ -212,6 +212,12 @@ func execSQLGenerateConnector(ctx *ExecContext, s *ast.SQLGenerateConnectorStmt) return err } + // An object the listing named and the generator could not read is reported, + // not dropped: nothing downstream can notice its absence (ako/mxcli#2422). + for _, skip := range result.SkippedObjects { + fmt.Fprintf(ctx.Output, "-- warning: skipped object the database listed: %s\n", skip) + } + // Report skipped columns for _, skip := range result.SkippedCols { fmt.Fprintf(ctx.Output, "-- warning: skipped unmappable column: %s\n", skip) diff --git a/mdl/executor/validate_mapping_handler_params.go b/mdl/executor/validate_mapping_handler_params.go index 5c2a629ce1..4022a5350d 100644 --- a/mdl/executor/validate_mapping_handler_params.go +++ b/mdl/executor/validate_mapping_handler_params.go @@ -37,7 +37,7 @@ const mappingHandlerParamRule = "MDL104" // A microflow that cannot be found — not the script's, not the project's, or no // project — is not judged: exec already refuses one it cannot find, and an // unanswered question is not "no". A parameter's TYPE against the member it is -// fed is a different error (CE0282) and is not judged here. +// fed is a different error (CE0282), judged by MDL107. // // This is NOT CE0281 ("Unknown value element … selected for parameter"), which the // same construct raises for a MEMBER that is not in the JSON structure: exec diff --git a/mdl/executor/validate_mapping_handler_types.go b/mdl/executor/validate_mapping_handler_types.go new file mode 100644 index 0000000000..04ac3a3008 --- /dev/null +++ b/mdl/executor/validate_mapping_handler_types.go @@ -0,0 +1,261 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/mdl/types" +) + +// mappingHandlerTypeRule is the rule ID for "an import mapping's custom handler +// feeds a JSON member to a microflow parameter of an incompatible type". +const mappingHandlerTypeRule = "MDL107" + +// A custom object handler's `P: member` feeds a JSON member to the microflow's +// parameter P, and the two types must agree. Measured on MxBuild 11.16.1, ONE +// import mapping per project (CE1613 hides every other error), against a sample +// holding a string, an integer, a decimal, a boolean and a date-time; every pairing +// not listed here is CE0282 "The microflow parameter … does not match the specified +// type": +// +// String parameter any member type +// Integer parameter an Integer member +// Long parameter an Integer member +// Decimal parameter an Integer or a Decimal member +// Boolean, DateTime param a member of the same type +// +// check and exec accepted every rejected pairing and wrote the mapping. +// +// The member is found by NAME in the whole structure rather than by following the +// handler's element path through the mapping, which would duplicate the builder's +// array and two-level-item handling (and drift from it, as a CE0281 mirror would). +// It is judged only when every place that name occurs is a value of one and the same +// known type; a name that is absent, or that is a string in one place and an integer +// in another, is left to the build. Only `path` sources are judged — `parent` and +// `parameter` feed objects. A microflow or structure that cannot be found, and a +// parameter of any other type (object, list, enumeration, date), is not judged. +func ValidateMappingHandlerTypes(prog *ast.Program, open func() backend.FullBackend) []linter.Violation { + if prog == nil { + return nil + } + scriptFlows := map[string]map[string]string{} // lower-cased microflow QN -> parameter -> type name + scriptSamples := map[string]string{} + for _, stmt := range prog.Statements { + switch s := stmt.(type) { + case *ast.CreateMicroflowStmt: + params := map[string]string{} + for _, p := range s.Parameters { + params[p.Name] = scriptParamTypeName(p.Type) + } + scriptFlows[strings.ToLower(s.Name.String())] = params + case *ast.CreateJsonStructureStmt: + scriptSamples[s.Name.String()] = s.JsonSnippet + } + } + + var projectFlows map[string]map[string]string + flowParams := func(qn string) (map[string]string, bool) { + if p, ok := scriptFlows[strings.ToLower(qn)]; ok { + return p, true + } + if projectFlows == nil { + projectFlows = loadProjectMicroflowParamTypes(open) + } + p, ok := projectFlows[strings.ToLower(qn)] + return p, ok + } + structures := map[string][]*types.JsonElement{} + structureElems := func(ref ast.QualifiedName) []*types.JsonElement { + key := ref.String() + if elems, done := structures[key]; done { + return elems + } + var elems []*types.JsonElement + if sample, ok := scriptSamples[key]; ok { + elems, _ = types.BuildJsonElementsFromSnippet(sample, nil, nil) + } else if open != nil { + if b := open(); b != nil { + if js, err := b.GetJsonStructureByQualifiedName(ref.Module, ref.Name); err == nil && js != nil { + elems = js.Elements + } + } + } + structures[key] = elems + return elems + } + + var out []linter.Violation + for _, stmt := range prog.Statements { + m, ok := stmt.(*ast.CreateImportMappingStmt) + if !ok || m.RootElement == nil || m.SchemaKind != "JSON_STRUCTURE" || m.SchemaRef.Module == "" { + continue + } + var walk func(e *ast.ImportMappingElementDef) + walk = func(e *ast.ImportMappingElementDef) { + if h := e.CustomHandler; h != nil && h.Microflow != "" { + qn := qualifyIn(m.Name.Module, h.Microflow) + if params, known := flowParams(qn); known { + out = append(out, handlerTypeViolations(m.Name, h, qn, params, structureElems(m.SchemaRef))...) + } + } + for _, c := range e.Children { + if c != nil { + walk(c) + } + } + } + walk(m.RootElement) + } + return out +} + +// scriptParamTypeName names the parameter types the matrix was measured for; any +// other type is "" and is not judged. +func scriptParamTypeName(t ast.DataType) string { + switch t.Kind { + case ast.TypeString: + return "String" + case ast.TypeInteger: + return "Integer" + case ast.TypeLong: + return "Long" + case ast.TypeDecimal: + return "Decimal" + case ast.TypeBoolean: + return "Boolean" + case ast.TypeDateTime: + return "DateTime" + } + return "" +} + +// loadProjectMicroflowParamTypes reads every project microflow's parameter types, +// keyed by lower-cased qualified name. A type outside the measured set reads "". +func loadProjectMicroflowParamTypes(open func() backend.FullBackend) map[string]map[string]string { + out := map[string]map[string]string{} + if open == nil { + return out + } + b := open() + if b == nil { + return out + } + modules, err := b.ListModules() + if err != nil { + return out + } + names := map[string]string{} + for _, mod := range modules { + if mod != nil { + names[string(mod.ID)] = mod.Name + } + } + mfs, err := b.ListMicroflows() + if err != nil { + return out + } + for _, mf := range mfs { + if mf == nil || names[string(mf.ContainerID)] == "" { + continue + } + params := map[string]string{} + for _, p := range mf.Parameters { + if p == nil || p.Type == nil { + continue + } + switch n := p.Type.GetTypeName(); n { + case "String", "Integer", "Long", "Decimal", "Boolean", "DateTime": + params[p.Name] = n + default: + params[p.Name] = "" + } + } + out[strings.ToLower(names[string(mf.ContainerID)]+"."+mf.Name)] = params + } + return out +} + +// handlerTypeCompatible is the measured matrix: may a member of type member be fed +// to a parameter of type param. +func handlerTypeCompatible(param, member string) bool { + switch param { + case "String": + return true + case "Integer", "Long": + return member == "Integer" + case "Decimal": + return member == "Integer" || member == "Decimal" + } + return param == member +} + +// memberPrimitiveType is the one type every occurrence of the member has, or "" when +// the name is absent, is not a value, is of a type outside the measured set, or +// differs between occurrences. +func memberPrimitiveType(elems []*types.JsonElement, member string) string { + suffix := "|" + strings.ReplaceAll(member, "/", "|") + found := "" + ambiguous := false + var walk func(es []*types.JsonElement) + walk = func(es []*types.JsonElement) { + for _, e := range es { + if e == nil { + continue + } + if strings.HasSuffix(e.Path, suffix) || (!strings.Contains(member, "/") && e.ExposedName == member) { + switch { + case e.ElementType != "Value": + ambiguous = true + case found == "": + found = e.PrimitiveType + case found != e.PrimitiveType: + ambiguous = true + } + } + walk(e.Children) + } + } + walk(elems) + switch found { + case "String", "Integer", "Decimal", "Boolean", "DateTime": + if !ambiguous { + return found + } + } + return "" +} + +func handlerTypeViolations(mapping ast.QualifiedName, h *ast.MappingCustomHandlerDef, qn string, params map[string]string, elems []*types.JsonElement) []linter.Violation { + loc := linter.Location{Module: mapping.Module, DocumentType: "import mapping", DocumentName: mapping.Name} + var out []linter.Violation + for _, p := range h.Parameters { + if p == nil || p.Source != "path" || p.Path == "" { + continue + } + paramType := params[p.Parameter] + if paramType == "" { + continue + } + memberType := memberPrimitiveType(elems, p.Path) + if memberType == "" || handlerTypeCompatible(paramType, memberType) { + continue + } + out = append(out, linter.Violation{ + RuleID: mappingHandlerTypeRule, + Severity: linter.SeverityError, + Message: fmt.Sprintf( + "import mapping %s: the handler `by %s` feeds %s, a %s member, to parameter %s of type %s — "+ + "mxbuild reports CE0282 \"The microflow parameter '%s' is of type '%s'. It does not match the specified type '%s'\"", + mapping.String(), h.Microflow, p.Path, memberType, p.Parameter, paramType, p.Parameter, paramType, memberType), + Location: loc, + Suggestion: fmt.Sprintf("Declare %s.%s as %s, or key on a %s member. Only a String parameter takes every member type; "+ + "Decimal also takes an Integer.", qn, p.Parameter, memberType, paramType), + }) + } + return out +} diff --git a/mdl/executor/validate_mapping_handler_types_test.go b/mdl/executor/validate_mapping_handler_types_test.go new file mode 100644 index 0000000000..ea4a28f427 --- /dev/null +++ b/mdl/executor/validate_mapping_handler_types_test.go @@ -0,0 +1,155 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// A custom object handler feeds a JSON member to a microflow parameter, and the +// two types must agree. Measured on MxBuild 11.16.1, ONE import mapping per project +// (CE1613 hides every other error), against a sample holding a string, an integer, +// a decimal, a boolean and a date-time. Every pairing not listed as clean is CE0282: +// +// String parameter clean for every member type +// Integer parameter clean for an Integer member only +// Long parameter clean for an Integer member (Long, Decimal, String members: CE0282) +// Decimal parameter clean for an Integer or a Decimal member +// Boolean, DateTime param clean for a member of the same type only +const handlerTypesSample = `{"name": "n", "items": [{"code": "c", "idx": 1, "d": 1.5, "b": true, "dt": "2020-01-01T00:00:00Z"}]}` + +func handlerTypesScript(paramType, member string) string { + return `mdl 1; +create persistent entity Ht.Root (Name: string(50)); +create persistent entity Ht.Child (Code: string(50)); +create association Ht.Child_Root from Ht.Child to Ht.Root type reference; +create json structure Ht.JS sample '` + handlerTypesSample + `'; +create microflow Ht.MF ($P: ` + paramType + `) returns Ht.Child begin + return empty; +end; +create import mapping Ht.IMM with json structure Ht.JS { + create Ht.Root { + Name = name, + find Ht.Child_Root/Ht.Child by Ht.MF ( P: ` + member + ` ) = items { Code = code } + } +}; +` +} + +func TestValidateMappingHandlerTypes_Matrix(t *testing.T) { + members := map[string]string{"String": "code", "Integer": "idx", "Decimal": "d", "Boolean": "b", "DateTime": "dt"} + clean := map[string][]string{ + "string": {"String", "Integer", "Decimal", "Boolean", "DateTime"}, + "integer": {"Integer"}, + "long": {"Integer"}, + "decimal": {"Integer", "Decimal"}, + "boolean": {"Boolean"}, + "datetime": {"DateTime"}, + } + for param, ok := range clean { + for mtype, member := range members { + want := 1 + for _, c := range ok { + if c == mtype { + want = 0 + } + } + got := ValidateMappingHandlerTypes(mustBuild(t, handlerTypesScript(param, member)), nil) + if len(got) != want { + t.Errorf("%s parameter fed a %s member: violations=%v, want %d", param, mtype, rulesOf(got), want) + continue + } + if want == 1 { + v := got[0] + if v.RuleID != mappingHandlerTypeRule || !strings.Contains(v.Message, "CE0282") || + !strings.Contains(v.Message, "Ht.MF") || v.Location.DocumentName != "IMM" { + t.Errorf("%s/%s: rule, CE code, microflow or location wrong: %+v", param, mtype, v) + } + } + } + } +} + +// What is not judged: a member that is not in the structure, a parameter that is +// an object, a source other than a JSON member, and a microflow that is not known. +func TestValidateMappingHandlerTypes_NotJudged(t *testing.T) { + cases := map[string]string{ + "a member the structure does not have": handlerTypesScript("integer", "missing"), + "a parent source": handlerTypesScript("integer", "parent"), + "an object parameter": handlerTypesScript("Ht.Child", "idx"), + } + for name, src := range cases { + if got := ValidateMappingHandlerTypes(mustBuild(t, src), nil); len(got) != 0 { + t.Errorf("%s: want no violation, got %v", name, rulesOf(got)) + } + } + unknown := strings.Replace(handlerTypesScript("integer", "code"), "by Ht.MF", "by Other.Elsewhere", 1) + if got := ValidateMappingHandlerTypes(mustBuild(t, unknown), nil); len(got) != 0 { + t.Errorf("a microflow that is not known here: want none, got %v", rulesOf(got)) + } +} + +// The same key at two places with two types cannot be judged without following +// the mapping's path, so it is left alone rather than guessed. +func TestValidateMappingHandlerTypes_AmbiguousMember(t *testing.T) { + src := strings.Replace(handlerTypesScript("integer", "code"), + `"items": [`, `"other": {"code": 5}, "items": [`, 1) + if got := ValidateMappingHandlerTypes(mustBuild(t, src), nil); len(got) != 0 { + t.Errorf("`code` is a string under items and an integer under other: want none, got %v", rulesOf(got)) + } + // A nested member reached through its parent is judged on the whole path. + nested := strings.Replace(handlerTypesScript("integer", "meta/k"), + `"items": [`, `"items": [{"meta": {"k": "s"}}, `, 1) + if got := ValidateMappingHandlerTypes(mustBuild(t, nested), nil); len(got) != 1 { + t.Errorf("meta/k is a string: want one violation, got %v", rulesOf(got)) + } +} + +// The structure and the microflow both live in the project. +func TestValidateMappingHandlerTypes_Project(t *testing.T) { + mod := &model.Module{BaseElement: model.BaseElement{ID: "mod-pj"}, Name: "Pj"} + mf := µflows.Microflow{ + BaseElement: model.BaseElement{ID: "mf1"}, ContainerID: mod.ID, Name: "Resolve", + Parameters: []*microflows.MicroflowParameter{{Name: "Index", Type: µflows.IntegerType{}}}, + } + js := &types.JsonStructure{Name: "JS", JsonSnippet: `{"items": [{"code": "c", "idx": 1}]}`} + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListMicroflowsFunc: func() ([]*microflows.Microflow, error) { return []*microflows.Microflow{mf}, nil }, + GetJsonStructureByQualifiedNameFunc: func(m, n string) (*types.JsonStructure, error) { + if m == "Pj" && n == "JS" { + elems, err := types.BuildJsonElementsFromSnippet(js.JsonSnippet, nil, nil) + if err != nil { + return nil, err + } + js.Elements = elems + return js, nil + } + return nil, nil + }, + } + open := func() backend.FullBackend { return mb } + src := func(member string) string { + return `mdl 1; +create import mapping Pj.IMM with json structure Pj.JS { + create Pj.Root { + find Pj.Child_Root/Pj.Child by Pj.Resolve ( Index: ` + member + ` ) = items { Code = code } + } +}; +` + } + if got := ValidateMappingHandlerTypes(mustBuild(t, src("idx")), open); len(got) != 0 { + t.Errorf("an Integer member into an Integer parameter (control): want none, got %v", rulesOf(got)) + } + if got := ValidateMappingHandlerTypes(mustBuild(t, src("code")), open); len(got) != 1 { + t.Errorf("a String member into an Integer parameter: want one, got %v", rulesOf(got)) + } +} diff --git a/mdl/executor/validate_program.go b/mdl/executor/validate_program.go index cfa9cded46..6df059a4e4 100644 --- a/mdl/executor/validate_program.go +++ b/mdl/executor/validate_program.go @@ -425,6 +425,8 @@ func ValidateProgram(prog *ast.Program, projectPath string) []linter.Violation { violations = append(violations, ValidateRestMappingResult(prog, projectPath)...) // A custom object handler naming a parameter its microflow does not declare (CE1613). violations = append(violations, ValidateMappingHandlerParams(prog, project)...) + // A custom handler feeding a JSON member to a microflow parameter of another type (CE0282). + violations = append(violations, ValidateMappingHandlerTypes(prog, project)...) // Flag a REST client operation whose Body/Response mapping clause has no // `{ ... }` body — Mendix cannot reference a mapping document from an diff --git a/mdl/linter/rules/mpr017_fixed_guard_attribute.go b/mdl/linter/rules/mpr017_fixed_guard_attribute.go index 4dca70c7dc..5143056f83 100644 --- a/mdl/linter/rules/mpr017_fixed_guard_attribute.go +++ b/mdl/linter/rules/mpr017_fixed_guard_attribute.go @@ -271,21 +271,20 @@ func collectAttributeFacts(ctx *linter.LintContext) (map[string]*attrFacts, stri byName[a.Name] = append(byName[a.Name], qn) } } - // No attributes readable. If the project genuinely has no entities that is - // simply nothing to check; but entities WITH no readable attributes means - // the attribute read failed, and this rule must then say so rather than - // report a clean run. + // No attributes readable. A project with no entities is simply nothing to + // check; entities WITH no readable attributes means the attribute read + // failed, and this rule then says so rather than reporting a clean run. // - // It is not hypothetical: on a freshly built catalog ctx.AttributesFor - // fails with "no such table: attributes" (ako/mxcli#2417) while entities - // and activities resolve, and the context stops answering afterwards. Left - // silent, this rule reported NOTHING on a project that provably has the - // defect, which is indistinguishable from a pass. + // The specific bug this was written for is fixed (ako/mxcli#2417: an + // in-memory catalog gave each pooled connection its own empty database, so + // AttributesFor could fail while Entities succeeded in the same run), and + // the advice it used to carry -- re-run to use the cached catalog -- is + // gone with it. The check itself stays: it is the invariant, not the + // workaround. A rule that cannot read its input must not be able to report + // a pass, whatever the next reason for an unreadable catalog turns out to be. if len(byName) == 0 { if entities > 0 { - return nil, fmt.Sprintf("read %d entities but no attributes "+ - "(ako/mxcli#2417: AttributesFor fails on a freshly built catalog); "+ - "re-run to use the cached catalog", entities) + return nil, fmt.Sprintf("read %d entities but could not read their attributes", entities) } return facts, "" } diff --git a/sql/generate.go b/sql/generate.go index 4f4809f3f1..117e45fcb7 100644 --- a/sql/generate.go +++ b/sql/generate.go @@ -26,6 +26,11 @@ type GenerateResult struct { TableCount int ViewCount int SkippedCols []string // "table.column: type" for unmappable columns + // SkippedObjects is every listed object the generator could not read, as + // "address: reason". An object the listing names and the output omits is + // invisible downstream, so it is reported rather than dropped + // (ako/mxcli#2422). + SkippedObjects []string } // GenerateConnector discovers schema and generates complete MDL for a Mendix Database Connector. @@ -33,9 +38,17 @@ func GenerateConnector(ctx context.Context, cfg *GenerateConfig) (*GenerateResul var schemas []*TableSchema result := &GenerateResult{} - // Discover tables - if cfg.Tables != nil { - // Specific tables + // Either clause scopes the whole command. Treating an absent TABLES as + // "every table" while VIEWS named a few generated the lot and then the + // named views on top, so a view came out twice under one entity name + // (ako/mxcli#2415). + if cfg.Tables == nil && cfg.Views == nil { + var err error + schemas, result.SkippedObjects, err = readListed(ctx, cfg.Conn, false) + if err != nil { + return nil, fmt.Errorf("reading tables: %w", err) + } + } else { for _, t := range cfg.Tables { ts, err := ReadTableSchema(ctx, cfg.Conn, t, false) if err != nil { @@ -43,18 +56,6 @@ func GenerateConnector(ctx context.Context, cfg *GenerateConfig) (*GenerateResul } schemas = append(schemas, ts) } - } else { - // All tables - var err error - schemas, err = ReadAllTableSchemas(ctx, cfg.Conn) - if err != nil { - return nil, fmt.Errorf("reading tables: %w", err) - } - } - result.TableCount = len(schemas) - - // Discover views - if cfg.Views != nil { for _, v := range cfg.Views { ts, err := ReadTableSchema(ctx, cfg.Conn, v, true) if err != nil { @@ -63,7 +64,18 @@ func GenerateConnector(ctx context.Context, cfg *GenerateConfig) (*GenerateResul ts.IsView = true schemas = append(schemas, ts) } - result.ViewCount = len(cfg.Views) + } + + // One object, one entity. Naming it in both clauses, or under two + // spellings of its address, is a request for it once -- two entities of one + // name is not a model Mendix can load. + schemas = dedupeByAddress(schemas) + for _, ts := range schemas { + if ts.IsView { + result.ViewCount++ + } else { + result.TableCount++ + } } if len(schemas) == 0 { @@ -218,6 +230,23 @@ type columnMapping struct { } // driverToMendixType maps our driver to the Mendix Database Connector TYPE string. +// dedupeByAddress keeps the first reading of each object, comparing addresses +// without case: the clause spells a name as the user typed it, the listing as +// the database stores it. +func dedupeByAddress(schemas []*TableSchema) []*TableSchema { + seen := make(map[string]bool, len(schemas)) + out := schemas[:0:0] + for _, ts := range schemas { + key := strings.ToLower(ts.QualifiedName()) + if seen[key] { + continue + } + seen[key] = true + out = append(out, ts) + } + return out +} + func driverToMendixType(driver DriverName) string { switch driver { case DriverPostgres: diff --git a/sql/generate_clause_scope_2415_test.go b/sql/generate_clause_scope_2415_test.go new file mode 100644 index 0000000000..76f5d0c377 --- /dev/null +++ b/sql/generate_clause_scope_2415_test.go @@ -0,0 +1,201 @@ +// SPDX-License-Identifier: Apache-2.0 + +package sql + +import ( + "context" + "strings" + "testing" +) + +// ako/mxcli#2415: `VIEWS (…)` without `TABLES (…)` generated every table as +// well, then appended the named views on top. On a database whose objects are +// views — a DuckGraphLake lake serving Mendix entity CDC — each named view came +// out twice, so one connector held two entities and two queries under the same +// name: +// +// 2 QUERY GetAllCustomers +// 2 ENTITY LakeSql.Customer +// +// Two causes compounded: `cfg.Tables == nil` meant "all tables" with nothing +// suppressing it when `Views` was set, and the table reader did not look at +// information_schema's table_type, so a view was already in the "tables" set +// before the view pass added it again. +// +// Same harness as the #2414 tests: modernc SQLite with an attached +// information_schema, DriverPostgres so the real query templates run. + +// twoSchemaLake is three CDC views in two schemas plus one real table, so a +// test can tell a table pass from a view pass and a schema from a name. +func twoSchemaLake(t *testing.T) *Connection { + conn := openSQLite(t, "clauses", + `ATTACH DATABASE ':memory:' AS information_schema`, + `CREATE TABLE information_schema.tables (table_schema TEXT, table_name TEXT, table_type TEXT)`, + `CREATE TABLE information_schema.columns (table_schema TEXT, table_name TEXT, + column_name TEXT, data_type TEXT, is_nullable TEXT, column_default TEXT, ordinal_position INTEGER)`, + `INSERT INTO information_schema.tables VALUES + ('cdcpublish', 'customer', 'VIEW'), + ('cdcpublish', 'order', 'VIEW'), + ('otherapp', 'invoice', 'VIEW'), + ('public', 'audit_log', 'BASE TABLE')`, + `INSERT INTO information_schema.columns VALUES + ('cdcpublish', 'customer', 'id', 'bigint', 'NO', NULL, 1), + ('cdcpublish', 'customer', 'Name', 'character varying', 'YES', NULL, 2), + ('cdcpublish', 'order', 'id', 'bigint', 'NO', NULL, 1), + ('otherapp', 'invoice', 'id', 'bigint', 'NO', NULL, 1), + ('public', 'audit_log', 'id', 'bigint', 'NO', NULL, 1)`) + conn.Driver = DriverPostgres + return conn +} + +func gen(t *testing.T, conn *Connection, tables, views []string) *GenerateResult { + t.Helper() + res, err := GenerateConnector(context.Background(), &GenerateConfig{ + Conn: conn, Module: "LakeSql", Alias: "lake", Tables: tables, Views: views, + }) + if err != nil { + t.Fatal(err) + } + return res +} + +func queryNames(mdl string) []string { + var out []string + for _, line := range strings.Split(mdl, "\n") { + if f := strings.Fields(strings.TrimSpace(line)); len(f) == 2 && f[0] == "QUERY" { + out = append(out, f[1]) + } + } + return out +} + +func duplicates(names []string) []string { + seen, dup := map[string]bool{}, []string(nil) + for _, n := range names { + if seen[n] { + dup = append(dup, n) + } + seen[n] = true + } + return dup +} + +func TestGenerateConnector_ViewsClauseDoesNotPullInEveryTable_Issue2415(t *testing.T) { + res := gen(t, twoSchemaLake(t), nil, []string{"cdcpublish.customer"}) + + if got := queryNames(res.MDL); len(got) != 1 || got[0] != "GetAllCustomers" { + t.Errorf("VIEWS (cdcpublish.customer) generated %v, want just [GetAllCustomers]", got) + } + for _, unwanted := range []string{"audit_log", "AuditLog", "invoice", "Invoice", "GetAllOrders"} { + if strings.Contains(res.MDL, unwanted) { + t.Errorf("an object nobody asked for reached the output: %q", unwanted) + } + } + if res.TableCount != 0 || res.ViewCount != 1 { + t.Errorf("counts = %d tables, %d views; want 0 and 1", res.TableCount, res.ViewCount) + } +} + +// Naming one object in both clauses must not emit it twice under one name — +// two entities called LakeSql.Customer in one connector is not a model Mendix +// can load. +func TestGenerateConnector_ObjectInBothClausesEmittedOnce_Issue2415(t *testing.T) { + for _, tc := range []struct{ name, table, view string }{ + {"same spelling", "cdcpublish.customer", "cdcpublish.customer"}, + // Resolving a bare name to its schema is what makes the two spellings + // one object; without it the dedupe key differs and the duplicate is + // back. + {"bare and qualified", "customer", "cdcpublish.customer"}, + } { + t.Run(tc.name, func(t *testing.T) { + res := gen(t, twoSchemaLake(t), []string{tc.table}, []string{tc.view}) + if dup := duplicates(queryNames(res.MDL)); len(dup) > 0 { + t.Errorf("duplicate query name(s) %v in\n%s", dup, res.MDL) + } + if n := strings.Count(res.MDL, "CREATE NON-PERSISTENT ENTITY"); n != 1 { + t.Errorf("%d entities, want 1 — the same object was emitted twice:\n%s", n, res.MDL) + } + }) + } +} + +// A bare name in a clause has to be addressed by its schema too, or the query +// fails exactly as ako/mxcli#2414 described. The #2414 fix reached the +// discovery path only. +func TestGenerateConnector_BareClauseNameIsQualified_Issue2415(t *testing.T) { + res := gen(t, twoSchemaLake(t), []string{"customer"}, nil) + if !strings.Contains(res.MDL, "FROM cdcpublish.customer") { + t.Errorf("TABLES (customer) must still address the schema it resolved to:\n%s", res.MDL) + } +} + +// CONTROL: with neither clause, everything is still generated — tables and +// views alike. Filtering views out of the table reader without enumerating +// them would silently empty the output on a view-only database, which is what +// a CDC lake is. +func TestGenerateConnector_NoClauseGeneratesEverythingOnce_Issue2415(t *testing.T) { + res := gen(t, twoSchemaLake(t), nil, nil) + + names := queryNames(res.MDL) + if dup := duplicates(names); len(dup) > 0 { + t.Errorf("duplicate query name(s) %v with no clause in\n%s", dup, res.MDL) + } + if len(names) != 4 { + t.Errorf("%d queries (%v), want 4 — three views and one table", len(names), names) + } + for _, want := range []string{ + "FROM cdcpublish.customer", "FROM cdcpublish.order", + "FROM otherapp.invoice", "FROM public.audit_log", + } { + if !strings.Contains(res.MDL, want) { + t.Errorf("want %q in\n%s", want, res.MDL) + } + } + // The counts have to say which is which, since that is the only report of + // what the command decided. + if res.TableCount != 1 || res.ViewCount != 3 { + t.Errorf("counts = %d tables, %d views; want 1 and 3", res.TableCount, res.ViewCount) + } +} + +// ReadAllTableSchemas is exported and says tables. It must not hand back views +// — that conflation is half of this defect. +func TestReadAllTableSchemas_ExcludesViews_Issue2415(t *testing.T) { + got, err := ReadAllTableSchemas(context.Background(), twoSchemaLake(t)) + if err != nil { + t.Fatal(err) + } + if len(got) != 1 || got[0].Name != "audit_log" || got[0].IsView { + var names []string + for _, ts := range got { + names = append(names, ts.QualifiedName()) + } + t.Errorf("ReadAllTableSchemas returned %v, want just the one BASE TABLE public.audit_log", names) + } +} + +// ReadAllSchemas is the one the no-clause path uses: every object, each marked +// with what it is. +func TestReadAllSchemas_MarksViewsAsViews_Issue2415(t *testing.T) { + got, err := ReadAllSchemas(context.Background(), twoSchemaLake(t)) + if err != nil { + t.Fatal(err) + } + if len(got) != 4 { + t.Fatalf("%d objects, want 4", len(got)) + } + want := map[string]bool{ // address -> IsView + "cdcpublish.customer": true, "cdcpublish.order": true, + "otherapp.invoice": true, "public.audit_log": false, + } + for _, ts := range got { + isView, known := want[ts.QualifiedName()] + if !known { + t.Errorf("unexpected object %q", ts.QualifiedName()) + continue + } + if ts.IsView != isView { + t.Errorf("%s: IsView = %v, want %v", ts.QualifiedName(), ts.IsView, isView) + } + } +} diff --git a/sql/generate_dotted_names_2422_test.go b/sql/generate_dotted_names_2422_test.go new file mode 100644 index 0000000000..2b6fafe034 --- /dev/null +++ b/sql/generate_dotted_names_2422_test.go @@ -0,0 +1,166 @@ +// SPDX-License-Identifier: Apache-2.0 + +package sql + +import ( + "context" + "strings" + "testing" +) + +// ako/mxcli#2422: an object whose own NAME contains a dot vanished from a +// generated connector with no warning. A DuckGraphLake lake names its raw +// topic tables after the Kafka topic, so four of its nine objects were missing: +// +// main | local__cdc.CdcPublish.Customer.1.local | VIEW +// +// Three things had to hold for that object to come out right, and none did. +// The address was flattened to a string and split again on every dot, so +// splitTableName read schema `1`, name `local`; DescribeTable then found +// nothing and the reader's `continue // skip objects that fail` discarded the +// error. Even past that, the emitted SQL and the entity name would both have +// been invalid. +// +// Measured on the lake: `select count(*) from main."local__cdc.CdcPublish.Customer.1.local"` +// returns 5 rows, the unquoted form is `syntax error at or near ".1."`, and +// information_schema.columns has the object's 7 columns under that exact +// table_name — so nothing is wrong with the database's side of it. + +func TestSplitTableName_QuotedNameKeepsItsDots_Issue2422(t *testing.T) { + for in, want := range map[string][2]string{ + // The shape this issue is about. + `main."local__cdc.CdcPublish.Customer.1.local"`: {"main", "local__cdc.CdcPublish.Customer.1.local"}, + `"main"."a.b"`: {"main", "a.b"}, + `"a.b"`: {"", "a.b"}, + // CONTROLS: every shape that worked before still has to. + "customers": {"", "customers"}, + "sales.customers": {"sales", "customers"}, + `"Sales"."Cust"`: {"Sales", "Cust"}, + "db.sales.customers": {"sales", "customers"}, + "[dbo].[Orders]": {"dbo", "Orders"}, + " sales . customers ": {"sales", "customers"}, + } { + schema, name := splitTableName(in) + if schema != want[0] || name != want[1] { + t.Errorf("splitTableName(%q) = (%q, %q), want (%q, %q)", in, schema, name, want[0], want[1]) + } + } +} + +// The address has to be re-quoted on the way out, or the generated query is a +// syntax error rather than a wrong table. +func TestTableSchemaQualifiedName_QuotesANameThatNeedsIt_Issue2422(t *testing.T) { + for _, tc := range []struct{ schema, name, want string }{ + {"main", "local__cdc.CdcPublish.Customer.1.local", `main."local__cdc.CdcPublish.Customer.1.local"`}, + {"", "a.b", `"a.b"`}, + {"main", `odd"name`, `main."odd""name"`}, // an embedded quote is doubled + {"main", "needs space", `main."needs space"`}, + {"main", "1startsdigit", `main."1startsdigit"`}, + // CONTROLS: a plain identifier is left alone, as #2414 established. + {"cdcpublish", "customer", "cdcpublish.customer"}, + {"", "customer", "customer"}, + {"cdcpublish", "order_history", "cdcpublish.order_history"}, + } { + ts := &TableSchema{Schema: tc.schema, Name: tc.name} + if got := ts.QualifiedName(); got != tc.want { + t.Errorf("QualifiedName(%q, %q) = %q, want %q", tc.schema, tc.name, got, tc.want) + } + } +} + +// A dot is not a word character in a Mendix name. Left in, it produced +// `CREATE NON-PERSISTENT ENTITY LakeSql.LocalCdc.cdcpublish.customer.1.local`, +// which is not an entity name at all — so fixing only the addressing would +// have traded a silent drop for silently invalid MDL. +func TestTableToEntityName_DotsAreWordSeparators_Issue2422(t *testing.T) { + for in, want := range map[string]string{ + "local__cdc.CdcPublish.Customer.1.local": "LocalCdcCdcpublishCustomer1Local", + "my.table": "MyTable", + // CONTROLS: the names everything else generates are unchanged. + "customer": "Customer", + "order_history": "OrderHistory", + "audit_log": "AuditLog", + } { + if got := TableToEntityName(in); got != want { + t.Errorf("TableToEntityName(%q) = %q, want %q", in, got, want) + } + if strings.ContainsAny(TableToEntityName(in), ". -") { + t.Errorf("TableToEntityName(%q) = %q, which is not a usable Mendix name", in, TableToEntityName(in)) + } + } +} + +// dottedLake mirrors the lake: the per-app CDC views plus the raw topic table +// whose name is the topic, dots and all. +func dottedLake(t *testing.T) *Connection { + conn := openSQLite(t, "dotted", + `ATTACH DATABASE ':memory:' AS information_schema`, + `CREATE TABLE information_schema.tables (table_schema TEXT, table_name TEXT, table_type TEXT)`, + `CREATE TABLE information_schema.columns (table_schema TEXT, table_name TEXT, + column_name TEXT, data_type TEXT, is_nullable TEXT, column_default TEXT, ordinal_position INTEGER)`, + `INSERT INTO information_schema.tables VALUES + ('cdcpublish', 'customer', 'VIEW'), + ('main', 'local__cdc.CdcPublish.Customer.1.local', 'VIEW')`, + `INSERT INTO information_schema.columns VALUES + ('cdcpublish', 'customer', 'id', 'bigint', 'NO', NULL, 1), + ('main', 'local__cdc.CdcPublish.Customer.1.local', 'offset', 'bigint', 'NO', NULL, 1), + ('main', 'local__cdc.CdcPublish.Customer.1.local', 'key', 'text', 'YES', NULL, 2)`) + conn.Driver = DriverPostgres + return conn +} + +func TestGenerateConnector_DottedObjectIsGenerated_Issue2422(t *testing.T) { + res, err := GenerateConnector(context.Background(), &GenerateConfig{ + Conn: dottedLake(t), Module: "LakeSql", Alias: "lake", + }) + if err != nil { + t.Fatal(err) + } + + if n := strings.Count(res.MDL, "CREATE NON-PERSISTENT ENTITY"); n != 2 { + t.Errorf("%d entities, want 2 — the dotted object was dropped:\n%s", n, res.MDL) + } + if want := `FROM main."local__cdc.CdcPublish.Customer.1.local"`; !strings.Contains(res.MDL, want) { + t.Errorf("want %q in\n%s", want, res.MDL) + } + if !strings.Contains(res.MDL, "ENTITY LakeSql.LocalCdcCdcpublishCustomer1Local (") { + t.Errorf("the dotted name did not become a usable entity name:\n%s", res.MDL) + } + // Its own columns, not another object's. + if !strings.Contains(res.MDL, "Offset") || !strings.Contains(res.MDL, "Key") { + t.Errorf("the dotted object's columns are missing:\n%s", res.MDL) + } + if res.ViewCount != 2 { + t.Errorf("ViewCount = %d, want 2", res.ViewCount) + } +} + +// The other half of the defect, and the half worth having whatever is decided +// about dotted names: an object the listing names and the generator cannot read +// must be reported, not dropped. Nothing downstream can notice its absence. +func TestGenerateConnector_UnreadableObjectIsReported_Issue2422(t *testing.T) { + conn := openSQLite(t, "unreadable", + `ATTACH DATABASE ':memory:' AS information_schema`, + `CREATE TABLE information_schema.tables (table_schema TEXT, table_name TEXT, table_type TEXT)`, + `CREATE TABLE information_schema.columns (table_schema TEXT, table_name TEXT, + column_name TEXT, data_type TEXT, is_nullable TEXT, column_default TEXT, ordinal_position INTEGER)`, + `INSERT INTO information_schema.tables VALUES + ('cdcpublish', 'customer', 'VIEW'), + ('cdcpublish', 'vanished', 'VIEW')`, + `INSERT INTO information_schema.columns VALUES + ('cdcpublish', 'customer', 'id', 'bigint', 'NO', NULL, 1)`) + conn.Driver = DriverPostgres + + res, err := GenerateConnector(context.Background(), &GenerateConfig{ + Conn: conn, Module: "LakeSql", Alias: "lake", + }) + if err != nil { + t.Fatal(err) + } + if n := strings.Count(res.MDL, "CREATE NON-PERSISTENT ENTITY"); n != 1 { + t.Errorf("%d entities, want 1", n) + } + if len(res.SkippedObjects) != 1 || !strings.Contains(res.SkippedObjects[0], "vanished") { + t.Errorf("SkippedObjects = %v, want one naming cdcpublish.vanished", res.SkippedObjects) + } +} diff --git a/sql/meta.go b/sql/meta.go index e2f1e52951..bbde5c5046 100644 --- a/sql/meta.go +++ b/sql/meta.go @@ -135,6 +135,22 @@ func refuseAmbiguousTable(ctx context.Context, conn *Connection, table string) e if schema != "" { return nil } + holders := schemasHolding(ctx, conn, name) + if len(holders) < 2 { + return nil + } + var names []string + for _, h := range holders { + names = append(names, h+"."+name) + } + return fmt.Errorf("%s is in more than one schema: name one of %s", name, strings.Join(names, ", ")) +} + +// schemasHolding is every schema that has a table or view of this (bare) name, +// in order. Empty when the lookup is not possible or finds nothing, so a caller +// must treat "no answer" and "not there" alike: DescribeTable is the one that +// reports a missing object. +func schemasHolding(ctx context.Context, conn *Connection, name string) []string { var q string switch conn.Driver { case DriverOracle: @@ -145,14 +161,16 @@ func refuseAmbiguousTable(ctx context.Context, conn *Connection, table string) e q = "SELECT DISTINCT table_schema FROM information_schema.columns WHERE table_name = " + sqlString(name) + " ORDER BY 1" } res, err := Execute(ctx, conn, q) - if err != nil || len(res.Rows) < 2 { + if err != nil { return nil } - var names []string + var out []string for _, r := range res.Rows { - names = append(names, fmt.Sprintf("%v.%s", r[0], name)) + if len(r) > 0 { + out = append(out, fmt.Sprintf("%v", r[0])) + } } - return fmt.Errorf("%s is in more than one schema: name one of %s", name, strings.Join(names, ", ")) + return out } // columnsFilter is the information_schema filter for table, qualified by its @@ -169,16 +187,85 @@ func columnsFilter(table, schemaCol, nameCol string) string { // splitTableName splits `schema.table` (a leading database part is dropped) // and removes identifier quotes. func splitTableName(table string) (schema, name string) { - parts := strings.Split(table, ".") - for i, p := range parts { - parts[i] = strings.Trim(strings.TrimSpace(p), `"[]`+"`") - } + parts := splitQualifiedIdent(table) if len(parts) == 1 { return "", parts[0] } return parts[len(parts)-2], parts[len(parts)-1] } +// splitQualifiedIdent splits a dotted identifier on the dots that are not +// inside quotes, and unquotes each part. A quoted part may contain dots of its +// own -- a table named after a Kafka topic does, `main."local__cdc.App.E.1.ns"` +// -- and splitting those made the object unaddressable, so it was dropped from +// a generated connector without a word (ako/mxcli#2422). +func splitQualifiedIdent(s string) []string { + var ( + parts []string + cur strings.Builder + closer rune // 0 when not inside quotes + ) + runes := []rune(s) + for i := 0; i < len(runes); i++ { + r := runes[i] + if closer != 0 { + if r != closer { + cur.WriteRune(r) + continue + } + // "" inside a quoted name is one literal quote, not the end of it. + if r == '"' && i+1 < len(runes) && runes[i+1] == '"' { + cur.WriteRune('"') + i++ + continue + } + closer = 0 + continue + } + switch r { + case '"': + closer = '"' + case '`': + closer = '`' + case '[': + closer = ']' + case '.': + parts = append(parts, strings.TrimSpace(cur.String())) + cur.Reset() + default: + cur.WriteRune(r) + } + } + return append(parts, strings.TrimSpace(cur.String())) +} + +// quoteIdent quotes an identifier that is not a bare word, doubling any quote +// inside it. A bare word is left alone, reserved or not: `cdcpublish.order` +// parses, which is what ako/mxcli#2414 measured, and quoting everything would +// change every query this tool has ever generated. +// +// The quote is the double quote, which PostgreSQL and Oracle take and SQL +// Server takes with QUOTED_IDENTIFIER on -- the default over JDBC, which is how +// a Database Connector reaches it. +func quoteIdent(ident string) string { + if !needsQuoting(ident) { + return ident + } + return `"` + strings.ReplaceAll(ident, `"`, `""`) + `"` +} + +func needsQuoting(ident string) bool { + for i, r := range ident { + switch { + case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r == '_': + case r >= '0' && r <= '9' && i > 0: + default: + return true + } + } + return false +} + func sqlString(s string) string { return "'" + strings.ReplaceAll(s, "'", "''") + "'" } // oracleTableFilter builds a WHERE clause for Oracle's all_tab_columns. diff --git a/sql/naming.go b/sql/naming.go index 77f8fa0fa6..848fee9207 100644 --- a/sql/naming.go +++ b/sql/naming.go @@ -31,8 +31,8 @@ func toPascalCase(s string) string { return s } - // Handle already-PascalCase names (no underscores, starts with upper) - if !strings.Contains(s, "_") && !strings.Contains(s, "-") { + // Handle already-PascalCase names (no separators, starts with upper) + if !strings.ContainsAny(s, "_- .") { // Just capitalize first letter runes := []rune(s) runes[0] = unicode.ToUpper(runes[0]) @@ -42,7 +42,10 @@ func toPascalCase(s string) string { var b strings.Builder capitalize := true for _, r := range s { - if r == '_' || r == '-' || r == ' ' { + // A dot separates words here. Left in, a table named after a Kafka + // topic produced `ENTITY LakeSql.LocalCdc.cdcpublish.customer.1.local`, + // which is not a Mendix name at all (ako/mxcli#2422). + if r == '_' || r == '-' || r == ' ' || r == '.' { capitalize = true continue } diff --git a/sql/schema.go b/sql/schema.go index 68732c0a13..4b0303bb6e 100644 --- a/sql/schema.go +++ b/sql/schema.go @@ -28,10 +28,11 @@ type TableSchema struct { // schema it was discovered in, when one is known. A database reports no schema // for some drivers, and there the bare name is the address (ako/mxcli#2414). func (t *TableSchema) QualifiedName() string { + name := quoteIdent(t.Name) if t.Schema == "" { - return t.Name + return name } - return t.Schema + "." + t.Name + return quoteIdent(t.Schema) + "." + name } // tableAddresses is the address of every object a SHOW TABLES / SHOW VIEWS @@ -40,10 +41,33 @@ func (t *TableSchema) QualifiedName() string { // a test covers (ako/mxcli#2414). func tableAddresses(rows [][]any) []string { var out []string + for _, o := range discoveredObjects(rows) { + out = append(out, o.address) + } + return out +} + +// discoveredObject is one row of a SHOW TABLES listing: where the object is and +// what it is. Every driver's listing returns table_type as its third column, +// and reading it is what keeps a view out of the set of tables -- the +// conflation that made one view generate two entities (ako/mxcli#2415). +type discoveredObject struct { + address string + isView bool +} + +func discoveredObjects(rows [][]any) []discoveredObject { + var out []discoveredObject for _, row := range rows { - if addr := qualifiedTableName(row); addr != "" { - out = append(out, addr) + addr := qualifiedTableName(row) + if addr == "" { + continue + } + isView := false + if len(row) > 2 { + isView = strings.EqualFold(strings.TrimSpace(fmt.Sprintf("%v", row[2])), "VIEW") } + out = append(out, discoveredObject{address: addr, isView: isView}) } return out } @@ -54,16 +78,18 @@ func tableAddresses(rows [][]any) []string { func qualifiedTableName(row []any) string { if len(row) < 2 { if len(row) == 1 { - return fmt.Sprintf("%v", row[0]) + return quoteIdent(fmt.Sprintf("%v", row[0])) } return "" } - name := fmt.Sprintf("%v", row[1]) + // Quoted, so the address can be split back into the same two parts: a + // listed name may contain dots of its own (ako/mxcli#2422). + name := quoteIdent(fmt.Sprintf("%v", row[1])) schema := fmt.Sprintf("%v", row[0]) if schema == "" { return name } - return schema + "." + name + return quoteIdent(schema) + "." + name } // pkQuery returns the query to find primary key columns for a given table. @@ -112,6 +138,16 @@ func ReadTableSchema(ctx context.Context, conn *Connection, tableName string, is // Schema and the bare name in Name: the name is what every Mendix entity // and query is called after (ako/mxcli#2414). schemaName, bareName := splitTableName(tableName) + if schemaName == "" { + // A name the user wrote bare still has a schema, and the generated + // query has to address it. DescribeTable has already refused a name + // more than one schema holds, so there is at most one here. Resolving + // it is also what makes `TABLES (customer)` and `VIEWS (sales.customer)` + // the same object rather than two entities of one name (ako/mxcli#2415). + if holders := schemasHolding(ctx, conn, bareName); len(holders) == 1 { + schemaName = holders[0] + } + } schema := &TableSchema{ Schema: schemaName, Name: bareName, @@ -138,22 +174,54 @@ func ReadTableSchema(ctx context.Context, conn *Connection, tableName string, is return schema, nil } -// ReadAllTableSchemas reads schemas for all user tables from the connection. -func ReadAllTableSchemas(ctx context.Context, conn *Connection) ([]*TableSchema, error) { +// ReadAllSchemas reads every object the connection's table listing names -- +// base tables and views alike -- each marked with what it is. This is what a +// GENERATE CONNECTOR with no TABLES/VIEWS clause enumerates: filtering views +// out here would empty the output on a database whose objects are all views, +// which is what a CDC lake is. +// +// On SQL Server the listing itself is restricted to base tables, so no views +// come back; name them in a VIEWS clause there. +func ReadAllSchemas(ctx context.Context, conn *Connection) ([]*TableSchema, error) { + schemas, _, err := readListed(ctx, conn, false) + return schemas, err +} + +// readListed reads the listed objects, optionally only the base tables. Its +// second return is every listed object it could not read, as "address: reason". +// Dropping those silently is what hid ako/mxcli#2422 and the ambiguous-name +// half of ako/mxcli#2414: an object the listing names and the output omits is +// invisible to everything downstream, so the caller has to be able to say so. +func readListed(ctx context.Context, conn *Connection, tablesOnly bool) ([]*TableSchema, []string, error) { result, err := ShowTables(ctx, conn) if err != nil { - return nil, err + return nil, nil, err } - - var schemas []*TableSchema - for _, addr := range tableAddresses(result.Rows) { - ts, err := ReadTableSchema(ctx, conn, addr, false) + var ( + schemas []*TableSchema + skipped []string + ) + for _, o := range discoveredObjects(result.Rows) { + if tablesOnly && o.isView { + continue + } + ts, err := ReadTableSchema(ctx, conn, o.address, o.isView) if err != nil { - continue // skip tables that fail + skipped = append(skipped, fmt.Sprintf("%s: %v", o.address, err)) + continue } + ts.IsView = o.isView schemas = append(schemas, ts) } - return schemas, nil + return schemas, skipped, nil +} + +// ReadAllTableSchemas reads schemas for the connection's base tables. A view is +// not a table: handing one back here is what let a view named in a VIEWS clause +// be generated twice (ako/mxcli#2415). Use ReadAllSchemas for everything. +func ReadAllTableSchemas(ctx context.Context, conn *Connection) ([]*TableSchema, error) { + schemas, _, err := readListed(ctx, conn, true) + return schemas, err } // ReadAllViewSchemas reads schemas for all user views from the connection.