diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-17-association-owner-both-ce0002.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-17-association-owner-both-ce0002.json new file mode 100644 index 0000000000..4f1e778e19 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-17-association-owner-both-ce0002.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-11-17", "refs": ["ako/mxcli#1258"], "ce": ["CE0002"], "symptom": "`create association M.A_A from M.A to M.A owner both` (and child->parent, parent->child, a longer chain, Reference and ReferenceSet alike) passed `mxcli check` and `exec`; `mx check` then reported CE0002. An unrelated pair with owner both, and every other owner, built with 0 errors.", "file": "mdl/executor/validate_association_owner_both.go ValidateAssociationOwnerBoth (MDL099), called from validate_program.go. Tests: validate_association_owner_both_test.go; fixtures mdl-examples/bug-tests/association-owner-both-self(.fail).mdl", "cause": "Nothing compared an association's two ends when owner is Both. A comment in validate_retrieve_association_list.go already said 'Owner Both on a self-reference is CE0002' as a reason a case could not arise, so the fact was known in the codebase but never enforced as a rule.", "insight": "**A code comment that cites a CE number as the reason a case 'cannot arise' is a pointer to an unenforced rule: grep the source for `CE0` in comments before measuring a gap list from scratch.** Boundary measured on MxBuild 11.16.1, 7 shapes in separate project copies: self, child->parent, parent->child, grandchild chain and ReferenceSet versions are CE0002; owner default and an unrelated pair are clean. Only generalizations the script declares are followed (an entity extending one in the project still reaches mx check), so it refuses less, not more. Sweep: 1,312 repo .mdl files, 0 MDL099 hits. Control: stubbing the rule fails the table test; the control fixture builds with 0 errors."} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-17-catalog-freshness-compared-whole-seconds.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-17-catalog-freshness-compared-whole-seconds.json new file mode 100644 index 0000000000..f3f910b702 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-17-catalog-freshness-compared-whole-seconds.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-11-17", "refs": ["ako/mxcli#2374"], "ce": [], "symptom": "TestSnapshotDependents_FindsAssociationsWithoutABaseline (cmd/mxcli/marketplace) failed intermittently: `dependents = []; want MyFirstModule.Config_Account`. Red in CI on main at cd4076243 and on an unrelated PR twice, green at 208a4e6e6; 1 failure in 5 local runs on main. Any read that follows a write within the same second as the previous catalog build answered from the catalog as it was before the write.", "file": "mdl/catalog/catalog.go ModTimeUnchanged (new) and SetCacheInfo/GetCacheInfo (RFC3339Nano); callers mdl/executor/cmd_catalog.go catalogCurrent and isCacheValid, cmd/mxcli/cmd_describe.go catalogMatchesProject. Tests: mdl/catalog/catalog_test.go, mdl/executor/catalog_freshness_subsecond_test.go, cmd/mxcli/catalog_matches_project_subsecond_test.go, cmd/mxcli/marketplace/snapshot_same_second_test.go", "cause": "Three freshness checks compared the project file's mtime as whole Unix seconds, and the cache stored it with time.RFC3339 (whole seconds). A write in the same second as the build left the cache looking fresh. The comment justifying it ('handle timezone/precision issues') was about a real worry, but Unix() is already zone independent and an exact Equal on the stored instant has neither problem.", "insight": "**An intermittent failure whose rate is a fraction of a second over a timing window is a cache key that is too coarse: look for a whole-second timestamp compare before blaming the test or the runner.** Settled by a scratch test that Chtimes the project to a fixed instant, then moves the write +500ms (same second: stale, dependents=0) versus +1500ms (control: seen, dependents=1) - deterministic, where the original test was red 1 run in 6. Three sites compared seconds, not one; grep `.Unix() !=` near MprModTime found all of them. Caches written before the fix hold whole seconds and never equal a fractional mtime, so each project rebuilds once; no wrong answer. Control: reverting ModTimeUnchanged to a seconds compare fails the same-second subtests (marketplace, executor, describe) while the one-second-later control still passes."} diff --git a/CHANGELOG.md b/CHANGELOG.md index 88b10eca2a..715c66e5e5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **`check` rejects `owner both` between an entity and itself or its generalization** (ako/mxcli#1258) — `create association M.A_A from M.A to M.A owner both`, or from a specialization to its parent (either direction, any chain, Reference or ReferenceSet), passed `check` and `exec` and the build reported CE0002. New MDL099, which needs no project; an unrelated pair and every other owner are unaffected. Only generalizations the script declares are followed. Measured on MxBuild 11.16.1. +- **The catalog cache sees a write made in the same second as the build** (ako/mxcli#2374) — freshness compared the project file's modification time as whole seconds, so a read that followed a write inside the second of the previous catalog build answered from the catalog as it was before the write (`marketplace update` finding no dependent associations about one run in six; `describe` and catalog queries could do the same). The time is now stored and compared to the nanosecond. A cache written by an older mxcli holds whole seconds and is rebuilt once per project. - **`alter page … set (visible: roles (…))` sets the module roles** (ako/mxcli#2371) — the alter form stored `roles (…)` as the visibility expression, reported success, and left a page `mx check` rejected with CE0117. It now writes the "Visible for module roles" setting as `create page` does, keeping the widget's condition, and `roles (…) and ` sets both. - **`check` reports a `limit 1` list returned as a single object** (ako/mxcli#2364) — under `mdl 1;` `retrieve … limit 1` binds a list of one, and returning it from a microflow that returns the entity passed check and exec, then failed `mx check` with CE0117 at the End event. MDL-RETRIEVE03 now reports it and suggests `first`, `head($Item)` or a list return type. - **`sql --describe schema.table` describes the table** (ako/mxcli#2361) — the qualified name `--tables` suggests returned an empty column list and no error on PostgreSQL and SQL Server. It now matches schema and table, a table that is not found is an error naming the accepted form, and an unqualified name held by several schemas is refused with the candidates instead of listing all their columns as one table. diff --git a/cmd/mxcli/catalog_matches_project_subsecond_test.go b/cmd/mxcli/catalog_matches_project_subsecond_test.go new file mode 100644 index 0000000000..768f34dadd --- /dev/null +++ b/cmd/mxcli/catalog_matches_project_subsecond_test.go @@ -0,0 +1,51 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "os" + "path/filepath" + "testing" + "time" + + "github.com/mendixlabs/mxcli/mdl/catalog" +) + +// catalogMatchesProject decides whether `describe` may answer from the cache +// instead of scanning the project. It compared whole seconds, so a project +// modified within the second of the build still matched. +func TestCatalogMatchesProject_WriteInTheSameSecondAsTheBuild(t *testing.T) { + mpr := filepath.Join(t.TempDir(), "app.mpr") + if err := os.WriteFile(mpr, []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + fi, err := os.Stat(mpr) + if err != nil { + t.Fatal(err) + } + cat, err := catalog.New() + if err != nil { + t.Fatal(err) + } + defer cat.Close() + if err := cat.SetCacheInfo(catalog.AbsMprPath(mpr), fi.ModTime(), "11.8.0", "fast", time.Second); err != nil { + t.Fatal(err) + } + if !catalogMatchesProject(cat, mpr) { + t.Fatal("control: an unchanged project must match its cache") + } + + moved := fi.ModTime().Add(300 * time.Millisecond) + if fi.ModTime().Nanosecond() > 500_000_000 { + moved = fi.ModTime().Add(-300 * time.Millisecond) + } + if err := os.Chtimes(mpr, moved, moved); err != nil { + t.Fatal(err) + } + if got, _ := os.Stat(mpr); got.ModTime().Unix() != fi.ModTime().Unix() || got.ModTime().Equal(fi.ModTime()) { + t.Fatalf("setup: %v -> %v is not a change inside one second", fi.ModTime(), got.ModTime()) + } + if catalogMatchesProject(cat, mpr) { + t.Error("a project modified inside the second of the build still matched its cache") + } +} diff --git a/cmd/mxcli/cmd_describe.go b/cmd/mxcli/cmd_describe.go index 6b72a49404..40ff321d0e 100644 --- a/cmd/mxcli/cmd_describe.go +++ b/cmd/mxcli/cmd_describe.go @@ -518,7 +518,7 @@ func resolveViaCatalog(projectPath, name string) []string { // catalogMatchesProject reports whether the cached catalog was built from the // current project file and is still up to date — same MPR path and unchanged -// modification time (Unix seconds, mirroring isCacheValid). Any read failure or +// modification time (the same instant, mirroring isCacheValid). Any read failure or // mismatch is treated as "not fresh" so the caller live-scans (safe but slower). func catalogMatchesProject(cat *catalog.Catalog, projectPath string) bool { info, err := cat.GetCacheInfo() @@ -529,7 +529,7 @@ func catalogMatchesProject(cat *catalog.Catalog, projectPath string) bool { if err != nil { return false } - if info.MprModTime.Unix() != fi.ModTime().Unix() { + if !catalog.ModTimeUnchanged(info.MprModTime, fi.ModTime()) { return false } // Compare absolute paths when both resolve; a path mismatch means the cache diff --git a/cmd/mxcli/marketplace/snapshot_same_second_test.go b/cmd/mxcli/marketplace/snapshot_same_second_test.go new file mode 100644 index 0000000000..2037f18d70 --- /dev/null +++ b/cmd/mxcli/marketplace/snapshot_same_second_test.go @@ -0,0 +1,51 @@ +// SPDX-License-Identifier: Apache-2.0 + +package marketplace + +import ( + "os" + "testing" + "time" +) + +// TestSnapshotDependents_SeesAWriteInTheSecondOfTheFirstRead is the deterministic +// form of what TestSnapshotDependents_FindsAssociationsWithoutABaseline did by +// chance: the first read builds the catalog, the write follows it, and when both +// land in one second the second read must still see the write. The catalog cache +// compared whole seconds, so it answered from the build made before the write +// and returned no dependents (about one run in six; red on main at cd4076243). +// +// The control is the same sequence with the write a second later, which was +// always seen. +func TestSnapshotDependents_SeesAWriteInTheSecondOfTheFirstRead(t *testing.T) { + for _, tc := range []struct { + name string + after time.Duration + }{ + {"write in the same second", 500 * time.Millisecond}, + {"write a second later (control)", 1500 * time.Millisecond}, + } { + t.Run(tc.name, func(t *testing.T) { + target := copyFixture(t) + at := time.Now().Truncate(time.Second) + if err := os.Chtimes(target, at, at); err != nil { + t.Fatal(err) + } + if _, err := SnapshotDependents(target, "Administration", testBackend); err != nil { + t.Fatal(err) + } + execMDL(t, target, crossModuleAssocMDL) + written := at.Add(tc.after) + if err := os.Chtimes(target, written, written); err != nil { + t.Fatal(err) + } + deps, err := SnapshotDependents(target, "Administration", testBackend) + if err != nil { + t.Fatal(err) + } + if len(deps) != 1 || deps[0].Key.Name != crossModuleAssoc { + t.Fatalf("dependents = %+v; want %s", deps, crossModuleAssoc) + } + }) + } +} diff --git a/docs/11-proposals/PROPOSAL_kafka_event_integration.md b/docs/11-proposals/PROPOSAL_kafka_event_integration.md new file mode 100644 index 0000000000..e040dbfde5 --- /dev/null +++ b/docs/11-proposals/PROPOSAL_kafka_event_integration.md @@ -0,0 +1,476 @@ +--- +title: Kafka event integration — consumed business events and change data capture +status: proposed +date: 2026-11-17 +related: + - archive/PROPOSAL_business_events_support.md + - data/kafka-events/ + - docs/03-development/MODELSDK_ENGINE_ARCHITECTURE.md +--- + +# Kafka event integration — consumed business events and change data capture + +Mendix has two features that ride on a Kafka-compatible broker: **business events** +(an app publishes and subscribes to CloudEvents through the Business Events +marketplace module) and **change data capture** (CDC, beta: the runtime streams +entity changes). mxcli authors the *publishing* half of business events today and +nothing else. This proposal covers the rest: + +| Part | What | Today | +|---|---|---| +| A | Read side: `describe` of a consumed business event service, `list`/`describe` of CDC services, catalog | consumer describe is broken; CDC invisible | +| B | Author a consumed business event service (contract + subscriptions) | not possible | +| C | Author a CDC service (sources, exposed attributes/associations, baseline) | not possible | +| D | Run-configuration Kafka settings (`Settings$KafkaConfiguration`) | not possible | + +**Written to be built without Studio Pro.** Every BSON shape below was captured from +a Studio Pro 11.16.1-authored project, and the raw units are committed under +[`data/kafka-events/`](data/kafka-events/). The `mx check` behaviour each design +decision rests on was measured (§6) and can be re-run on Linux with a downloaded +mxbuild. What still needs Studio Pro is listed in §9 and is *not* on the critical +path. + +## 1. How it works at runtime (why the model is small) + +Measured by disassembling the Business Events module's `userlib/Business_Events-all.jar` +(module from the ComposableFactory test project, Feb 2024; newer module versions add +BYOK constants but the model contract is the same): + +- **Topic is per channel, not per event.** `ConfigurationProvider.getConsumerTopicName` + resolves, in order: the `BusinessEvents.ClientConfiguration` config map (Mendix + Cloud Event Broker, per-event `topic`/`partialTopic`), else + `businessevents..`, else the `ChannelName` constant. + `` is the `ChannelName` UUID stored in the publisher's definition. +- **One Kafka consumer per subscribed event**, group id + `--` (normalised, lowercased; prefixed variants for + a space or a config-map prefix). Every subscription reads the whole channel topic and + filters on CloudEvent `type == eventName` (`shouldCallMicroFlow`), logging others as + skipped. App instances share a group, so an event is handled once per app. +- `enable.auto.commit=false`; `auto.offset.reset` earliest/latest from "subscribe from + beginning". Poll 250 ms → map the CloudEvent onto the consumed entity → **commit it to + the database** → run the handler microflow with `executeInBackground` on task queue + `BusinessEvents.Consumer_Queue` → `commitSync()` offsets. At-least-once; deserialisation + failures go to `BusinessEvents.DeadLetterQueue`. + +**Consequence for mxcli:** a subscription in the model is only *(message name, consumed +entity, handler microflow)*. Topic, group, offsets and credentials are derived at runtime +from the module's constants and the contract's channel id. MDL needs no Kafka settings +for business events. (The constants are already authorable with +`alter settings constant @BusinessEvents.ServerUrl …`.) + +## 2. Fixtures + +All from `github.com/ako/TestApp` commit `619864b`, Mendix 11.16.1, every element +authored in Studio Pro. `*.mxunit` is the raw unit; `*.json` is a lossless decode +(binary GUIDs as base64, list markers kept as the first array element). + +| File | Content | +|---|---| +| `be-publisher-OrdersEventService.{mxunit,json}` | publisher service (what mxcli already writes) | +| `be-consumer-OrdersEventClient.{mxunit,json}` | **consumer** service: `Definition: null`, AsyncAPI `Document`, one `subscribe` operation | +| `asyncapi-OrdersEventService.yaml` | the consumer's `Document`, extracted | +| `cdc-OrdersCDCService.{mxunit,json}` | **CDC** service: 2 sources, attributes + association, baseline | +| `domainmodel-CdcPublish.{mxunit,json}` | entities/attributes/association whose GUIDs the CDC baseline references | +| `domainmodel-BusinessEventsClient.{mxunit,json}` | the consumed entity `CBE_OrderPlaceEvent` | +| `testapp-describe.mdl` | `describe` of every related element, including today's defective consumer describe | + +These are the golden inputs for codec and executor tests (copy them into +`testdata/` of the package that tests them; do not read `docs/` from a test). + +## 3. BSON — consumed business event service + +Same document type as the publisher, `BusinessEvents$BusinessEventService`, in a +different shape (from `be-consumer-OrdersEventClient.json`): + +```json +{ + "$Type": "BusinessEvents$BusinessEventService", + "Name": "OrdersEventClient", + "Definition": null, + "Document": "asyncapi: 2.2.0\ninfo:\n title: \"OrdersEventService\" ...", + "Documentation": "", "Excluded": false, "ExportLevel": "Hidden", + "OperationImplementations": [2, { + "$Type": "BusinessEvents$ServiceOperation", + "MessageName": "OrderPlaceEvent", + "Operation": "subscribe", + "Entity": "BusinessEventsClient.CBE_OrderPlaceEvent", + "Microflow": "BusinessEventsClient.HandleBE_OrderPlaceEvent" + }], + "SourceApi": null +} +``` + +Facts the implementation depends on: + +- **Publisher vs consumer is "Definition present" vs "Definition null + Document".** + The publisher stores `Document: ""`; the AsyncAPI text exists only on the consumer + (Studio Pro's *Export AsyncAPI document* → *import* in the consuming app). +- `Operation` is the lower-case string `"subscribe"` / `"publish"`; `Entity` and + `Microflow` are by-name references. The publisher writes `Microflow: ""`. +- `OperationImplementations` list marker is **2**. +- The **contract is parsed by mxbuild**: an empty `Document` with `Definition: null` + makes `mx check` throw `System.Exception: business events service should have + definitions` (measured, §6). A `MessageName` that is not in the contract **passes** + `mx check` (measured) — so mxcli's own check must catch it. +- Studio Pro generated, when the subscription was ticked: a **persistent** entity + `CBE_` `extends BusinessEvents.ConsumedBusinessEvent` with one attribute per + payload field (no access rules), and a handler microflow `HandleBE_` with a + single parameter of that entity, returning nothing. See `testapp-describe.mdl`. Its + publisher counterpart is a **non-persistent** `PBE_` extending + `BusinessEvents.PublishedBusinessEvent`. + +### The AsyncAPI document Studio Pro generates + +`asyncapi-OrdersEventService.yaml` is the reference rendering. Structure: +`asyncapi: 2.2.0`; `info.title` = service name, `version: "1.0.0"`, +`x-mendix-generated: true`; one entry under `channels` per channel, keyed by the +32-hex `ChannelName`, with `subscribe.operationId: receiveEvents` (AsyncAPI 2 +"subscribe" = consumers may subscribe; the publisher's message has `CanSubscribe: true`); +`components.messages.` with `payload.$ref` to +`components.schemas.Payload`; fixed `messageTraits.commonHeaders` (CloudEvents +headers) and `operationTraits.kafka` blocks — copy those verbatim from the fixture, +including the `\r\n` inside the `summary` strings. + +Payload type mapping. Only `Long → {type: integer, format: int64}` is in the fixture; +the rest are inferred from the format strings present in +`Mendix.Modeler.BusinessEvents.dll` (`int32`, `int64`, `number`, `boolean`, +`date-time`, enum handling) and **must be confirmed against a Studio Pro export before +the generator claims them** (§9): + +| Mendix attribute | AsyncAPI schema (to confirm except Long) | +|---|---| +| Long | `type: integer`, `format: int64` ✔ measured | +| Integer | `type: integer`, `format: int32` | +| Decimal | `type: number` | +| Boolean | `type: boolean` | +| String | `type: string` | +| DateTime | `type: string`, `format: date-time` | +| Enumeration | `type: string` + `enum: [...]` (`MessageAttribute.EnumerationDefinition`) | + +## 4. BSON — CDC service + +`ChangeDataCapture$EntityChangeDataCaptureService` (document, from +`cdc-OrdersCDCService.json`, abridged): + +```json +{ + "$Type": "ChangeDataCapture$EntityChangeDataCaptureService", + "Name": "OrdersCDCService", "ServiceName": "OrdersCDCService", "AppName": "TestApp", + "Documentation": "", "Excluded": false, "ExportLevel": "Hidden", + "Sources": [3, { + "$Type": "ChangeDataCapture$EntityChangeSource", + "Entity": "CdcPublish.Order", "ExposedName": "Order", + "VersionedSourcePointer": "<$ID of the baseline source>", + "Properties": [3, + { "$Type": "ChangeDataCapture$AttributeProperty", + "Attribute": "CdcPublish.Order.OrderId", "ExposedName": "OrderId", + "VersionedPropertyPointer": "<$ID of the baseline property>" }, + { "$Type": "ChangeDataCapture$AssociationProperty", + "Association": "CdcPublish.Order_Customer", "ExposedName": "Order_Customer", + "VersionedPropertyPointer": "..." } ] }], + "Baseline": { + "$Type": "ChangeDataCapture$VersionedEntityChangeDataCaptureService", + "Version": { "$Type": "ChangeDataCapture$Version", "Major": 1, "Minor": 1 }, + "Sources": [2, { + "$Type": "ChangeDataCapture$VersionedEntityChangeSource", + "EntityGuid": "", "ExposedName": "Order", + "Version": { "Major": 1, "Minor": 1 }, + "Properties": [2, + { "$Type": "ChangeDataCapture$VersionedProperty", + "PropertyGuid": "", "ExposedName": "OrderId", + "Type": "Int64", "AssociatedEntityGuid": "" }, + { "...": "...", "PropertyGuid": "", + "Type": "ArrayOfInt64", "AssociatedEntityGuid": "" } ] }] + } +} +``` + +Facts (each verified against `domainmodel-CdcPublish.json`): + +- **`EntityGuid` / `PropertyGuid` are the domain model's `GUID`s** of the entity, + attribute or association — not `$ID`s. For an association, `AssociatedEntityGuid` + is the GUID of the **TO** entity (`ChildPointer` side: `Order_Customer from Order to + Customer` → Customer's GUID); for an attribute it is the all-zero GUID (16 zero + bytes). All are BSON binary subtype 3/4, like every GUID. +- `VersionedSourcePointer` / `VersionedPropertyPointer` are **intra-unit pointers** to the + baseline elements' `$ID`s (CLAUDE.md: never renumber an `$ID` without its pointers). +- List markers: live `Sources` and `Properties` are **3**; baseline `Sources` and + `Properties` are **2**. +- Baseline `Type` enum (gen `PropertyType`): `ArrayOfInt64 Binary Boolean DateTime + Decimal Enumeration Int32 Int64 String`. Measured mapping: Long→`Int64`, + String→`String`, Decimal→`Decimal`, reference association→`ArrayOfInt64` (yes, even + for a single reference). Integer→`Int32`, Boolean, DateTime, Enumeration, Binary are + the obvious remaining rows — confirm (§9); AutoNumber and HashString unknown (refuse). +- HAM (11.16 metamodel) shows two runtime-only, non-writable properties on + `EntityChangeSource`: `channelName` and `debeziumSchemaContent` — the runtime + publishes Debezium-format change events. Do not write them. +- Studio Pro's own MCP schema (`ped_get_schema`) annotates: *serviceName defaults to + the document name*; *appName is set automatically by the model*; *exposedName + defaults to the entity name without module prefix*; *never author the versioned + types* (they are model-managed). mxcli has no model to manage them, so it must. + +### Gaps in the vendored gen (`modelsdk/gen/changedatacapture`) + +The gen predates 11.16 and is missing properties Studio Pro writes: + +| Type | Missing (storage name) | HAM name | +|---|---|---| +| `EntityChangeDataCaptureService` | `AppName` (string) | `appName` | +| `EntityChangeSource` | `VersionedSourcePointer` (pointer) | `versionedSource` | +| `AttributeProperty`, `AssociationProperty` | `VersionedPropertyPointer` (pointer) | `versionedProperty` | +| `VersionedProperty` | `AssociatedEntityGuid` (binary GUID) | `associatedEntityGuid` | + +The generator is not in-tree (MODELSDK_ENGINE_ARCHITECTURE.md §"gen is vendored"), so +add them **by hand to both `init` and `InitFromRaw`**, house-style commented, and +`gofmt`. CLAUDE.md rule 4 (write Studio Pro's exact property set) makes this required, +not optional: `mx check` tolerates the omission, Studio Pro may not. + +## 5. Proposed MDL + +### 5.1 Consumed business event service (part B) + +Same noun as the publisher — it is the same document type, and Studio Pro calls both +"business event service". What makes it a consumer is that its property list names a +**contract** instead of `ServiceName`/`EventNamePrefix`, following the image-collection +precedent (`image logo ( File: 'path' )` or `( Data: '…' )`, describe always printing +`Data`): + +```sql +mdl 1; +-- the contract of a publisher in this project, rendered by mxcli +create or modify business event service BusinessEventsClient.OrdersEventClient +( + ContractFrom: CdcPublish.OrdersEventService +) +{ + subscribe OrderPlaceEvent + entity BusinessEventsClient.CBE_OrderPlaceEvent + microflow BusinessEventsClient.HandleBE_OrderPlaceEvent; +}; + +-- another app's contract: the AsyncAPI document Studio Pro exported +create business event service Orders.OrdersEventClient +( + ContractFile: 'contracts/OrdersEventService.yaml' +) +{ + subscribe OrderPlaceEvent entity Orders.CBE_OrderPlaceEvent microflow Orders.HandleBE_OrderPlaceEvent; +}; + +-- what describe prints, whichever key created it +create or modify business event service Orders.OrdersEventClient +( + Contract: $$ +asyncapi: 2.2.0 +info: + title: "OrdersEventService" +... +$$ +) +{ + subscribe OrderPlaceEvent entity Orders.CBE_OrderPlaceEvent microflow Orders.HandleBE_OrderPlaceEvent; +}; +``` + +- Exactly one of `Contract` / `ContractFile` / `ContractFrom`; any of them together with + `ServiceName`, `EventNamePrefix` or a `message` in the body is an error from the + visitor (R11 key table). +- `ContractFile` resolves like an image's `File:` (relative to the script) and stores the + text verbatim as `Document`; `ContractFrom` renders the AsyncAPI from that publisher's + `Definition` (§3). Either way the stored `Document` *is* the contract — nothing later + depends on the file or the publisher. +- `Contract:` takes the lexer's existing `DOLLAR_STRING` (`$$ … $$`, already used by a + Java action's `extra code`); the property-list value rule needs to accept that token. + `describe` always prints `Contract: $$ … $$`, so its output needs no files and + round-trips. Check refuses a contract containing `$$` rather than escape it. +- `subscribe entity … [microflow …]` writes one `ServiceOperation`. + The consumed entity and handler are ordinary MDL, declared before the service + (`create persistent entity … extends BusinessEvents.ConsumedBusinessEvent (…)`, + `create microflow …`). mxcli does not generate them implicitly — statements stay + self-contained (design-mdl-syntax §3); a skill shows the three-statement recipe. +- `create or modify` carries the operation `$ID`s by message name, as + `businessEventIdentityCarry` already does for the publisher. + +**Check rules** (new `MDL-BE*` codes; mx does not catch these): + +| Rule | Condition | +|---|---| +| contract parses | the YAML is AsyncAPI 2.x with at least one channel (else mxbuild crashes) | +| message exists | each `subscribe` names a message in `components.messages` (mx passes it, §6) | +| entity shape | entity is persistent, `extends BusinessEvents.ConsumedBusinessEvent`, and has an attribute of the mapped type for each payload property | +| handler shape | microflow has exactly one parameter, of the consumed entity | +| module present | the `BusinessEvents` marketplace module is in the project | + +### 5.2 CDC service (part C) + +```sql +mdl 1; +create or modify change data capture service CdcPublish.OrdersCDCService +( + ServiceName: 'OrdersCDCService' +) +{ + entity CdcPublish.Order ( + OrderId, + CustomerName, + ProductName, + Price as 'PriceAmount', -- exposed name only when it differs + Order_Customer -- an association of the entity + ); + entity CdcPublish.Customer as 'Client' (CustomerId, CustomerName, Address); +}; + +drop change data capture service CdcPublish.OrdersCDCService; +list change data capture services [in CdcPublish]; +describe change data capture service CdcPublish.OrdersCDCService; +``` + +- `ServiceName` defaults to the document name; `AppName` is not MDL — written as the + project name (see §10) and preserved on modify. +- A member name resolves to an attribute or association of that entity (attributes win; + ambiguity is a check error). +- **The baseline is computed, not authored.** After building the live sources, mxcli + rebuilds `Baseline` so it is structurally equal to them (this is what Studio Pro's + *accept changes* does) and sets the pointers. Version policy, since `mx check` ignores + the numbers (§6) and they are a contract signal to consumers: + - create: every source and the service at `1.0`; + - modify, only additions to a source: that source `minor+1`; + - modify with a removal, a rename of an exposed name, or a type change: that source + `major+1, minor 0`; + - the service version takes the same bump as its largest source bump; + - unchanged sources keep their stored version and baseline element `$ID`s. + The fixture is consistent with this (Order is 1.1 after its association was added; + Customer 1.0; service 1.1), but the rule is inferred — §9. +- Gate: Mendix ≥ 11.10 (where `Baseline` was introduced and CE1585 enforces it). Below + that, refuse with a message rather than guess the older shape. + +### 5.3 Kafka run configuration (part D) + +Studio Pro's *App Settings → Configurations → Kafka* is `Settings$KafkaConfiguration` +on `Settings$Configuration` (HAM: `bootstrapServers`, `username`, `password`; stored +keys `BootstrapServers`, `Username`, `Password`, nested under `KafkaConfiguration`). +Extend the existing statement: + +```sql +alter settings configuration 'Default' ( + KafkaBootstrapServers: 'localhost:9092', + KafkaUsername: '', + KafkaPassword: '' +); +``` + +Overlay rule (MODELSDK_ENGINE_ARCHITECTURE.md, "never invent a key"): write only when +the stored configuration already carries a `KafkaConfiguration` part; otherwise refuse +with the Mendix version that introduced it. + +### 5.4 Read side (part A — do first) + +- **Defect today:** `describe business event service BusinessEventsClient.OrdersEventClient` + prints only `create or modify business event service …` — no contract, no + subscription, no property list, no `;`. Measured: `mxcli exec` of that output is a + parse error (`mismatched input '' expecting '('`), so nothing is lost, but the + round-trip is broken and an LLM reading it learns nothing. Fix with 5.1's describe + form; until 5.1's grammar lands, print the contract and subscriptions as comments. + Note `create or modify` of a *publisher* over an existing consumer must refuse, not + overwrite the contract. +- `list business event services` gains a `Kind` column (published / consumed). +- `list business event clients` (today: prints "not yet implemented") is removed or + becomes a filter for consumed services — see §10. +- CDC: `list` / `describe`, a catalog table `change_data_capture_services` (+ a sources + table), `RefKind`s for entity/attribute/association references so `show callers` / + impact analysis see CDC exposure, and a row in `MDL_FEATURE_MATRIX.md`. +- Catalog: bump `CatalogSchemaVersion`. + +## 6. Measured `mx check` behaviour (mxbuild 11.16.1) + +On a copy of the fixture project; each variant mutates one unit's BSON. Control: the +unmodified copy → **0 errors**. + +| Variant | Result | +|---|---| +| CDC: `Baseline = null`, pointers removed | **CE1585** "There are changes in your 'OrdersCDCService' document. Please accept the changes or undo them." | +| CDC: as above and `AppName` removed | **CE1577** "'' is not allowed. App name should start with a letter …" + CE1585 | +| CDC: pointers removed, baseline kept | 0 errors | +| CDC: every `Version` reset to 1.0 | 0 errors | +| CDC: one live property removed, baseline kept | **CE1585** | +| CDC: one live `ExposedName` changed, baseline kept | **CE1585** | +| BE consumer: `Document = ""` | mxbuild throws `System.Exception: business events service should have definitions` | +| BE consumer: `MessageName` not in contract | 0 errors | + +So: the baseline must mirror the live sources exactly (set of properties and exposed +names); version numbers and pointers are not validated by mxbuild (write them anyway, +§4); `AppName` must be a valid app name; a consumer must carry a contract. + +To re-run on Linux: `mxcli setup mxbuild --version 11.16.1`, restore the fixture units +into an `mxcli new` 11.16.1 project with matching modules/entities (or build the +entities from `testapp-describe.mdl`), mutate with a few lines of Go over +`bson.D`, and `mxcli docker check`. + +## 7. Implementation plan + +| Step | Files | Notes | +|---|---|---| +| A1 consumer describe/list | `mdl/executor/cmd_businessevents.go` | fix the lossy describe first; it is a data-loss bug on round-trip | +| A2 CDC read | new `mdl/backend/modelsdk/cdc_read.go`, `mdl/backend/service.go` (interface), mock, `mdl/executor/cmd_cdc.go`, catalog builder + tables, registry | gen types exist; add the 4 missing properties (§4) first so reads see them | +| B1 grammar/AST | `MDLParser.g4` (`Contract`/`ContractFile`/`ContractFrom` keys, `DOLLAR_STRING` as a property value, `subscribe` body item), `ast_businessevents.go`, `visitor_businessevents.go` | | +| B2 AsyncAPI renderer | new `mdl/businessevents/asyncapi.go` | golden test = `asyncapi-OrdersEventService.yaml`, byte-for-byte | +| B3 write | `mdl/backend/modelsdk/business_event_write.go` | `Definition` null, `Document`, subscribe ops; identity carry by message name | +| B4 check rules | `mdl/executor/validate_*` | §5.1 table | +| C1 grammar/AST/visitor | new `ast_cdc.go`, `visitor_cdc.go` | | +| C2 write | new `mdl/backend/modelsdk/cdc_write.go` | live sources + computed baseline + pointers; GUIDs read from the domain model; wire through `canon.Reconcile`; consider a `canon.identityFields` row (identity = `Entity` / `Attribute` / `Association` by name) | +| C3 version policy | `cdc_write.go` | §5.2 | +| D1 settings | settings overlay | §5.3 | +| docs | `docs-site/src/`, `cmd/mxcli/syntax/`, a skill `.claude/skills/mendix/business-events/SKILL.md` | the consumer recipe and the runtime facts in §1 | + +MCP backend: out of scope here. Studio Pro 11.16.1's PED has **no** create path for +`BusinessEvents$BusinessEventService` (`ped_read_document` answers "Unknown document +type") but **does** return a constructor schema for +`ChangeDataCapture$EntityChangeDataCaptureService` (`name`, `appName`, `serviceName`), +so CDC over `--mcp` is likely feasible as a follow-up — it needs a live Studio Pro. + +## 8. Test plan + +- **Codec:** decode each fixture `.mxunit`; re-encode with no change → byte-identical + (proves the hand-added gen properties and list markers). +- **Writer, golden:** build the consumer and CDC services from MDL against an in-memory + project containing the fixture domain models; compare to the fixture JSON ignoring + `$ID` values (but checking pointer consistency: every `Versioned*Pointer` resolves to a + baseline `$ID` in the same unit). +- **GUID test with a control** (CLAUDE.md §GUID): the fixture entities are Studio + Pro-authored, so `GUID != $ID` — assert `EntityGuid == entity.GUID` *and* `!= entity.$ID`, + so a writer that used `$ID` fails. +- **Modify:** `create or modify` adding `ProductName` to a source keeps every other + baseline `$ID`, bumps that source's minor, and is elided entirely when re-run unchanged + (ADR-0008 conditional write; include a control that a real change does write). +- **Describe round-trip:** describe → exec → describe is a fixed point for both + documents; `mdl-examples/doctype-tests/` gets a consumer and a CDC script. +- **Integration (`-tags integration`):** each §6 variant as a test — the valid writes pass + `mx check`; a writer forced to omit the baseline reproduces CE1585 (the control that + proves the test detects the defect). + +## 9. Needs Studio Pro later (not blocking) + +1. AsyncAPI rendering of Integer, Decimal, Boolean, String, DateTime and Enumeration + payload fields: add one attribute of each to `OrdersEventService`, export, diff + against the generator. Until then the renderer refuses those types for + `ContractFrom` (`Contract` / `ContractFile` are unaffected). +2. CDC version policy (§5.2): add, rename and remove a property in Studio Pro, accept, + and record the versions it picks. +3. CDC `Type` for Integer/Boolean/DateTime/Enumeration/Binary/AutoNumber/HashString. +4. Whether Studio Pro (not just mxbuild) opens a document with the pointers present but + pointing at mxcli-minted baseline `$ID`s — expected yes; confirm once. +5. What registers the subscriptions at startup: no microflow in the project calls + `BusinessEvents.ConsumeBusinessEvent`, and no 11.15 runtime jar references it by + name. Not needed to author the model; needed for the skill's troubleshooting section. + +## 10. Open questions + +1. **`AppName`:** the fixture holds the project name (`TestApp`). Write the `.mpr` + basename, or read it from somewhere authoritative (project settings)? Must satisfy + CE1577's pattern either way. +2. **`list business event clients`:** remove it, or keep it as `list … services` filtered + to consumed ones? (Recommendation: remove; one noun, a `Kind` column.) +3. Should a service with `ContractFrom: X` also *create* the + `CBE_` entity and `HandleBE_` microflow when they are absent, as Studio Pro does? + Recommendation: no (explicit statements), but `mxcli` could offer a + `--scaffold`-style generator later. diff --git a/docs/11-proposals/README.md b/docs/11-proposals/README.md index d8fe06102f..2c0bf9ee71 100644 --- a/docs/11-proposals/README.md +++ b/docs/11-proposals/README.md @@ -27,7 +27,7 @@ for display in this README): -## Active Proposals (123) +## Active Proposals (125) ### In Progress (partial) (18) @@ -62,7 +62,7 @@ for display in this README): | [MDL Language Critique and Beta Syntax Freeze](PROPOSAL_mdl_beta_syntax_freeze.md) | Accepted | Related: ADR-0003; resulting decisions ADR-0010, ADR-0011, ADR-0012; design-mdl-syntax skill, PROPOSAL_workflow_microflow_syntax_alignment.m | | [Size containers from their contents, not from a statement count](PROPOSAL_container_autosize.md) | Accepted | Closes the last open item of upstream #884 — | -### Proposed (39) +### Proposed (40) | Proposal | Status | Summary | |----------|--------|---------| @@ -73,6 +73,7 @@ for display in this README): | [Consumed REST Services (SHOW / DESCRIBE / CREATE)](show-describe-consumed-rest-services.md) | Proposed | Document type: rest$ConsumedRestService | | [Feature Request: Entity Access Rules in Starlark Lint API](starlark-security.md) | Proposed | The CATALOG.PERMISSIONS table (added in a recent mxcli release) already contains full entity access rule data — including module role, acces | | [Improving LLM Assistance for MDL Code Generation](PROPOSAL_llm_mdl_assistance.md) | Proposed | MDL (Mendix Definition Language) is a custom DSL that does not exist in LLM training data. | +| [Kafka event integration — consumed business events and change data capture](PROPOSAL_kafka_event_integration.md) | Proposed | Mendix has two features that ride on a Kafka-compatible broker: business events | | [Large Source Files](refactor-large-files.md) | Proposed | This proposal addresses the refactoring of 6 large non-generated source files to improve maintainability and extensibility, especially for a | | [Mendix Automated Testing Pipeline](proposal-playwright-testing.md) | Proposed | Implementation Proposal for Claude Code | | [mxcli developer panel — ask the agent about the interaction you just performed](proposal-browser-dev-panel.md) | Proposed | mxcli run --local produces four runtime signals — logs, metrics, traces, and the | @@ -106,7 +107,7 @@ for display in this README): | [VS Code Search — Quick Pick + Workspace Symbol](PROPOSAL_vscode_search.md) | Proposed | Full-text search exists in mxcli (mxcli search) but is only accessible via the terminal. | | [Workflow Improvements: ALTER WORKFLOW + Cross-References](PROPOSAL_workflow_improvements.md) | Proposed | Workflow support in mxcli has full CREATE/DESCRIBE/DROP/SHOW coverage with 13 activity types and BSON round-trip fidelity. | -### Draft (51) +### Draft (52) | Proposal | Status | Summary | |----------|--------|---------| @@ -114,7 +115,6 @@ for display in this README): | [Agent loop efficiency — making an mxcli session cost what the work costs](PROPOSAL_agent_loop_efficiency.md) | Draft | measured with diag loop-report, overturning the check-dominant assumption and | | [Architecture Graph Visualization (communities, layers, god-nodes)](PROPOSAL_architecture_graph_visualization.md) | Draft | Issue: TBD (file before implementation) | | [Association Mapping in IMPORT](PROPOSAL_import_associations.md) | Draft | Parent: PROPOSAL_mxcli_sql.md (Phase 3 extension) | -| [Associations to Include on a Call External Action Argument](PROPOSAL_external_action_include_associations.md) | Draft | Issue: ako/mxcli#1914 (mendixlabs/mxcli#1721) | | [Authorable layouts, page templates and building blocks — CREATE, not COPY](PROPOSAL_authorable_layouts.md) | Draft | An app built entirely through MDL cannot put anything in its own topbar. | | [Bulk Change Custom Widget Properties](PROPOSAL_bulk_widget_property_updates.md) | Draft | Custom widgets (pluggable widgets) in Mendix have complex nested property structures. | | [Bulk External Action Support from OData Contracts](PROPOSAL_external_actions_bulk_create.md) | Draft | Issue #143 requests importing all entities and actions from a consumed OData service. | @@ -148,6 +148,7 @@ for display in this README): | [mxcli companion, slice 2 — a long-lived mxcli the mod talks to over a Unix socket](proposal-mxcli-companion-serve.md) | Draft | Slice 2 of the companion umbrella. | | [mxcli marketplace diff — detect local modification and plan a GUID-preserving module upgrade](PROPOSAL_marketplace_module_upgrade.md) | Draft | 2026-08-11 (DESCRIBE coverage measured — §7; GUID = database identity measured — §8) | | [mxcli Playground](mxcli-playground.md) | Draft | A public GitHub repository (mendixlabs/mxcli-playground) containing a ready-to-use Mendix project pre-configured with mxcli, Claude Code ski | +| [mxcli verify — one gate that reports only what a change made worse](PROPOSAL_verify_gate.md) | Draft | Issue: ako/mxcli#2333 | | [Offline synchronization configuration — the one thing an offline profile needs that MDL cannot say](PROPOSAL_offline_sync_configuration.md) | Draft | mxcli can author an offline navigation profile — PhoneOffline, | | [OpenAPI import — generate response entities and wire operations to them](PROPOSAL_openapi_response_entities.md) | Draft | Issue: ako/mxcli#1234 (mendixlabs/mxcli#280) | | [Owning the modelsdk/gen codegen — why the vendored generator cannot be adopted as-is](PROPOSAL_codegen_ownership.md) | Draft | Constraint set by the maintainer (2026-08-13): the reflection data is an | diff --git a/docs/11-proposals/data/kafka-events/asyncapi-OrdersEventService.yaml b/docs/11-proposals/data/kafka-events/asyncapi-OrdersEventService.yaml new file mode 100644 index 0000000000..e9936d782d --- /dev/null +++ b/docs/11-proposals/data/kafka-events/asyncapi-OrdersEventService.yaml @@ -0,0 +1,88 @@ +asyncapi: 2.2.0 +info: + title: "OrdersEventService" + version: "1.0.0" + description: "" + x-summary: "" + x-mendix-generated: true +channels: + f2477dc90f714d69bb12df0df1ee9cce: + subscribe: + operationId: receiveOrderPlaceEventEvents + traits: + - $ref: '#/components/operationTraits/kafka' + message: + $ref: '#/components/messages/OrderPlaceEvent' +components: + messages: + OrderPlaceEvent: + name: OrderPlaceEvent + title: OrderPlaceEvent event + description: "" + contentType: application/json + traits: + - $ref: '#/components/messageTraits/commonHeaders' + payload: + $ref: '#/components/schemas/OrderPlaceEventPayload' + schemas: + OrderPlaceEventPayload: + type: object + properties: + OrderId: + type: integer + format: int64 + CustomerId: + type: integer + format: int64 + messageTraits: + commonHeaders: + name: cloudevents-headers + summary: "\r\nMessage headers used by CloudEvents spec in binary content mode Kafka\r\nbinding (see\r\nhttps://github.com/cloudevents/spec/blob/v1.0.1/kafka-protocol-binding.md)" + headers: + type: object + required: + - ce_specversion + - ce_id + - ce-source + - ce_type + properties: + ce_specversion: + type: string + enum: + - 1.0 + description: "The version of the CloudEvents specification which the event uses." + ce_id: + type: string + description: "Identifies the event." + minLength: 1 + ce_source: + type: string + format: uri-reference + description: "Identifies the context in which an event happened." + minLength: 1 + ce_type: + type: string + description: "Describes the type of event related to the originating occurrence." + minLength: 1 + ce_service: + type: string + description: "Describes the service name in which this event was defined." + ce_subject: + type: string + description: "Describes the subject of the event in the context of the event\r\nproducer (identified by source)." + ce_time: + type: string + format: date-time + description: "Timestamp of when the occurrence happened. Must adhere to RFC\r\n3339." + content-type: + type: string + enum: + - application/json + operationTraits: + kafka: + bindings: + kafka: + key: + type: string + description: "Event Id" +defaultContentType: application/json diff --git a/docs/11-proposals/data/kafka-events/be-consumer-OrdersEventClient.json b/docs/11-proposals/data/kafka-events/be-consumer-OrdersEventClient.json new file mode 100644 index 0000000000..26549ae53b --- /dev/null +++ b/docs/11-proposals/data/kafka-events/be-consumer-OrdersEventClient.json @@ -0,0 +1,22 @@ +{ + "$ID": "FDWQWXR2YkadD2ikKCOuLQ==", + "$Type": "BusinessEvents$BusinessEventService", + "Definition": null, + "Document": "asyncapi: 2.2.0\ninfo:\n title: \"OrdersEventService\"\n version: \"1.0.0\"\n description: \"\"\n x-summary: \"\"\n x-mendix-generated: true\nchannels:\n f2477dc90f714d69bb12df0df1ee9cce:\n subscribe:\n operationId: receiveOrderPlaceEventEvents\n traits:\n - $ref: '#/components/operationTraits/kafka'\n message:\n $ref: '#/components/messages/OrderPlaceEvent'\ncomponents:\n messages:\n OrderPlaceEvent:\n name: OrderPlaceEvent\n title: OrderPlaceEvent event\n description: \"\"\n contentType: application/json\n traits:\n - $ref: '#/components/messageTraits/commonHeaders'\n payload:\n $ref: '#/components/schemas/OrderPlaceEventPayload'\n schemas:\n OrderPlaceEventPayload:\n type: object\n properties:\n OrderId:\n type: integer\n format: int64\n CustomerId:\n type: integer\n format: int64\n messageTraits:\n commonHeaders:\n name: cloudevents-headers\n summary: \"\\r\\nMessage headers used by CloudEvents spec in binary content mode Kafka\\r\\nbinding (see\\r\\nhttps://github.com/cloudevents/spec/blob/v1.0.1/kafka-protocol-binding.md)\"\n headers:\n type: object\n required:\n - ce_specversion\n - ce_id\n - ce-source\n - ce_type\n properties:\n ce_specversion:\n type: string\n enum:\n - 1.0\n description: \"The version of the CloudEvents specification which the event uses.\"\n ce_id:\n type: string\n description: \"Identifies the event.\"\n minLength: 1\n ce_source:\n type: string\n format: uri-reference\n description: \"Identifies the context in which an event happened.\"\n minLength: 1\n ce_type:\n type: string\n description: \"Describes the type of event related to the originating occurrence.\"\n minLength: 1\n ce_service:\n type: string\n description: \"Describes the service name in which this event was defined.\"\n ce_subject:\n type: string\n description: \"Describes the subject of the event in the context of the event\\r\\nproducer (identified by source).\"\n ce_time:\n type: string\n format: date-time\n description: \"Timestamp of when the occurrence happened. Must adhere to RFC\\r\\n3339.\"\n content-type:\n type: string\n enum:\n - application/json\n operationTraits:\n kafka:\n bindings:\n kafka:\n key:\n type: string\n description: \"Event Id\"\ndefaultContentType: application/json\n", + "Documentation": "", + "Excluded": false, + "ExportLevel": "Hidden", + "Name": "OrdersEventClient", + "OperationImplementations": [ + 2, + { + "$ID": "HaoHsjsuJ0OMRZfT4oHkBg==", + "$Type": "BusinessEvents$ServiceOperation", + "Entity": "BusinessEventsClient.CBE_OrderPlaceEvent", + "MessageName": "OrderPlaceEvent", + "Microflow": "BusinessEventsClient.HandleBE_OrderPlaceEvent", + "Operation": "subscribe" + } + ], + "SourceApi": null +} diff --git a/docs/11-proposals/data/kafka-events/be-consumer-OrdersEventClient.mxunit b/docs/11-proposals/data/kafka-events/be-consumer-OrdersEventClient.mxunit new file mode 100644 index 0000000000..5d97c44939 Binary files /dev/null and b/docs/11-proposals/data/kafka-events/be-consumer-OrdersEventClient.mxunit differ diff --git a/docs/11-proposals/data/kafka-events/be-publisher-OrdersEventService.json b/docs/11-proposals/data/kafka-events/be-publisher-OrdersEventService.json new file mode 100644 index 0000000000..3364ed1395 --- /dev/null +++ b/docs/11-proposals/data/kafka-events/be-publisher-OrdersEventService.json @@ -0,0 +1,74 @@ +{ + "$ID": "sDWw3K2G3ke4jw1vyj2JIg==", + "$Type": "BusinessEvents$BusinessEventService", + "Definition": { + "$ID": "2/8tIBboPUWydCebss/zmQ==", + "$Type": "BusinessEvents$BusinessEventDefinition", + "Channels": [ + 2, + { + "$ID": "2bj71HSNLEudbgOzYtmvkQ==", + "$Type": "BusinessEvents$Channel", + "ChannelName": "f2477dc90f714d69bb12df0df1ee9cce", + "Description": "", + "Messages": [ + 2, + { + "$ID": "/+scPugMmUmW6uX32NpN0w==", + "$Type": "BusinessEvents$Message", + "Attributes": [ + 2, + { + "$ID": "JihY/BScc0SL50ODkNrCmQ==", + "$Type": "BusinessEvents$MessageAttribute", + "AttributeName": "OrderId", + "AttributeType": { + "$ID": "05oJpqEDaU6kzm/E80XWKA==", + "$Type": "DomainModels$LongAttributeType" + }, + "Description": "", + "EnumerationDefinition": null + }, + { + "$ID": "27CIQFu2QkSPY2Oucs+0Zw==", + "$Type": "BusinessEvents$MessageAttribute", + "AttributeName": "CustomerId", + "AttributeType": { + "$ID": "jtLV1JnSgkS9t7FDt02oXg==", + "$Type": "DomainModels$LongAttributeType" + }, + "Description": "", + "EnumerationDefinition": null + } + ], + "CanPublish": false, + "CanSubscribe": true, + "Description": "", + "MessageName": "OrderPlaceEvent" + } + ] + } + ], + "Description": "", + "EventNamePrefix": "", + "ServiceName": "OrdersEventService", + "Summary": "" + }, + "Document": "", + "Documentation": "", + "Excluded": false, + "ExportLevel": "Hidden", + "Name": "OrdersEventService", + "OperationImplementations": [ + 2, + { + "$ID": "NkDp8wwM10S9QqbbMDIapA==", + "$Type": "BusinessEvents$ServiceOperation", + "Entity": "CdcPublish.PBE_OrderPlaceEvent", + "MessageName": "OrderPlaceEvent", + "Microflow": "", + "Operation": "publish" + } + ], + "SourceApi": null +} diff --git a/docs/11-proposals/data/kafka-events/be-publisher-OrdersEventService.mxunit b/docs/11-proposals/data/kafka-events/be-publisher-OrdersEventService.mxunit new file mode 100644 index 0000000000..81f2d242cb Binary files /dev/null and b/docs/11-proposals/data/kafka-events/be-publisher-OrdersEventService.mxunit differ diff --git a/docs/11-proposals/data/kafka-events/cdc-OrdersCDCService.json b/docs/11-proposals/data/kafka-events/cdc-OrdersCDCService.json new file mode 100644 index 0000000000..77b1fa5fce --- /dev/null +++ b/docs/11-proposals/data/kafka-events/cdc-OrdersCDCService.json @@ -0,0 +1,196 @@ +{ + "$ID": "XJcmnizTcUGevz8cZk2vnQ==", + "$Type": "ChangeDataCapture$EntityChangeDataCaptureService", + "AppName": "TestApp", + "Baseline": { + "$ID": "1tkN9G4D/UCRdP1O2htHvQ==", + "$Type": "ChangeDataCapture$VersionedEntityChangeDataCaptureService", + "Sources": [ + 2, + { + "$ID": "kA3kcKWz0kys+quBWLm8iQ==", + "$Type": "ChangeDataCapture$VersionedEntityChangeSource", + "EntityGuid": "4WR5BXj9dE69ZsdKBhMVEQ==", + "ExposedName": "Order", + "Properties": [ + 2, + { + "$ID": "r3o8+Pi3EkmU04X/VY1lkQ==", + "$Type": "ChangeDataCapture$VersionedProperty", + "AssociatedEntityGuid": "AAAAAAAAAAAAAAAAAAAAAA==", + "ExposedName": "OrderId", + "PropertyGuid": "n6q889hO3kabfdrYFCW8Mw==", + "Type": "Int64" + }, + { + "$ID": "rUsjPFMmd0qQ48Hys5P7Ow==", + "$Type": "ChangeDataCapture$VersionedProperty", + "AssociatedEntityGuid": "AAAAAAAAAAAAAAAAAAAAAA==", + "ExposedName": "CustomerName", + "PropertyGuid": "cVSRjNaSskCeuva7LtNWLA==", + "Type": "String" + }, + { + "$ID": "3fKZjmvtuk6HRkcEsWmm1w==", + "$Type": "ChangeDataCapture$VersionedProperty", + "AssociatedEntityGuid": "AAAAAAAAAAAAAAAAAAAAAA==", + "ExposedName": "ProductName", + "PropertyGuid": "0R7AyIdW9kaxFiUajfBG2Q==", + "Type": "String" + }, + { + "$ID": "rJCDqTQqh0i+cTP+elVmNQ==", + "$Type": "ChangeDataCapture$VersionedProperty", + "AssociatedEntityGuid": "AAAAAAAAAAAAAAAAAAAAAA==", + "ExposedName": "Price", + "PropertyGuid": "DVFayQ9N0EivYrn/9+xydw==", + "Type": "Decimal" + }, + { + "$ID": "1tbI0VDneU2hMPfZru9wwA==", + "$Type": "ChangeDataCapture$VersionedProperty", + "AssociatedEntityGuid": "cvHjuruupkmtkng/U89QFA==", + "ExposedName": "Order_Customer", + "PropertyGuid": "rd9XQAxoy0S/0pdwa5GdPg==", + "Type": "ArrayOfInt64" + } + ], + "Version": { + "$ID": "1rwm1TV+skmCyaufPkb5QQ==", + "$Type": "ChangeDataCapture$Version", + "Major": 1, + "Minor": 1 + } + }, + { + "$ID": "NvM40bLHVEGsjmCOHF2vMg==", + "$Type": "ChangeDataCapture$VersionedEntityChangeSource", + "EntityGuid": "cvHjuruupkmtkng/U89QFA==", + "ExposedName": "Customer", + "Properties": [ + 2, + { + "$ID": "CtSi0JbPZ0yooiAFs66PJQ==", + "$Type": "ChangeDataCapture$VersionedProperty", + "AssociatedEntityGuid": "AAAAAAAAAAAAAAAAAAAAAA==", + "ExposedName": "CustomerId", + "PropertyGuid": "EJMsBnbJHkiH+0X5aC13cA==", + "Type": "Int64" + }, + { + "$ID": "e5z3yWUI2k+fnUgj+RMSig==", + "$Type": "ChangeDataCapture$VersionedProperty", + "AssociatedEntityGuid": "AAAAAAAAAAAAAAAAAAAAAA==", + "ExposedName": "CustomerName", + "PropertyGuid": "24ASpoOGMkKUdy+TZGTKaQ==", + "Type": "String" + }, + { + "$ID": "EMn6RtUoiUmdEvQFu1BERg==", + "$Type": "ChangeDataCapture$VersionedProperty", + "AssociatedEntityGuid": "AAAAAAAAAAAAAAAAAAAAAA==", + "ExposedName": "Address", + "PropertyGuid": "kuE7E74iaEa9XB0ly38FvA==", + "Type": "String" + } + ], + "Version": { + "$ID": "Ax6wt635lUSVrUeuPOsdbw==", + "$Type": "ChangeDataCapture$Version", + "Major": 1, + "Minor": 0 + } + } + ], + "Version": { + "$ID": "y2yrouvW902oKcCHITzRlA==", + "$Type": "ChangeDataCapture$Version", + "Major": 1, + "Minor": 1 + } + }, + "Documentation": "", + "Excluded": false, + "ExportLevel": "Hidden", + "Name": "OrdersCDCService", + "ServiceName": "OrdersCDCService", + "Sources": [ + 3, + { + "$ID": "6ytDr9odtUCVd2W4hYHekw==", + "$Type": "ChangeDataCapture$EntityChangeSource", + "Entity": "CdcPublish.Order", + "ExposedName": "Order", + "Properties": [ + 3, + { + "$ID": "jtKkmQXZPEGAbNEDqZToNg==", + "$Type": "ChangeDataCapture$AttributeProperty", + "Attribute": "CdcPublish.Order.OrderId", + "ExposedName": "OrderId", + "VersionedPropertyPointer": "r3o8+Pi3EkmU04X/VY1lkQ==" + }, + { + "$ID": "ZxFZh2nJDk6Hyjl8X0dOGQ==", + "$Type": "ChangeDataCapture$AttributeProperty", + "Attribute": "CdcPublish.Order.CustomerName", + "ExposedName": "CustomerName", + "VersionedPropertyPointer": "rUsjPFMmd0qQ48Hys5P7Ow==" + }, + { + "$ID": "c7WUiqyE4Uu/p9p6rObngA==", + "$Type": "ChangeDataCapture$AttributeProperty", + "Attribute": "CdcPublish.Order.ProductName", + "ExposedName": "ProductName", + "VersionedPropertyPointer": "3fKZjmvtuk6HRkcEsWmm1w==" + }, + { + "$ID": "oWK7PIAQoU2BUYgaDMja4g==", + "$Type": "ChangeDataCapture$AttributeProperty", + "Attribute": "CdcPublish.Order.Price", + "ExposedName": "Price", + "VersionedPropertyPointer": "rJCDqTQqh0i+cTP+elVmNQ==" + }, + { + "$ID": "wf/9yNUTG0S8fmzJKKexeQ==", + "$Type": "ChangeDataCapture$AssociationProperty", + "Association": "CdcPublish.Order_Customer", + "ExposedName": "Order_Customer", + "VersionedPropertyPointer": "1tbI0VDneU2hMPfZru9wwA==" + } + ], + "VersionedSourcePointer": "kA3kcKWz0kys+quBWLm8iQ==" + }, + { + "$ID": "MGiJEokBmUyEuUuKEuJrgA==", + "$Type": "ChangeDataCapture$EntityChangeSource", + "Entity": "CdcPublish.Customer", + "ExposedName": "Customer", + "Properties": [ + 3, + { + "$ID": "/57KqEuxFkW40IDnSHtpig==", + "$Type": "ChangeDataCapture$AttributeProperty", + "Attribute": "CdcPublish.Customer.CustomerId", + "ExposedName": "CustomerId", + "VersionedPropertyPointer": "CtSi0JbPZ0yooiAFs66PJQ==" + }, + { + "$ID": "drJSpb2FEkaViQeoJspebQ==", + "$Type": "ChangeDataCapture$AttributeProperty", + "Attribute": "CdcPublish.Customer.CustomerName", + "ExposedName": "CustomerName", + "VersionedPropertyPointer": "e5z3yWUI2k+fnUgj+RMSig==" + }, + { + "$ID": "y/CR1GYUDkiQBUMGOp8P8A==", + "$Type": "ChangeDataCapture$AttributeProperty", + "Attribute": "CdcPublish.Customer.Address", + "ExposedName": "Address", + "VersionedPropertyPointer": "EMn6RtUoiUmdEvQFu1BERg==" + } + ], + "VersionedSourcePointer": "NvM40bLHVEGsjmCOHF2vMg==" + } + ] +} diff --git a/docs/11-proposals/data/kafka-events/cdc-OrdersCDCService.mxunit b/docs/11-proposals/data/kafka-events/cdc-OrdersCDCService.mxunit new file mode 100644 index 0000000000..0e82793f3f Binary files /dev/null and b/docs/11-proposals/data/kafka-events/cdc-OrdersCDCService.mxunit differ diff --git a/docs/11-proposals/data/kafka-events/domainmodel-BusinessEventsClient.json b/docs/11-proposals/data/kafka-events/domainmodel-BusinessEventsClient.json new file mode 100644 index 0000000000..521584d404 --- /dev/null +++ b/docs/11-proposals/data/kafka-events/domainmodel-BusinessEventsClient.json @@ -0,0 +1,83 @@ +{ + "$ID": "yePm6oWQ8k+X7FfC7Bqg9g==", + "$Type": "DomainModels$DomainModel", + "Annotations": [ + 3 + ], + "Associations": [ + 3 + ], + "CrossAssociations": [ + 3 + ], + "Documentation": "", + "Entities": [ + 3, + { + "$ID": "1DCqXigMB0C82GT3z28fFA==", + "$Type": "DomainModels$EntityImpl", + "AccessRules": [ + 3 + ], + "Attributes": [ + 3, + { + "$ID": "/caVOFHrxEeSuls8GPDdqw==", + "$Type": "DomainModels$Attribute", + "Documentation": "", + "ExportLevel": "Hidden", + "GUID": "9icTbmoXhUCu0Acr5jn98A==", + "Name": "OrderId", + "NewType": { + "$ID": "aKBMh9hCc0u+ZageWmanGg==", + "$Type": "DomainModels$LongAttributeType" + }, + "Value": { + "$ID": "YCfVGUiG7EOWZf1MDrpwfg==", + "$Type": "DomainModels$StoredValue", + "DefaultValue": "0" + } + }, + { + "$ID": "eAA1wZ1p4UWh37DwxDaDIg==", + "$Type": "DomainModels$Attribute", + "Documentation": "", + "ExportLevel": "Hidden", + "GUID": "sv4zKikar0ibfoC1wdTFMg==", + "Name": "CustomerId", + "NewType": { + "$ID": "U0FqRsgivUyLgMGLUBTjUA==", + "$Type": "DomainModels$LongAttributeType" + }, + "Value": { + "$ID": "e5o18n7sWkSX2IV5jnhCxw==", + "$Type": "DomainModels$StoredValue", + "DefaultValue": "0" + } + } + ], + "Documentation": "", + "Events": [ + 3 + ], + "ExportLevel": "Hidden", + "GUID": "Ea5JHF1qKEi/LWBh+AHZOw==", + "Image": "", + "ImageData": "", + "Indexes": [ + 3 + ], + "Location": "370;310", + "MaybeGeneralization": { + "$ID": "eaRTdhsMr0+d0/BOPv/Ocw==", + "$Type": "DomainModels$Generalization", + "Generalization": "BusinessEvents.ConsumedBusinessEvent" + }, + "Name": "CBE_OrderPlaceEvent", + "Source": null, + "ValidationRules": [ + 3 + ] + } + ] +} diff --git a/docs/11-proposals/data/kafka-events/domainmodel-BusinessEventsClient.mxunit b/docs/11-proposals/data/kafka-events/domainmodel-BusinessEventsClient.mxunit new file mode 100644 index 0000000000..4796e4755f Binary files /dev/null and b/docs/11-proposals/data/kafka-events/domainmodel-BusinessEventsClient.mxunit differ diff --git a/docs/11-proposals/data/kafka-events/domainmodel-CdcPublish.json b/docs/11-proposals/data/kafka-events/domainmodel-CdcPublish.json new file mode 100644 index 0000000000..0614fa93c5 --- /dev/null +++ b/docs/11-proposals/data/kafka-events/domainmodel-CdcPublish.json @@ -0,0 +1,392 @@ +{ + "$ID": "i+Ky3Xuoskion+JKyv6hVQ==", + "$Type": "DomainModels$DomainModel", + "Annotations": [ + 3 + ], + "Associations": [ + 3, + { + "$ID": "al4eVZ1mfk203kpBll+uGg==", + "$Type": "DomainModels$Association", + "ChildConnection": "0;67", + "ChildPointer": "xBQh3mBN7U+4g3DWQgS0Xw==", + "DeleteBehavior": { + "$ID": "2B6SdLP9fUKngSEWoMr6dA==", + "$Type": "DomainModels$DeleteBehavior", + "ChildDeleteBehavior": "DeleteMeButKeepReferences", + "ChildErrorMessage": null, + "ParentDeleteBehavior": "DeleteMeButKeepReferences", + "ParentErrorMessage": null + }, + "Documentation": "", + "ExportLevel": "Hidden", + "GUID": "rd9XQAxoy0S/0pdwa5GdPg==", + "Name": "Order_Customer", + "Owner": "Default", + "ParentConnection": "100;46", + "ParentPointer": "bUK3dfX1oU+NnNgl4+9u6w==", + "Source": null, + "StorageFormat": "Column", + "Type": "Reference" + } + ], + "CrossAssociations": [ + 3 + ], + "Documentation": "", + "Entities": [ + 3, + { + "$ID": "bUK3dfX1oU+NnNgl4+9u6w==", + "$Type": "DomainModels$EntityImpl", + "AccessRules": [ + 3, + { + "$ID": "2Z3TOxRbRU+lLhPQT1V5dw==", + "$Type": "DomainModels$AccessRule", + "MemberAccesses": [ + 3, + { + "$ID": "A+c/J3MnTE631TlwmNvvDQ==", + "$Type": "DomainModels$MemberAccess", + "Attribute": "CdcPublish.Order.OrderId", + "AccessRights": "ReadWrite", + "Association": "" + }, + { + "$ID": "4p3Mpyw9Ok+D0QZNJCUZDA==", + "$Type": "DomainModels$MemberAccess", + "Attribute": "CdcPublish.Order.CustomerName", + "AccessRights": "ReadWrite", + "Association": "" + }, + { + "$ID": "Kn0PC4+Fs0+uuiOZGC3rYQ==", + "$Type": "DomainModels$MemberAccess", + "Attribute": "CdcPublish.Order.ProductName", + "AccessRights": "ReadWrite", + "Association": "" + }, + { + "$ID": "fWMzYwa/l0yL8HNEvReuWA==", + "$Type": "DomainModels$MemberAccess", + "Attribute": "CdcPublish.Order.Price", + "AccessRights": "ReadWrite", + "Association": "" + }, + { + "$ID": "NUocLTKSvUOCix3GDWypdA==", + "$Type": "DomainModels$MemberAccess", + "Association": "CdcPublish.Order_Customer", + "AccessRights": "ReadWrite", + "Attribute": "" + } + ], + "AllowedModuleRoles": [ + 1, + "CdcPublish.User" + ], + "AllowCreate": true, + "AllowDelete": true, + "DefaultMemberAccessRights": "ReadWrite", + "XPathConstraint": "", + "Documentation": "", + "XPathConstraintCaption": "" + } + ], + "Attributes": [ + 3, + { + "$ID": "jg3jNTOroEipynEftpsGJg==", + "$Type": "DomainModels$Attribute", + "Documentation": "", + "ExportLevel": "Hidden", + "GUID": "n6q889hO3kabfdrYFCW8Mw==", + "Name": "OrderId", + "NewType": { + "$ID": "k6UYzXiJ5E6Tm/xj3DSu1Q==", + "$Type": "DomainModels$LongAttributeType" + }, + "Value": { + "$ID": "FB2eXhhLD0G3+v/WG67gsg==", + "$Type": "DomainModels$StoredValue", + "DefaultValue": "0" + } + }, + { + "$ID": "dA/2SWYgQEiYhbfoWtYeFg==", + "$Type": "DomainModels$Attribute", + "Documentation": "", + "ExportLevel": "Hidden", + "GUID": "cVSRjNaSskCeuva7LtNWLA==", + "Name": "CustomerName", + "NewType": { + "$ID": "UPOm+OVTg0SSxpb0+4NLxw==", + "$Type": "DomainModels$StringAttributeType", + "Length": 200 + }, + "Value": { + "$ID": "ZMCoG4q2TE+8wZvn+QlMRA==", + "$Type": "DomainModels$StoredValue", + "DefaultValue": "" + } + }, + { + "$ID": "Jwj8d3gogU+sgUjePLU2hg==", + "$Type": "DomainModels$Attribute", + "Documentation": "", + "ExportLevel": "Hidden", + "GUID": "0R7AyIdW9kaxFiUajfBG2Q==", + "Name": "ProductName", + "NewType": { + "$ID": "GRCdS2DmEU6H0vhw0938hQ==", + "$Type": "DomainModels$StringAttributeType", + "Length": 200 + }, + "Value": { + "$ID": "JtS88AeWpECKht2qI8dvEA==", + "$Type": "DomainModels$StoredValue", + "DefaultValue": "" + } + }, + { + "$ID": "gCj3yjN4REWsnWfC1yOOZg==", + "$Type": "DomainModels$Attribute", + "Documentation": "", + "ExportLevel": "Hidden", + "GUID": "DVFayQ9N0EivYrn/9+xydw==", + "Name": "Price", + "NewType": { + "$ID": "D/PpPMiBJUa1hG/iLD/Tzw==", + "$Type": "DomainModels$DecimalAttributeType" + }, + "Value": { + "$ID": "N13jqUuuQ02NPkq2/osvgQ==", + "$Type": "DomainModels$StoredValue", + "DefaultValue": "0" + } + } + ], + "Documentation": "", + "Events": [ + 3 + ], + "ExportLevel": "Hidden", + "GUID": "4WR5BXj9dE69ZsdKBhMVEQ==", + "Image": "", + "ImageData": "", + "Indexes": [ + 3 + ], + "Location": "70;100", + "MaybeGeneralization": { + "$ID": "E4gDKiAyN0Kezl6o3LmZgA==", + "$Type": "DomainModels$NoGeneralization", + "HasChangedByAttr": false, + "HasChangedDateAttr": false, + "HasCreatedDateAttr": false, + "HasOwnerAttr": false, + "Persistable": true + }, + "Name": "Order", + "Source": null, + "ValidationRules": [ + 3 + ] + }, + { + "$ID": "fIE59NjtCk+mp5rzHF0/KQ==", + "$Type": "DomainModels$EntityImpl", + "AccessRules": [ + 3 + ], + "Attributes": [ + 3, + { + "$ID": "QdboHulo00mq/RhEl883JA==", + "$Type": "DomainModels$Attribute", + "Documentation": "", + "ExportLevel": "Hidden", + "GUID": "spvSSfXc4E2lkLznWZ8ZUw==", + "Name": "OrderId", + "NewType": { + "$ID": "USARa1ou006bAyRbmI5m0g==", + "$Type": "DomainModels$LongAttributeType" + }, + "Value": { + "$ID": "SBhQbiOZUEukLaJzmAdjZw==", + "$Type": "DomainModels$StoredValue", + "DefaultValue": "0" + } + }, + { + "$ID": "vy41CaBkQEixxp/3lIYkBA==", + "$Type": "DomainModels$Attribute", + "Documentation": "", + "ExportLevel": "Hidden", + "GUID": "q9I/mv2yykW2hjpodim2eQ==", + "Name": "CustomerId", + "NewType": { + "$ID": "vGaM8oFbVUC4QbF5t85zkg==", + "$Type": "DomainModels$LongAttributeType" + }, + "Value": { + "$ID": "G/oyCU4dEUyHDghDCOnY9A==", + "$Type": "DomainModels$StoredValue", + "DefaultValue": "0" + } + } + ], + "Documentation": "", + "Events": [ + 3 + ], + "ExportLevel": "Hidden", + "GUID": "lPd8azfldkCH2Fj+LcUfCA==", + "Image": "", + "ImageData": "", + "Indexes": [ + 3 + ], + "Location": "70;250", + "MaybeGeneralization": { + "$ID": "EvbasZctpEWWoxQhPF8MxA==", + "$Type": "DomainModels$Generalization", + "Generalization": "BusinessEvents.PublishedBusinessEvent" + }, + "Name": "PBE_OrderPlaceEvent", + "Source": null, + "ValidationRules": [ + 3 + ] + }, + { + "$ID": "xBQh3mBN7U+4g3DWQgS0Xw==", + "$Type": "DomainModels$EntityImpl", + "AccessRules": [ + 3, + { + "$ID": "Zdpv0KMIbEmYJa8JX84DJg==", + "$Type": "DomainModels$AccessRule", + "MemberAccesses": [ + 3, + { + "$ID": "Lt7MO3WeAk2wnTcc267fBw==", + "$Type": "DomainModels$MemberAccess", + "Attribute": "CdcPublish.Customer.CustomerId", + "AccessRights": "ReadWrite", + "Association": "" + }, + { + "$ID": "jDIvnwYk4UqkYMauhnWmdw==", + "$Type": "DomainModels$MemberAccess", + "Attribute": "CdcPublish.Customer.CustomerName", + "AccessRights": "ReadWrite", + "Association": "" + }, + { + "$ID": "4aRN/mOMA0SByDWJiy4OaQ==", + "$Type": "DomainModels$MemberAccess", + "Attribute": "CdcPublish.Customer.Address", + "AccessRights": "ReadWrite", + "Association": "" + } + ], + "AllowedModuleRoles": [ + 1, + "CdcPublish.User" + ], + "AllowCreate": true, + "AllowDelete": true, + "DefaultMemberAccessRights": "ReadWrite", + "XPathConstraint": "", + "Documentation": "", + "XPathConstraintCaption": "" + } + ], + "Attributes": [ + 3, + { + "$ID": "MUDwQzRw+0OI/UBSFlLekQ==", + "$Type": "DomainModels$Attribute", + "Documentation": "", + "ExportLevel": "Hidden", + "GUID": "EJMsBnbJHkiH+0X5aC13cA==", + "Name": "CustomerId", + "NewType": { + "$ID": "QynNS49BDUmM0EXVev25Pw==", + "$Type": "DomainModels$LongAttributeType" + }, + "Value": { + "$ID": "EJVDIX9xc0m51uaVsxcuew==", + "$Type": "DomainModels$StoredValue", + "DefaultValue": "0" + } + }, + { + "$ID": "FNx49ClFN0iv48dQ/Fz+nw==", + "$Type": "DomainModels$Attribute", + "Documentation": "", + "ExportLevel": "Hidden", + "GUID": "24ASpoOGMkKUdy+TZGTKaQ==", + "Name": "CustomerName", + "NewType": { + "$ID": "87LneYsOdk+w8yTaRSgvow==", + "$Type": "DomainModels$StringAttributeType", + "Length": 200 + }, + "Value": { + "$ID": "w2TLg9JJGk+eui/h9gFz3g==", + "$Type": "DomainModels$StoredValue", + "DefaultValue": "" + } + }, + { + "$ID": "6NvOjNK2t0Cahak4CG/GBA==", + "$Type": "DomainModels$Attribute", + "Documentation": "", + "ExportLevel": "Hidden", + "GUID": "kuE7E74iaEa9XB0ly38FvA==", + "Name": "Address", + "NewType": { + "$ID": "kzLf/J1JxEyLRqQBKWl0pQ==", + "$Type": "DomainModels$StringAttributeType", + "Length": 200 + }, + "Value": { + "$ID": "JGG5VqRFfkKPjygsnQppTQ==", + "$Type": "DomainModels$StoredValue", + "DefaultValue": "" + } + } + ], + "Documentation": "", + "Events": [ + 3 + ], + "ExportLevel": "Hidden", + "GUID": "cvHjuruupkmtkng/U89QFA==", + "Image": "", + "ImageData": "", + "Indexes": [ + 3 + ], + "Location": "380;100", + "MaybeGeneralization": { + "$ID": "uNMAHgPnVUShHNzWyrkD5w==", + "$Type": "DomainModels$NoGeneralization", + "HasChangedByAttr": false, + "HasChangedDateAttr": false, + "HasCreatedDateAttr": false, + "HasOwnerAttr": false, + "Persistable": true + }, + "Name": "Customer", + "Source": null, + "ValidationRules": [ + 3 + ] + } + ] +} diff --git a/docs/11-proposals/data/kafka-events/domainmodel-CdcPublish.mxunit b/docs/11-proposals/data/kafka-events/domainmodel-CdcPublish.mxunit new file mode 100644 index 0000000000..04fd628ca1 Binary files /dev/null and b/docs/11-proposals/data/kafka-events/domainmodel-CdcPublish.mxunit differ diff --git a/docs/11-proposals/data/kafka-events/testapp-describe.mdl b/docs/11-proposals/data/kafka-events/testapp-describe.mdl new file mode 100644 index 0000000000..718b1c497f --- /dev/null +++ b/docs/11-proposals/data/kafka-events/testapp-describe.mdl @@ -0,0 +1,53 @@ +-- Captured with mxcli describe from TestApp commit 619864b (Mendix 11.16.1, Studio Pro-authored). +-- NOT runnable as-is: the last statement shows the current describe defect for a consumer service. +mdl 1; +@Position(70, 100) +create or modify persistent entity CdcPublish.Order ( + OrderId: Long default 0, + CustomerName: String(200), + ProductName: String(200), + Price: Decimal default 0 +); + +grant create, delete, read *, write * on entity CdcPublish.Order to CdcPublish.User; +@Position(380, 100) +create or modify persistent entity CdcPublish.Customer ( + CustomerId: Long default 0, + CustomerName: String(200), + Address: String(200) +); + +grant create, delete, read *, write * on entity CdcPublish.Customer to CdcPublish.User; +@Position(70, 250) +create or modify non-persistent entity CdcPublish.PBE_OrderPlaceEvent extends BusinessEvents.PublishedBusinessEvent ( + OrderId: Long default 0, + CustomerId: Long default 0 +); +@Position(370, 310) +create or modify persistent entity BusinessEventsClient.CBE_OrderPlaceEvent extends BusinessEvents.ConsumedBusinessEvent ( + OrderId: Long default 0, + CustomerId: Long default 0 +); +@anchor(from: (100, 46), to: (0, 67)) +create or modify association CdcPublish.Order_Customer +from CdcPublish.Order to CdcPublish.Customer; +create or modify microflow BusinessEventsClient.HandleBE_OrderPlaceEvent ( + @position(100, 0) + $CBE_OrderPlaceEvent: BusinessEventsClient.CBE_OrderPlaceEvent +) +begin + @start(100, 200) + @position(700, 200) + return; +end; +create or modify business event service CdcPublish.OrdersEventService +( + ServiceName: 'OrdersEventService', + EventNamePrefix: '' +) +{ + message OrderPlaceEvent (OrderId: Long, CustomerId: Long) publish + entity CdcPublish.PBE_OrderPlaceEvent; +}; +-- describe of the consumer service, today (defect: contract and subscription are dropped): +create or modify business event service BusinessEventsClient.OrdersEventClient diff --git a/mdl-examples/bug-tests/association-owner-both-self.fail.mdl b/mdl-examples/bug-tests/association-owner-both-self.fail.mdl new file mode 100644 index 0000000000..94ae38d5f9 --- /dev/null +++ b/mdl-examples/bug-tests/association-owner-both-self.fail.mdl @@ -0,0 +1,15 @@ +mdl 1; +-- NEGATIVE TEST — `mxcli check` must REFUSE this file (MDL099), with no project. +-- +-- An association with `owner both` cannot connect an entity to itself or to one of +-- its generalizations. The build reports +-- +-- [CE0002] +-- +-- Measured on MxBuild 11.16.1, Reference and ReferenceSet alike: same entity, +-- child -> parent, parent -> child and a longer chain are all CE0002; an unrelated +-- pair and every other owner build (see association-owner-both-self.mdl). +create entity MyFirstModule.OwnerBothA (Name: string(10)); +create entity MyFirstModule.OwnerBothB extends MyFirstModule.OwnerBothA (X: string(10)); +create association MyFirstModule.OwnerBothA_OwnerBothA from MyFirstModule.OwnerBothA to MyFirstModule.OwnerBothA type reference owner both; +create association MyFirstModule.OwnerBothB_OwnerBothA from MyFirstModule.OwnerBothB to MyFirstModule.OwnerBothA type referenceset owner both; diff --git a/mdl-examples/bug-tests/association-owner-both-self.mdl b/mdl-examples/bug-tests/association-owner-both-self.mdl new file mode 100644 index 0000000000..aed420a7ab --- /dev/null +++ b/mdl-examples/bug-tests/association-owner-both-self.mdl @@ -0,0 +1,9 @@ +mdl 1; +-- Control for association-owner-both-self.fail.mdl: the same shapes with another owner, +-- and `owner both` between unrelated entities, all build with 0 errors on MxBuild 11.16.1. +create entity MyFirstModule.OwnerBothC (Name: string(10)); +create entity MyFirstModule.OwnerBothD extends MyFirstModule.OwnerBothC (X: string(10)); +create entity MyFirstModule.OwnerBothE (Y: string(10)); +create association MyFirstModule.OwnerBothC_OwnerBothC from MyFirstModule.OwnerBothC to MyFirstModule.OwnerBothC type reference; +create association MyFirstModule.OwnerBothD_OwnerBothC from MyFirstModule.OwnerBothD to MyFirstModule.OwnerBothC type referenceset; +create association MyFirstModule.OwnerBothC_OwnerBothE from MyFirstModule.OwnerBothC to MyFirstModule.OwnerBothE type reference owner both; diff --git a/mdl/catalog/catalog.go b/mdl/catalog/catalog.go index 9528df35c0..53bc1a06ab 100644 --- a/mdl/catalog/catalog.go +++ b/mdl/catalog/catalog.go @@ -666,7 +666,9 @@ func (c *Catalog) GetCacheInfo() (*CacheInfo, error) { info.BuildMode, _ = c.GetMeta(MetaBuildMode) if modTimeStr, _ := c.GetMeta(MetaMprModTime); modTimeStr != "" { - info.MprModTime, _ = time.Parse(time.RFC3339, modTimeStr) + // Parse accepts the fractional seconds RFC3339Nano writes, and a whole-second + // value written by an older mxcli. + info.MprModTime, _ = time.Parse(time.RFC3339Nano, modTimeStr) } if buildTimeStr, _ := c.GetMeta(MetaBuildTime); buildTimeStr != "" { info.BuildTime, _ = time.Parse(time.RFC3339, buildTimeStr) @@ -684,12 +686,26 @@ func (c *Catalog) GetCacheInfo() (*CacheInfo, error) { return info, nil } +// ModTimeUnchanged reports whether a project file still has the modification time +// a cache recorded when it was built. It compares the instant exactly, to the +// precision the file system reports. +// +// It used to compare whole seconds, so a write landing in the same second as the +// build left the cache looking fresh and the next read answered from the catalog +// as it was before the write (a test that reads, writes and reads again failed +// about one run in six). The stored value is now kept to the nanosecond; a cache +// written before that holds whole seconds and so never equals a fractional time, +// which costs one rebuild per project and no wrong answer. +func ModTimeUnchanged(cached, current time.Time) bool { + return cached.Equal(current) +} + // SetCacheInfo stores cache metadata after building. func (c *Catalog) SetCacheInfo(mprPath string, mprModTime time.Time, mendixVersion, buildMode string, buildDuration time.Duration) error { if err := c.SetMeta(MetaMprPath, mprPath); err != nil { return err } - if err := c.SetMeta(MetaMprModTime, mprModTime.Format(time.RFC3339)); err != nil { + if err := c.SetMeta(MetaMprModTime, mprModTime.UTC().Format(time.RFC3339Nano)); err != nil { return err } if err := c.SetMeta(MetaMendixVersion, mendixVersion); err != nil { diff --git a/mdl/catalog/catalog_test.go b/mdl/catalog/catalog_test.go index 2b8e3db24c..e044b05867 100644 --- a/mdl/catalog/catalog_test.go +++ b/mdl/catalog/catalog_test.go @@ -782,3 +782,56 @@ func TestPartialViewsExposeSnapshotColumns(t *testing.T) { t.Errorf("external_entities view unexpectedly exposes SourceBranch (drift from original schema)") } } + +// A write that lands in the same second as the build that made the cache must +// still make the cache stale: the modification time is compared as an instant, +// not as whole seconds. A write one second later was always seen; one inside the +// same second was answered from the stale catalog. +func TestModTimeUnchanged(t *testing.T) { + base := time.Date(2026, 11, 17, 19, 25, 30, 250_000_000, time.UTC) + cases := []struct { + name string + cached, current time.Time + want bool + }{ + {"same instant", base, base, true}, + {"same instant, other zone", base, base.In(time.FixedZone("x", 3600)), true}, + {"later in the same second", base, base.Add(300 * time.Millisecond), false}, + {"earlier in the same second", base, base.Add(-200 * time.Millisecond), false}, + {"one nanosecond later", base, base.Add(time.Nanosecond), false}, + {"a second later", base, base.Add(time.Second), false}, + // A cache written before sub-second times were kept holds whole seconds; it + // cannot tell whether the file changed inside that second, so it rebuilds once. + {"legacy whole-second cache against a fractional time", base.Truncate(time.Second), base, false}, + {"legacy whole-second cache against a whole-second file", base.Truncate(time.Second), base.Truncate(time.Second), true}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + if got := ModTimeUnchanged(c.cached, c.current); got != c.want { + t.Errorf("ModTimeUnchanged(%v, %v) = %v, want %v", c.cached, c.current, got, c.want) + } + }) + } +} + +// The recorded time survives the round trip through the cache file to the +// nanosecond; RFC3339 kept only whole seconds, which is what made the check above +// impossible. +func TestCacheInfoKeepsModTimeToTheNanosecond(t *testing.T) { + cat, err := New() + if err != nil { + t.Fatal(err) + } + defer cat.Close() + mt := time.Date(2026, 11, 17, 19, 25, 30, 123_456_789, time.UTC) + if err := cat.SetCacheInfo("/p/app.mpr", mt, "11.8.0", "fast", time.Second); err != nil { + t.Fatal(err) + } + info, err := cat.GetCacheInfo() + if err != nil { + t.Fatal(err) + } + if !ModTimeUnchanged(info.MprModTime, mt) { + t.Errorf("recorded %v, read back %v", mt, info.MprModTime) + } +} diff --git a/mdl/executor/catalog_freshness_subsecond_test.go b/mdl/executor/catalog_freshness_subsecond_test.go new file mode 100644 index 0000000000..301d3bf71d --- /dev/null +++ b/mdl/executor/catalog_freshness_subsecond_test.go @@ -0,0 +1,63 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "context" + "os" + "testing" + "time" +) + +// inSameSecond moves the project file's modification time by a fraction of a +// second without crossing a second boundary, as a write that follows a catalog +// build within the same second does. It fails the test if the second changed, so +// the test cannot pass for the wrong reason. +func inSameSecond(t *testing.T, mprPath string) { + t.Helper() + fi, err := os.Stat(mprPath) + if err != nil { + t.Fatal(err) + } + before := fi.ModTime() + moved := before.Add(300 * time.Millisecond) + if before.Nanosecond() > 500_000_000 { + moved = before.Add(-300 * time.Millisecond) + } + if err := os.Chtimes(mprPath, moved, moved); err != nil { + t.Fatal(err) + } + after, err := os.Stat(mprPath) + if err != nil { + t.Fatal(err) + } + if after.ModTime().Unix() != before.Unix() || after.ModTime().Equal(before) { + t.Fatalf("setup: mtime %v -> %v is not a change inside one second (file system precision?)", + before, after.ModTime()) + } +} + +// A project written in the same second as the catalog build left the cache +// looking fresh, so the next read answered from the catalog as it was before the +// write: `SnapshotDependents` after an `exec` returned no dependents about one +// run in six. The control is the unchanged file, which must stay fresh. +func TestCacheValid_WriteInTheSameSecondAsTheBuild(t *testing.T) { + exec, cachePath := sourceCachedFixture(t) + ctx := exec.newExecContext(context.Background()) + + if ok, why := isCacheValid(ctx, cachePath, "fast"); !ok { + t.Fatalf("control: an unchanged project must leave the cache valid, got %q", why) + } + if !catalogCurrent(ctx, "fast") { + t.Fatal("control: an unchanged project must leave the in-memory catalog current") + } + + inSameSecond(t, ctx.MprPath) + + if ok, _ := isCacheValid(ctx, cachePath, "fast"); ok { + t.Error("isCacheValid: a project modified inside the second of the build was still treated as unchanged") + } + if catalogCurrent(ctx, "fast") { + t.Error("catalogCurrent: a project modified inside the second of the build was still treated as unchanged") + } +} diff --git a/mdl/executor/cmd_catalog.go b/mdl/executor/cmd_catalog.go index b421b0d948..3f0d296a15 100644 --- a/mdl/executor/cmd_catalog.go +++ b/mdl/executor/cmd_catalog.go @@ -235,7 +235,7 @@ func catalogCurrent(ctx *ExecContext, mode string) bool { if info.MprPath != "" && !catalog.SameMprPath(info.MprPath, ctx.MprPath) { return false } - if mt := getMprModTime(ctx); !mt.IsZero() && !info.MprModTime.IsZero() && info.MprModTime.Unix() != mt.Unix() { + if mt := getMprModTime(ctx); !mt.IsZero() && !info.MprModTime.IsZero() && !catalog.ModTimeUnchanged(info.MprModTime, mt) { return false } return true @@ -465,9 +465,10 @@ func isCacheValid(ctx *ExecContext, cachePath string, requiredMode string) (bool return false, "MPR path changed" } - // Check MPR modification time (compare Unix seconds to handle timezone/precision issues) + // Check MPR modification time, as an instant (a zone-independent, exact + // comparison; whole seconds missed a write in the second of the build). currentModTime := getMprModTime(ctx) - if info.MprModTime.Unix() != currentModTime.Unix() { + if !catalog.ModTimeUnchanged(info.MprModTime, currentModTime) { return false, "project file modified" } diff --git a/mdl/executor/validate_association_owner_both.go b/mdl/executor/validate_association_owner_both.go new file mode 100644 index 0000000000..49fcf31177 --- /dev/null +++ b/mdl/executor/validate_association_owner_both.go @@ -0,0 +1,69 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// CE0002 — an association with owner Both between an entity and itself, or between +// an entity and one of its generalizations. Measured on MxBuild 11.16.1, for +// Reference and ReferenceSet alike: same entity, child -> parent, parent -> child +// and a longer chain are all rejected; an unrelated pair, and every other owner, build. +// +// Only generalizations the script itself declares are followed. An entity that +// extends another in the project is not seen here, so that case still reaches +// `mx check`; refusing less is the safe direction. +const associationOwnerBothRule = "MDL099" + +// ValidateAssociationOwnerBoth reports each `owner both` association whose two ends +// are one entity or sit on one generalization chain. +func ValidateAssociationOwnerBoth(prog *ast.Program) []linter.Violation { + extends := map[string]string{} + for _, s := range prog.Statements { + if e, ok := s.(*ast.CreateEntityStmt); ok && e.Generalization != nil { + extends[strings.ToLower(e.Name.String())] = strings.ToLower(e.Generalization.String()) + } + } + // chainOf is the entity followed by each of its generalizations; the visited + // set ends a cycle, which Mendix rejects separately. + chainOf := func(name string) map[string]bool { + seen := map[string]bool{} + for n := name; n != "" && !seen[n]; n = extends[n] { + seen[n] = true + } + return seen + } + var out []linter.Violation + for _, s := range prog.Statements { + a, ok := s.(*ast.CreateAssociationStmt) + if !ok || a.Owner != ast.OwnerBoth { + continue + } + end := func(q ast.QualifiedName) string { + if q.Module == "" { + q.Module = a.Name.Module + } + return strings.ToLower(q.String()) + } + from, to := end(a.Parent), end(a.Child) + if from != to && !chainOf(from)[to] && !chainOf(to)[from] { + continue + } + out = append(out, linter.Violation{ + RuleID: associationOwnerBothRule, + Severity: linter.SeverityError, + Message: fmt.Sprintf( + "association %s has owner both but %s and %s are the same entity or one extends the other — "+ + "Mendix rejects it (CE0002: an association owned by both sides cannot connect an entity to itself or to its generalization)", + a.Name.String(), a.Parent.String(), a.Child.String()), + Location: linter.Location{DocumentType: "association", DocumentName: a.Name.String()}, + Suggestion: "Use the default owner (`owner default`) or `owner parent`, or relate two unrelated entities.", + }) + } + return out +} diff --git a/mdl/executor/validate_association_owner_both_test.go b/mdl/executor/validate_association_owner_both_test.go new file mode 100644 index 0000000000..14eb33069b --- /dev/null +++ b/mdl/executor/validate_association_owner_both_test.go @@ -0,0 +1,67 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +func ownerBothProg(owner ast.OwnerType, from, to string, extends map[string]string) *ast.Program { + qn := func(s string) ast.QualifiedName { + i := strings.Index(s, ".") + return ast.QualifiedName{Module: s[:i], Name: s[i+1:]} + } + prog := &ast.Program{} + for _, n := range []string{"M.A", "M.B", "M.C"} { + e := &ast.CreateEntityStmt{Name: qn(n)} + if p, ok := extends[n]; ok { + g := qn(p) + e.Generalization = &g + } + prog.Statements = append(prog.Statements, e) + } + a := assocStmt("M.Link", from, to) + a.Owner = owner + prog.Statements = append(prog.Statements, a) + return prog +} + +// Measured on MxBuild 11.16.1: CE0002 for owner Both when both ends are the same +// entity or one specializes the other, for Reference and ReferenceSet alike; an +// unrelated pair and any other owner are clean (controls). +func TestValidateAssociationOwnerBoth(t *testing.T) { + ext := map[string]string{"M.B": "M.A"} + cases := []struct { + name string + owner ast.OwnerType + from string + to string + extends map[string]string + want bool + }{ + {"self", ast.OwnerBoth, "M.A", "M.A", nil, true}, + {"self, other owner (control)", ast.OwnerDefault, "M.A", "M.A", nil, false}, + {"self, owner parent (control)", ast.OwnerParent, "M.A", "M.A", nil, false}, + {"child to parent", ast.OwnerBoth, "M.B", "M.A", ext, true}, + {"parent to child", ast.OwnerBoth, "M.A", "M.B", ext, true}, + {"unrelated (control)", ast.OwnerBoth, "M.A", "M.C", ext, false}, + {"grandchild chain", ast.OwnerBoth, "M.C", "M.A", map[string]string{"M.B": "M.A", "M.C": "M.B"}, true}, + {"case-insensitive", ast.OwnerBoth, "M.a", "M.A", nil, true}, + {"related but not declared (control)", ast.OwnerBoth, "M.B", "M.A", nil, false}, + {"cyclic generalization terminates", ast.OwnerBoth, "M.C", "M.A", map[string]string{"M.A": "M.B", "M.B": "M.A"}, false}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + v := ValidateAssociationOwnerBoth(ownerBothProg(c.owner, c.from, c.to, c.extends)) + if got := len(v) > 0; got != c.want { + t.Fatalf("violations=%v, want refusal=%v", v, c.want) + } + if c.want && (v[0].RuleID != associationOwnerBothRule || !strings.Contains(v[0].Message, "CE0002")) { + t.Errorf("rule/message should name %s and CE0002: %+v", associationOwnerBothRule, v[0]) + } + }) + } +} diff --git a/mdl/executor/validate_program.go b/mdl/executor/validate_program.go index 378a5ad242..8e57904560 100644 --- a/mdl/executor/validate_program.go +++ b/mdl/executor/validate_program.go @@ -46,6 +46,8 @@ func ValidateProgram(prog *ast.Program, projectPath string) []linter.Violation { // declares no variable (MDL063, #953). The script answers for the actions // it creates; the project for the rest. voids := newVoidCodeActions(prog, project) + // owner Both between an entity and itself or its generalization (CE0002). + violations = append(violations, ValidateAssociationOwnerBoth(prog)...) for _, stmt := range prog.Statements { // Check enumeration values for reserved words if enumStmt, ok := stmt.(*ast.CreateEnumerationStmt); ok {