diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-03-brain-before-project-accidental.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-03-brain-before-project-accidental.json new file mode 100644 index 0000000000..d63bed5c32 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-03-brain-before-project-accidental.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-11-03", "symptom": "A brain could be started before its project only by accident: store commands took -p and used filepath.Dir of a path nothing opened; `brain init ` accepted the directory and silently ignored it; and `mxcli new` refused any directory that already held the brain", "cause": "No way to name a brain's directory (`brainProjectPath` required -p, and `init` declared no Args), and `cmd_new.go`'s non-empty check plus `moveProject`'s rename-then-os.CopyFS fallback assumed an empty destination", "file": "`cmd/mxcli/cmd_brain.go` (`resolveBrainLocation`, `brainLocatedCommands`), `cmd/mxcli/newproject_paths.go` (`checkNewProjectDestination`, `mergeProject`)", "insight": "**A feature that works only because nothing opens a path is one Stat away from gone — and the fallback that seems to support it is unsafe.** os.CopyFS merges into an existing directory (MkdirAll on dirs, O_EXCL on files), so moving a project onto a brain appeared to work; the first shared file stopped it half way, leaving App.mpr moved and the stage gutted. The merge test passes against that accident; only the conflict test (6 files where there were 4) proves the pre-check. Adoption is kept to exactly docs/brain (with mxcli's marker) and .mxcli/brain, so `new` never merges into someone's files. Test helpers that drive the shared rootCmd must reset -p/-C, or a -C from one run turns the next run's -p into 'both passed'", "refs": ["ako/mxcli#1949"]} diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-03-brain-plan-check-need-model.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-03-brain-plan-check-need-model.json new file mode 100644 index 0000000000..48465cc0e5 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-03-brain-plan-check-need-model.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-11-03", "symptom": "`brain plan` and `brain check` fail with `CONNECT LOCAL '…/App.mpr': failed to connect: mpr file not found` when the brain was started before the project, although init/capture/promote/brief all work against the same nonexistent -p path", "cause": "Both commands call `openBrainResolver(projectPath)` and `brainFatal` on any error; the store commands only ever use `filepath.Dir` of -p", "file": "`cmd/mxcli/cmd_brain.go` (`openBrainResolverOrNoModel`), `cmd/mxcli/brain/check.go` (`NoModel`, `Unverifiable`, `checkSlice`)", "insight": "**A null resolver needs its own state, and the existing state switch silently decides what it means.** Returning NotFound reports the whole store stale; adding a state without touching `checkSlice` is worse, because that branch counts every non-NotFound state as BUILT (so NotIndexable does not under-report) — a project that does not exist read 100% built. The test asserts Built 0, not just exit 0; stubbing the guard gives Built:1. Only os.IsNotExist selects NoModel: a corrupt .mpr still errors, or it would read as merely unbuilt. Verified end to end: same store, `-p Nothing.mpr` 0 of 1, `-p PedApp.mpr` 1 of 1", "refs": ["ako/mxcli#1948"]} diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-03-drop-statements-undocumented-in-syntax.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-03-drop-statements-undocumented-in-syntax.json new file mode 100644 index 0000000000..bd58aaa35e --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-03-drop-statements-undocumented-in-syntax.json @@ -0,0 +1 @@ +{"area":"cmd/mxcli/syntax","date":"2026-11-03","symptom":"`mxcli syntax drop` showed three unrelated topics (drop-if-exists, layout.drop, workflow.drop); DROP ASSOCIATION was only inside domain-model.association's text, and DROP ENTITY / MICROFLOW / NANOFLOW / SNIPPET were in no topic at all","cause":"each DROP was left to its document type's topic, with nothing holding the set to the grammar; the segment-match fallback answers a verb with whatever path happens to contain it","file":"cmd/mxcli/syntax/features_misc.go (drop topic), cmd/mxcli/syntax/drop_statements_drift_test.go","insight":"For 'statement X is undocumented' reports, diff the grammar rule against `mxcli syntax --json` before writing prose: the rule's alternatives are the complete list (one grep found four more undocumented drops than the report named). Then pin it with a drift test that reads the .g4 (widget_keywords_drift_test.go is the template), mapping shared *Kw sub-rules to their first (canonical) alternative so deprecated aliases are not demanded in the docs.","refs":["ako/mxcli#1911","mendixlabs/mxcli#1716"]} diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-03-expression-functions-missing-from-syntax.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-03-expression-functions-missing-from-syntax.json new file mode 100644 index 0000000000..49a850a7a6 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-03-expression-functions-missing-from-syntax.json @@ -0,0 +1 @@ +{"area":"cmd/mxcli/syntax","date":"2026-11-03","symptom":"`mxcli syntax expression` named no isNew, add*/subtract* or trimTo* function although `mxcli check` accepted and type-checked all of them","cause":"the only expression topic was hand-written around specific pitfalls (regex, *Between, braces); the authoritative list of built-ins lived only in mdl/exprcheck funcTable","file":"mdl/exprcheck/signatures.go, cmd/mxcli/syntax/features_expression_functions.go","insight":"When the checker already holds a measured table, render the docs from it (exprcheck.Functions()) instead of adding the reported names by hand: one change documented ~90 functions with the exact types E005 enforces. Leave out rows marked unverified — hour()/dayOfYear() etc. are CE0117 on a real build, and documenting them would send readers into a build failure.","refs":["ako/mxcli#1866","mendixlabs/mxcli#1708"]} diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-04-brain-edit-keeps-id.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-04-brain-edit-keeps-id.json new file mode 100644 index 0000000000..6dc5b6198c --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-04-brain-edit-keeps-id.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-11-04", "symptom": "No way to change a brain entry's wording: every correction was drop + capture, which minted a new id and orphaned the old handle; editing the id-stable way exposed that duplicate detection compared stored ids, so the edited wording re-captured was accepted twice and the original wording collided with the edited entry's handle ('already carries entry 66dcf4' quoting text that no longer says that)", "cause": "`Queue.Append` and `Store.Promote` deduplicated on `existing.ID == e.ID`, which is only a content check while an id still matches its text; resolve and rename already broke that, edit breaks it routinely", "file": "`cmd/mxcli/brain/edit.go` (`ContentKey`, `Store.Update`, `Queue.Update`, `AssignFreeID`), `cmd/mxcli/brain/staged.go`, `cmd/mxcli/brain/store.go`, `cmd/mxcli/cmd_brain.go` (`brainEditCmd`, `brainMoveCmd`)", "insight": "**Once an id is a handle, identity and content are two keys, and every comparison has to say which one it means.** Duplicate = same ContentKey; collision = same id with different content, solved by deriving a fresh id (content + counter) at capture rather than refusing a legitimate new fact. Routing is the other trap: a decision follows its anchors only if it was where its anchors put it, or an entry pinned to project.md is evicted by an edit. Stubbing AssignFreeID reproduces the exact predicted error through the CLI. Harness gotcha found on the way: cobra StringSlice flags APPEND across runs of the shared rootCmd, so a second `capture -a X` in one test captured [X, X] and silently became a different entry — reset every flag to DefValue, and Changed, between runs", "refs": ["ako/mxcli#1950"]} diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-04-brain-entry-provenance.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-04-brain-entry-provenance.json new file mode 100644 index 0000000000..cf20a5ed93 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-04-brain-entry-provenance.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-11-04", "symptom": "A brain entry could not record where it came from (which document, which lines), so a requirements UI could neither show the source line behind a requirement nor answer which parts of a document produced none", "cause": "`Entry` had no provenance field, and the shard's one metadata line is matched by an exact regex (`metaLine`), so extending that line would make every older build report such entries MALFORMED and fail `brain check`", "file": "`cmd/mxcli/brain/source.go` (`NormalizeSource`, `missingSources`), `cmd/mxcli/brain/shard.go` (`renderEntry`, `parseEntry`, `sourceLine`), `cmd/mxcli/cmd_brain.go` (`brainSource`)", "insight": "**When a format is shared across versions, place new data where the OLD reader already preserves it.** The Source line goes directly after the metadata line: an older parser takes it as the first body line and keeps it on rewrite (measured: main e4baec63 checks such a store OK and its drop rewrites the line back as `\\n\\nSource: …`), while lines between title and metadata are silently discarded by old rewrites. The new parser must therefore accept the line after blank lines too — and only as the FIRST non-blank line, or a body that mentions a source becomes provenance. Removing the blank-line skip makes the old-rewrite test read the source as body. Provenance is not content: excluded from ContentKey, stored project-relative, refused outside the project or when missing, and a later-missing file fails check like an anchor naming nothing", "refs": ["ako/mxcli#1951"]} diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-04-syntax-versions-from-registry.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-04-syntax-versions-from-registry.json new file mode 100644 index 0000000000..f0d8ae5242 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-04-syntax-versions-from-registry.json @@ -0,0 +1 @@ +{"area":"cmd/mxcli/syntax","date":"2026-11-04","symptom":"`show features` had no `mxcli syntax` topic; 8 of 115 topics carried a hand-written MinVersion that disagreed with sdk/versions (boundary event 10.6.0 vs 10.14.0, view entities 10.18.0 vs 10.21.0, notify 11.7.0 although the statement is 10.0), data-transformer had none; `list features for version 10.20.0` and `list features in workflows` were parse errors","cause":"two version tables maintained by hand with no join; NUMBER_LITERAL lexes one dot, and the area slot took IDENTIFIER though the area names are lexer keywords","file":"cmd/mxcli/syntax/registry.go (Features, Register refuses MinVersion), cmd/mxcli/syntax/versions.go, cmd/mxcli/syntax/features_versions_test.go (ledger), sdk/versions/registry.go (Lookup, Keys), mdl/grammar/domains/MDLCatalog.g4 (featureVersion, identifierOrKeyword)","insight":"Remove the second table instead of syncing it: topics name registry features ('area/name') and versions are filled from the registry at lookup, and Register panics on a hand-written MinVersion, so drift cannot reappear. Close the reverse direction with a shrinking ledger test (every registry key referenced by some topic, or listed with a reason). A topic spanning several features lists each with its own version; use the FIRST feature as the topic's MinVersion, not the lowest — 'lowest' made data-transformer read 11.4 (the transform action) though its document is 11.9. When documenting a statement, run every documented form: two of the four `list features` forms did not parse.","refs":["ako/mxcli#1367"]} diff --git a/.claude/skills/fix-issue/findings/mdl-backend/2026-11-03-mcp-localhost-dials-127-first.json b/.claude/skills/fix-issue/findings/mdl-backend/2026-11-03-mcp-localhost-dials-127-first.json new file mode 100644 index 0000000000..e49cf10723 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-backend/2026-11-03-mcp-localhost-dials-127-first.json @@ -0,0 +1 @@ +{"area": "mdl/backend", "date": "2026-11-03", "symptom": "`mxcli mcp capabilities -p App.mpr --mcp http://localhost:7782/mcp` on Windows failed with `dial tcp :7782: i/o timeout`, still after the host.docker.internal fix; `--mcp-dial 127.0.0.1:7782` connected. The same report said `exec --mcp` / `-c --mcp` wrote the .mpr and mprcontents/ and skipped the .mpr.lock refusal (mendixlabs/mxcli#1684)", "cause": "dialPlan dialled the URL host verbatim, so `localhost` went through the resolver, which on the reporter's machine answered with a LAN address first; Studio Pro on Windows listens on 127.0.0.1 only and answers 400 to any Host header but localhost. The write-to-disk claim does not reproduce: the CLI with --mcp sends ped_create_document and leaves .mpr + mprcontents/ byte-identical with the lock present, and an un-routed statement is refused by the MCP backend's unsupportedBackend; the trace that showed no PED call was the tracer wiring fixed earlier (2026-11-01-mcp-trace-lists-no-ped-call)", "file": "mdl/backend/mcp/client.go (dialPlan)", "fix": "A localhost endpoint dials 127.0.0.1: first, then localhost: via the resolver (a ::1-only server), then host.docker.internal; the request URL, and so the Host header, stays localhost", "insight": "Before fixing 'writes the file instead of the server', run the real binary with the reporter's exact flags against a logging fake server and a hash of the model files, with the lock file present: one run separated what the server received from what the trace printed. Pin it with a child-process test (runMainStreams) and prove the test by forcing the fault back in (factory ignoring --mcp plus the guard off), which reproduces the report verbatim: exit 0, `Created constant`, no PED call, files changed. Do not 'restore' the lock refusal for --mcp: it guards the local Writer, which the MCP backend never opens (ConnectReadOnly); with --mcp the lock is always present, so refusing on it would make --mcp unusable", "refs": "ako/mxcli#1833, mendixlabs/mxcli#1684", "test": "mdl/backend/mcp/client_dial_test.go; cmd/mxcli/mcp_exec_no_file_write_test.go"} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-accessibilityhelper-item-attribute-read-as-binding.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-accessibilityhelper-item-attribute-read-as-binding.json new file mode 100644 index 0000000000..82be836e40 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-accessibilityhelper-item-attribute-read-as-binding.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-11-03","symptom":"ako/mxcli#904: describe → exec of an AccessibilityHelper failed. Half 1 (describe emitted `content { }`, MDL-WIDGET26 \"`content` is not a container of `accessibilityhelper` — it declares: attr, template\") no longer reproduces on main — fixed by mendixlabs/mxcli#1400. Half 2 still did: an attributesList item `attr (attribute: 'aria-label', …)` made check --references, which exec runs first, report \"widget '': Attribute 'aria-label' is bound but there is no enclosing data container providing entity context\"","cause":"walkWidgetsWithContext asked GetAttribute() of every AST node, including object-list ITEMS, whose properties are the item's own; AccessibilityHelper's item has a string property literally named `attribute` (the HTML attribute's name)","file":"mdl/executor/validate_page_context.go (walkWidgetsWithContext)","fix":"skip the binding check on a child whose type is a container keyword of the enclosing widget's definition (unstoredContainerKinds, already used by the CE0495 rule for the same item-vs-widget question)","test":"TestAccessibilityHelper_ItemAttributeIsNotABinding (+ RealUnboundAttributeStillReported control, SlotKeywordIsTemplate regression for half 1)","insight":"**Reproduce each half of a multi-part report on main before fixing**: the headline error (MDL-WIDGET26) was already gone, and the half that remained is in a different layer — only check --references (project needed) refuses; plain check and exec --no-check both accept and round-trip it (mx 11.12.1: exec → describe → exec 'Unchanged page', mx check 0 errors). The item-vs-widget question had already been answered once in the same file for duplicate names; a second rule walking the same tree did not ask it. Housekeeping: knownUnsupportedContainers (ATTRIBUTE/ATTR/EVENT, #239) was stale since widgetTypeV3 gained `| keyword`; the test now asks the parser instead of keeping a list","refs":["ako/mxcli#904","mendixlabs/mxcli#1400","#239"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-association-hop-off-context-entity.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-association-hop-off-context-entity.json new file mode 100644 index 0000000000..6e28a030f4 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-association-hop-off-context-entity.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-11-03","symptom":"A page data source `$Order/M.Flag_Account/M.Flag` (or `database from $currentObject/…` with the same path) passes `check` and `exec` writes it, although M.Flag_Account joins Flag and Account and cannot be followed from an Order","cause":"resolveAssociationSourceHops checked an author-supplied destination against the association's ends, and that the association exists, but never that the association touches the entity the hop starts from. A derived destination was safe by accident: resolveAssociationDestination finds none when the start is off the association, so only the spelled-out `Assoc/Dest` form slipped through","file":"`mdl/executor/cmd_pages_builder_v3.go` (resolveAssociationHops, isEndOrSpecialization), `mdl/executor/validate_assoc_hop_start.go` (MDL-ASSOCDS03)","insight":"A destination check alone is not a hop check: an association has two ends, so naming either one passes it from anywhere. Check the START against the ends too, accepting a specialization (Administration.Account over System.User's associations). At check time report only what is provable — walk the start's generalization chain through the script's entities and the loaded models and stay silent at the first entity in neither (System entities are not loaded), the same rule as provablyNotSpecializationOf","refs":["#834"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-call-external-action-blank-argument-describes-as-nothing.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-call-external-action-blank-argument-describes-as-nothing.json new file mode 100644 index 0000000000..0412affec6 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-call-external-action-blank-argument-describes-as-nothing.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-11-03","symptom":"describe microflow printed an unbound call external action argument stored as Argument: \"\" (Studio Pro 11.16, an optional OData parameter left blank) as `additional = )` / `additional = ,`, which does not parse, so every mdl 1; create or modify of the microflow was refused with MDL-V1-REBUILD \"the description does not parse\".","cause":"formatAction's CallExternalAction branch printed every mapping through describeExpr, which renders \"\" as nothing at all. The call microflow / call nanoflow branches had already been fixed for the same stored state with describeCallArgument (`= nothing`), and the visitor refused `nothing` outside those two calls.","file":"mdl/executor/cmd_microflows_format_action.go (CallExternalAction branch -> describeCallArgument); mdl/visitor/visitor_argument_binding.go (isBlankableCallArgument now admits CallExternalActionStatementContext)","insight":"Before designing a spelling for a stored state, grep how the language already spells it on a sibling: `Param = nothing` existed for exactly Argument \"\" on flow calls, with the same three-way distinction (blank / the value `empty` / no mapping). The issue's suggested fix - omit the mapping and have the builder re-create \"\" for a nullable parameter from the contract - would also round-trip, but only with the OData contract at hand, changes what omitting a parameter means, and is a second spelling of a state that already had one. `empty` (11.14) and \"\" (11.16) are both real stored states; the control test keeps `= empty` byte-identical. Not measured on mxbuild here: whether 11.16 accepts a \"\" argument on a NON-nullable parameter (a flow call gives CE0127).","refs":["ako/mxcli#1915","ako/mxcli#1914","mendixlabs/mxcli#1073"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-database-over-association-pairs-only.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-database-over-association-pairs-only.json new file mode 100644 index 0000000000..a01f7cad8a --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-database-over-association-pairs-only.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-11-03","symptom":"`database from $currentObject/M.A1/M.A2` (a List View or pluggable database source over two associations) is refused with `entity \"M.A2\" in datasource … does not exist`, while the association source `$currentObject/M.A1/M.A2` builds; `database from $ctx/M.A1/M.E` with E an existing entity off A1 is written as stated","cause":"buildDatabaseOverAssociationV3 (ako/mxcli#721 L5) walked the path two segments at a time — association, destination — and only checked the destination exists. resolveAssociationSourceHops had been taught every multi-hop spelling and the end-of-association check (mendixlabs/mxcli#1467), but it was not shared, so the second builder kept the old reading","file":"`mdl/executor/cmd_pages_builder_v3.go` (resolveAssociationHops, buildDatabaseOverAssociationV3)","insight":"When an issue names a defect in \"the builder\", grep for every function that splits the same path: the one the triage comment cited had been fixed by another issue, and its twin, written later for a different datasource kind, still had the bug. Make both read the path through one function rather than porting the fix, so the two cannot drift again","refs":["#834","mendixlabs/mxcli#1467"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-delete-button-grid-selection-source.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-delete-button-grid-selection-source.json new file mode 100644 index 0000000000..e40763a387 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-delete-button-grid-selection-source.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-11-03","symptom":"ako/mxcli#1932 (mendixlabs/mxcli#1730): a Delete button in a Data grid 2 control bar — Studio Pro stores Forms$DeleteClientAction.SourceVariable = Forms$PageVariable{Widget: grid1}; MDL `action: delete` wrote no source and mx check reported CE7005 \"No value selection has been made\" while check --references passed; describe printed a stored source as plain `delete`, so describe → exec dropped it","cause":"Three layers each lacked the property: pages.DeleteClientAction had no field, clientActionToGen never set gen's SourceVariable, renderClientActionBaseMDL read only ClosePage. The grammar had no slot either (`delete $x`)","file":"mdl/executor/cmd_pages_delete_source.go (deleteSource, deleteWithoutObjectProblem, validateDeleteSource); mdl/backend/modelsdk/widget_write.go (DeleteClientAction case); mdl/executor/cmd_pages_describe_output.go; mdl/grammar/domains/MDLPage.g4 (DELETE VARIABLE?)","insight":"**Reuse the selection machinery, do not invent one**: `$grid1` already meant a grid's selection for flow arguments (#1560) — classifyFlowArgValue resolves it, pageVariableToGen writes it, flowArgVariableValue reads it back — so the fix is three call sites, not a new resolver. Measured on mx 11.12.1, both variants: fixed `delete $grid1` → 0 errors; the writer's SetSourceVariable disabled → CE7005 on both buttons; a bare delete at a page root (via ALTER INSERT, where the guard stands down) → CE7005, which is what justifies refusing it at the root too; `delete $Order` (page parameter) → 0 errors. **A second trap behind the first**: with the grid's itemSelection left at None, the selection source is CE7001 \"Widget should be placed inside Data grid 2 'grid1' to use it as a source widget\" — the message names placement, the cause is the grid having no selection; check does not report that yet. The property-set half: all 13 Studio Pro Delete actions in ako/TestApp store SourceVariable null and mxcli omitted the key — registered as a NullFields default, which the version stripper drops below 10.4. The VARIABLE alternative on `delete` collides with actionExprV3's first branch (`$handler` fragment param) — buildActionV3 must exclude delete before testing VARIABLE","refs":["ako/mxcli#1932","mendixlabs/mxcli#1730","ako/mxcli#1560"],"ce":["CE7005","CE7001"],"rules":["MDL-PAGEDEL01"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-describe-dollar-literal-ends-at-dollar-dollar.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-describe-dollar-literal-ends-at-dollar-dollar.json new file mode 100644 index 0000000000..cbe32b76a3 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-describe-dollar-literal-ends-at-dollar-dollar.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-11-03","symptom":"`describe json structure` output of a sample holding `$$` (Teamcenter UID \"SR::N::BOMLine..10.7$$Otklm9GclKA\") fails `mxcli check` with `token recognition error at: '$$`; the same for a Java/JavaScript action body or EXTRA CODE, an agent prompt or a JSLT step holding `$$`","cause":"Every describer wrapped long text as `$$` + text + `$$`, and the lexer's DOLLAR_STRING (`'$$' .*? '$$'`) has no escape, so it ends at the first `$$` in the text; a text ending in `$` breaks it too (the closer pairs with that `$`). The Java/JavaScript `as` and `extra code` positions accepted only DOLLAR_STRING, so there was no other spelling to fall back to","file":"`mdl/executor/identifier_quoting.go` (mdlDollarQuote, mdlSourceBlock), `mdl/grammar/domains/MDLMicroflow.g4` (sourceCodeText), `mdl/visitor/visitor_javaaction.go`","insight":"Route every `$$` emit through one helper that falls back to mdlQuote (the version-aware single-quoted literal: it already handles backslashes and line breaks under mdl 0 and mdl 1) instead of adding PostgreSQL `$tag$` quoting to the lexer: a STRING_LITERAL already spans lines, and every DOLLAR_STRING position but the Java/JS bodies already accepted one. Grep for the delimiter itself, not one spelling of it: the emitters wrote it as \"$$\", \"\\nas $$\\n\" and \" %s $$\\n\". A round-trip test must parse describe output with its header (describedSource): a headerless parse is mdl 0 and misreads a backslash the mdl 1 fallback writes literally","refs":["#1003"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-describe-warning-cites-fork-923.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-describe-warning-cites-fork-923.json new file mode 100644 index 0000000000..93a46d949d --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-describe-warning-cites-fork-923.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-11-03","symptom":"describe microflow's non-nesting decision warning ended '(mxcli #923, interleaved)'; in the ako/mxcli fork #923 is 'MCP backend: CREATE ENTITY stores every attribute as String(200)', unrelated. The unrepresented-merge warning, the NORMALIZED note and the `mxcli syntax` text carried the same bare citation.","cause":"User-facing strings cited an issue number without its owner; the code comments around them already said mendixlabs/mxcli#923, but the emitted text did not.","file":"mdl/executor/cmd_microflows_show.go, cmd_microflows_show_merge.go, cmd_microflows_normalize.go, cmd/mxcli/syntax/features_microflow.go","insight":"Any issue number printed to a user must be owner-qualified: the binary ships from two repos with independent numbering, so a bare #N points at the wrong issue for half the readers. grep '\"[^\"]*#[0-9]' over non-test Go finds the emitted ones; comments are fine as long as the nearest one names the owner. Historical bug reports and findings quoting the old output were left as quotes.","refs":["ako/mxcli#1005","mendixlabs/mxcli#923"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-dynamictext-content-hidden-by-shadowing-def.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-dynamictext-content-hidden-by-shadowing-def.json new file mode 100644 index 0000000000..3909462403 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-dynamictext-content-hidden-by-shadowing-def.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-11-03", "symptom": "`mxcli check -p` (0.28.0) warned MDL-WIDGET10 that `dynamictext x (Content: '…')` — top-level or in a Data grid 2 column — had its `content` hidden and ignored; the page rendered it and mx check accepted it; no warning without a project (ako/mxcli#1865, mendixlabs/mxcli#1707)", "cause": "the project's `.mxcli/widgets/` held a pluggable definition registered under the builder-owned keyword `dynamictext`, so check validated the Forms$DynamicText against that widget's editorConfig visibility rules; already closed on main by mendixlabs/mxcli#1463 (registry refuses builder-owned names, lookupWidgetDef never resolves one)", "file": "`mdl/executor/widget_keywords.go` (`builderOwnedWidgetKeywords`), `widget_registry.go` (`loadDefinitionsFromDir`), `validate_widgets.go` (`lookupWidgetDef`) — regression test only", "insight": "**A project-only false MDL-WIDGET10 on a NON-pluggable keyword means a definition is answering for a keyword the builder owns** — not a visibility-rule bug. Check what `.mxcli/widgets/*.def.json` registers under that mdlName before touching the rule evaluation. Reproduced by writing the shadowing def into a temp project; control (dropping `dynamictext` from the builder-owned set) brings back the warning at both locations.", "refs": ["ako/mxcli#1865", "mendixlabs/mxcli#1707", "mendixlabs/mxcli#1463"], "rules": ["MDL-WIDGET10"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-ended-branch-decision-label-adds-merge-per-round-trip.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-ended-branch-decision-label-adds-merge-per-round-trip.json new file mode 100644 index 0000000000..3ba2a4acac --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-ended-branch-decision-label-adds-merge-per-round-trip.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-11-03","symptom":"DESCRIBE -> drop + create of a decision one of whose branches ends the flow (case / split type with a returning arm, if-elsif with a later returning branch) added an ExclusiveMerge: describe printed `end case; join shared1; merge shared1;`, the rebuild had the decision's merge plus the label's wired merge -> merge at one point, and each further round trip added `join junctionN; merge junctionN;`. create or modify (splice) was unaffected.","cause":"labelCrossedMerges labels the Recombinable entry when it is not the split's post-dominator; with an arm that returns, the post-dominator is the exit, so the decision's OWN merge got a crossed label, and continueAfterSplitJoin printed `join L` after `end case`. The builder always draws the decision's merge when two arms fall through, then drew the join edge from it to the label's merge.","file":"mdl/executor/cmd_microflows_builder_merge.go (resolveJoins -> adoptDecisionMerge)","insight":"Fixed in the builder, not the describer: `end X; join L;` is printed exactly when the split's own join is the labelled merge, so the builder can contract it - when the join leaves an unlabelled merge with no other way out and is the label's only way in - and that holds whatever reason the labeller had. Suppressing the label in describe instead would have had to re-derive each labeller's reason (sharedArmEntries labels the split's join on purpose). assertDescriptionRebuildsGraph contracts merges by design, so it is blind to this class: count merges and require a fixed point. The loop-body shape from the report (`if … then break; elsif … else … end if;`) did not reproduce on HEAD in three variants - not covered by the test.","refs":["ako/mxcli#1929","mendixlabs/mxcli#1727"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-filter-attributes-hidden-under-auto.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-filter-attributes-hidden-under-auto.json new file mode 100644 index 0000000000..81d2c48794 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-filter-attributes-hidden-under-auto.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-11-03", "symptom": "`mxcli check -p … --references` warned MDL-WIDGET10 \"property `attributes` is hidden when `attrChoice` is \\\"auto\\\" — the value will be ignored\" on every `textfilter`/`numberfilter`/`datefilter` with `Attributes: [...]`, although exec stores the list and attrChoice \"linked\" (ako/mxcli#1684)", "cause": "text/number/date filter .def.json declared attrChoice as the constant \"auto\"; the builder's SetAttributeObjects silently flips it to \"linked\" when it fills the list, so widgetValueMap (which only reads the definition) evaluated the editorConfig rule against a configuration exec never writes", "file": "`sdk/widgets/definitions/{text,number,date}filter.def.json` (new `attributes` mode on `hasAttributes`: attrChoice=linked + attributes; `default` mode keeps auto)", "insight": "**A builder side effect the definition does not state is invisible to check.** Any Set* that writes a second property (here SetAttributeObjects → attrChoice) must be mirrored as a mode constant, or MDL-WIDGET10 and the hidden-property reset reason about a configuration nobody stores. #1519 fixed the drop-down filter this way and left its three siblings; grep the defs for a constant the builder also writes when fixing one. Only reproduces with a project (rules come from the .mpk; testdata/pedapp's lifts 0 rules), so test the value map plus a supplied rule, with a control rule that hides under \"linked\".", "refs": ["ako/mxcli#1684", "mendixlabs/mxcli#1519"], "rules": ["MDL-WIDGET10"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-message-definition-describe-after-rename-attribute.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-message-definition-describe-after-rename-attribute.json new file mode 100644 index 0000000000..49d4bc9bbc --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-message-definition-describe-after-rename-attribute.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-11-03","symptom":"after `alter entity … rename attribute Reference to CustomerReference`, `describe message definition collection` printed `Reference,`; check --references passed that output and exec refused it: has no attribute \"Reference\"","cause":"describe printed the element's stored OriginalName, which lags a rename (Studio Pro refreshes it; CE6524 until then), while create resolves the member by attribute name; check never resolved message members at all","file":"mdl/executor/cmd_messagedefinitions.go (memberAttributeName, findChildByName), mdl/executor/validate_message_definition_members.go","insight":"A stored element can carry two names for one thing — a reference (Attribute: Module.Entity.CustomerReference) and its own label (OriginalName: Reference) — and only the reference follows a rename. Describe must print what create resolves against (the reference's last segment) and put the label in `as '…'`; that round-trips AND keeps the exposed contract name. Apply the same to member lookup by name (alter … drop member). For check/exec agreement, reuse newColumnAttrJudge: it answers only 'provably not an attribute', so a script that adds the attribute before the definition still passes.","refs":["ako/mxcli#1935","mendixlabs/mxcli#1733"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-note-linked-to-start-or-note-missing-from-describe.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-note-linked-to-start-or-note-missing-from-describe.json new file mode 100644 index 0000000000..0f86178438 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-note-linked-to-start-or-note-missing-from-describe.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-11-03","symptom":"describe microflow printed only one of two notes: Note2, wired to two activities, came out as @annotation(id: n1, …); Note1 was missing entirely, so describe -> exec deleted it. Reported as 'annotations not linked to an activity are missing'.","cause":"collectFreeAnnotations called a note free only when NO AnnotationFlow originated from it, and buildAnnotationsByTarget files a note under its flow's destination, which the traversal prints only for statements. A note whose only link ends at the StartEvent or at another Annotation was therefore neither free nor printed.","file":"mdl/executor/cmd_microflows_show_helpers.go (collectFreeAnnotations, unstatableNoteEnd, prependFreeAnnotationLines)","insight":"The literal reading of the report - a note with no link at all - already round-tripped; reproducing it as stated passes on HEAD. Enumerate the link ends the describer has no statement for (start event, another note, a link drawn element -> note) and build each: two of them vanish. The fix leans on an existing coupling worth knowing: useDerivedFlowLayout keeps the first statement's @position whenever collectFreeAnnotations is non-empty, because the visitor reads `@annotation(text:…)` as free only when activity metadata follows it - so widening collectFreeAnnotations also keeps the canonical describe unambiguous. The link itself has no MDL form; the describer says so in a -- WARNING line rather than dropping it silently. A start-event note link would need new syntax.","refs":["ako/mxcli#1933","mendixlabs/mxcli#1731"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-queued-call-retry-policy.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-queued-call-retry-policy.json new file mode 100644 index 0000000000..b1103e8db7 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-queued-call-retry-policy.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-11-03","symptom":"`call microflow … in queue Q retry exponential (…)` was a syntax error, and a create or modify of a flow whose queued call had a Studio Pro retry was refused, so the flow was locked for MDL","cause":"Queues$QueueFixedRetry / Queues$QueueExponentialRetry had no grammar, AST, writer or describe; the reader kept Retry as opaque raw","file":"mdl/grammar/domains/MDLMicroflow.g4 (queueRetryClause), mdl/visitor/visitor_queue_retry.go, mdl/backend/modelsdk/microflow_write.go (queueRetryToGen), microflow_read_actions.go (queueRetryFromRaw), mdl/executor/validate_queued_calls.go (checkNoDroppedQueueRetries)","insight":"Two measurements decided it. (1) A control that does NOT discriminate: writing IntervalType 'Days' still gave mx check 0 errors, so a clean check proved nothing about whether mx reads Retry at all; Retries: 0 gave CE7044 'Retries should be at least 1.' on the call — the control that does, and a rule for the visitor. (2) Making a stored property authorable changes what the splice path sees: on main a script restating the call without the retry was 'Unchanged' (retry invisible, so equal), after the change it became an in-place edit that silently dropped the retry, because checkNoQueuedCalls only runs on the rebuild path. Any guard written for the rebuild path must also run before modifyFlowInPlace once its property enters the compared model. Build the 'main' control binary in a git worktree: git stash leaves new untracked files behind and the build of 'main' then fails or, worse, builds the branch.","refs":["ako/mxcli#1931","mendixlabs/mxcli#1729"],"ce":["CE7044"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-rebuild-deletes-unsupported-actions.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-rebuild-deletes-unsupported-actions.json new file mode 100644 index 0000000000..6379fef66f --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-rebuild-deletes-unsupported-actions.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-11-03", "symptom": "A flow holding an action mxcli has no MDL for (describe prints `-- Unsupported action: <$Type>` — Microflows$SendExternalObject before ako/mxcli#1939, GenerateDocumentAction in ako/mxcli#1942, GenerateJumpToOptions/ApplyJumpToOption in WorkflowCommons) lost that activity when rebuilt from MDL: a drop + create from its describe output (the route MDL-V1-REBUILD recommends) or the whole-flow rebuild `create or modify` falls back to under mdl 0. check passed, exec printed Created.", "cause": "No guard: the reader keeps such an action as UnsupportedAction/UnknownAction (type name only), describe prints a comment, and every rebuild path builds the flow from the statements alone. The existing guard-don't-drop checks (checkNoQueuedCalls, checkNoUnwritableRestBody, ADR-0005) covered only a rewrite of an existing unit, never the drop + create route.", "file": "mdl/executor/validate_flow_verdicts.go + flow_unsupported_guard.go (FlowUnsupportedRule: a create or modify predicted as an mdl 0 rebuild of a flow holding unsupported actions is refused in the verdict pass check -p and exec's preflight share); the drop + create and exec-time rebuild guards are validate_unsupported_actions.go (35398f63, landed in parallel)", "insight": "Landed in parallel with 35398f63 (the same guard for drop + create and the exec-time rebuild), so this change was reduced to its delta when merged: exec's rebuild guard fires only when exec reaches the statement, after every earlier statement was written — a half-applied script — so the verdict pass refuses the predicted mdl 0 rebuild up front, as it already does a drop + create. Lesson: check main for a parallel session's fix to the same issue before opening the PR, not after the merge conflicts. The drop + create route cannot be refused at the create: by then the drop has deleted the flow, so a refusal there turns 'activity lost' into 'flow lost'. It has to be refused before the script runs — the verdict pass that check -p and exec's preflight share — which is why the exec-time guard covers only the rewrite of an existing unit. Studio Pro-authored subjects exist in ako/TestApp WorkflowCommons; ACT_Workflow_JumpTo cannot even be built from its description (the unsupported action declares $WorkflowJumpToDetails, which later statements read), so its mdl 0 verdict is 'unpredictable' — the mdl 0 test needed ACT_WorkflowJumpToDetails_Apply and an edit the splice structurally cannot make (taking a branch's return away); a crowded insert is refused under mdl 1 but spliced under mdl 0, and dropping an else branch splices under both. The splice under mdl 1 keeps unsupported activities and is the control. Proven by stubbing the verdict (both refusal tests fail) and the exec guard (exec of the mdl 0 rebuild succeeds and loses the activity); the CLI exec of the drop + create says 'Refusing to execute … Nothing was written'.", "refs": ["ako/mxcli#1939", "ako/mxcli#1942", "mendixlabs/mxcli#1735", "mendixlabs/mxcli#1738", "docs/13-decisions ADR-0005"], "test": "mdl/executor/flow_unsupported_guard_test.go"} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-rebuild-drops-unexpressible-activity.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-rebuild-drops-unexpressible-activity.json new file mode 100644 index 0000000000..51f48e62bd --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-rebuild-drops-unexpressible-activity.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-11-03","symptom":"drop microflow + create microflow (the route MDL-V1-REBUILD recommends) of a flow holding an activity MDL has no statement for passed check and exec and wrote the flow without it; mendixlabs/mxcli#1735 lost a Send external object, and with it every save of the external entity","cause":"the reader turns such an activity into UnsupportedAction, describe prints it as a comment that parses to nothing, and the writer deliberately drops it; only the splice path (which does not rewrite it) kept it, and nothing compared a rebuilt body with the stored one","file":"mdl/executor/validate_unsupported_actions.go (pre-flight drop+create guard), mdl/executor/cmd_microflows_build.go (rebuild-path guard), mdl/executor/nanoflow_validation.go (CE0009)","insight":"Adding the missing statement (#1947) fixes one activity; the class stays open for every other UnsupportedAction, so guard the class: refuse at the pre-flight (check -p and exec run it before writing) because once a drop has run there is nothing left to keep, and refuse the rebuild path too. To exercise it end to end without a Studio Pro sample of some other unsupported type, build a variant with one reader case removed — the stored activity then reads as unsupported, which is exactly the reporter's state. Also measured on 11.12.6: Send/Delete external object in a nanoflow is CE0009, which check passed.","refs":["ako/mxcli#1939","mendixlabs/mxcli#1735"],"ce":["CE0009"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-retrieve-offset-without-limit-dropped.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-retrieve-offset-without-limit-dropped.json new file mode 100644 index 0000000000..688a97cf89 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-retrieve-offset-without-limit-dropped.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-11-03", "symptom": "ako/mxcli#1941 (mendixlabs/mxcli#1737): `retrieve $Companies from M.Company offset 5000;` (no limit) was written as the 'all' range — Microflows$ConstantRange, SingleObject false — so re-running the script silently made the retrieve return the first objects instead of skipping 5000; describe then printed no offset. `limit 10 offset 5000` and `limit 10` were unaffected.", "cause": "addRetrieveAction built a Custom range only when `s.Limit != \"\"`; the offset-only case fell through to no range (the writer's default ConstantRange), although the comment above it already said 'a limit or offset the Custom range'. The writer (rangeToGen) and describe both handled an empty limit correctly.", "file": "mdl/executor/cmd_microflows_builder_actions.go (addRetrieveAction: Custom range when a limit OR an offset is stated)", "insight": "A condition keyed on one of two optional fields that together decide a shape is the silent-property-drop pattern in one line: grep for the sibling field in the same branch. Settled by the stored BSON (ConstantRange vs CustomRange) rather than by describe alone, and by mx check 11.12.6 on a fresh project: a CustomRange with OffsetExpression and no LimitExpression is 0 errors, so it is a model Mendix accepts. Test fails with range = before the fix, for mdl 0 and mdl 1.", "refs": ["ako/mxcli#1941", "mendixlabs/mxcli#1737", "ako/mxcli#734", "ako/mxcli#881"], "test": "mdl/executor/cmd_microflows_retrieve_first_test.go (TestRetrieveRange_OffsetWithoutLimit)"} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-send-delete-external-object-no-mdl.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-send-delete-external-object-no-mdl.json new file mode 100644 index 0000000000..a658e28a66 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-send-delete-external-object-no-mdl.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-11-03", "symptom": "ako/mxcli#1939 (mendixlabs/mxcli#1735): Microflows$SendExternalObject and Microflows$DeleteExternalObject had no MDL. A flow that saves an external (OData) object could not be written (a plain commit on an external entity is a Studio Pro error), describe printed `-- Unsupported action: Microflows$SendExternalObject`, and the drop + create rebuild that MDL-V1-REBUILD recommends deleted the send silently — check passed, exec printed no warning.", "cause": "No statement, AST node, builder, writer, reader case or describe spelling existed; the reader returned an UnknownAction, which describe prints as a comment and a rebuild therefore omits.", "file": "mdl/grammar/domains/MDLMicroflow.g4 (externalObjectStatement: (SEND | DELETE) EXTERNAL OBJECT VARIABLE REFRESH? onErrorClause?), mdl/ast (ExternalObjectStmt), sdk/microflows (SendExternalObjectAction, DeleteExternalObjectAction), mdl/backend/modelsdk microflow_write.go / microflow_read_actions.go, mdl/executor builder + every statement switch rollback appears in (graph, validate, flows refs, control error handling, loop scope, rule validation, mermaid, elk, exprcheck StatementErrorHandling), validate_microflow_error_handling.go (MDL076 for continue), cmd/mxcli/syntax odata.write", "insight": "Rollback has exactly this shape (variable, refresh in client, error handling), so 'grep every switch rollback appears in' is the checklist — the builder test passed while the flow builder's dispatch still did not know the statement, because it called the builder function directly. Measured on 11.12.6 with one microflow per project copy (a shared project's CE list does not say which flow each error is in): Continue is CE6035 on both actions; the default (Rollback) and custom handlers with or without rollback are accepted; a delete on a set the $metadata does not mark deletable is CE9435. The written BSON drew no structural error from mx check, and drop + create from describe output keeps the send (descriptions equal, 0 errors). A rebuilt mxcli binary is needed before an end-to-end check run — a stale one passed the continue case. Not covered: the generic hazard that any other UnknownAction is still lost by a drop + create. Studio Pro reference afterwards: ako/TestApp 31a34ca, Odata.ODataSendDeleteMF (Studio Pro 11.16) — both activities store exactly ErrorHandlingType (\"Rollback\" with no handler, unlike a plain delete, which stores none), the variable and RefreshInClient; mxcli's writer produces that property set and those values (only the key order differs, which BSON does not care about), the reader describes the Studio Pro unit as `send external object $NewOrders;` / `delete external object $NewOrders;`, and drop + create from that description gives the same two activities. The write test fails with the 'omit when empty' default a plain delete uses — the guess this reference settled.", "refs": ["ako/mxcli#1939", "mendixlabs/mxcli#1735", "ako/mxcli#1914"], "test": "mdl/executor/cmd_microflows_external_object_test.go; mdl-examples/bug-tests/1939-send-delete-external-object.mdl; mdl/backend/modelsdk/microflow_external_object_test.go (Studio Pro reference BSON)", "ce": ["CE6035", "CE9435"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-tostring-decimal-overflows-short-string.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-tostring-decimal-overflows-short-string.json new file mode 100644 index 0000000000..f49e713252 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-03-tostring-decimal-overflows-short-string.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-11-03","symptom":"`change $i (Lbl = toString(minutesBetween($a, $b)))` with Lbl String(10) passed check and built; at runtime the Decimal rendered as ~43 characters and overflowed the attribute","cause":"MDL041 only judged Decimal into Integer/Long targets; nothing related a toString's argument kind to the target attribute's length, which the no-project validator cannot see","file":"mdl/executor/validate_tostring_short_attribute.go, mdl/exprcheck/unknown_funcs.go (ToStringOfDecimal)","insight":"A multi-item expression report goes stale item by item: re-run each repro on main before coding — four of the six items in #1397 had already been fixed by separate PRs. The remaining one needs BOTH halves established (argument provably Decimal via exprcheck inference, attribute provably bounded via script decl or stored StringAttributeType.Length>0; Length 0 is unlimited), so it lives in the --references pass and warns rather than refuses, since mxbuild accepts it. Also: the unquoted `[%CurrentDateTime%]` in a retrieve where is stored quoted, so the reported CE0161 no longer reproduces.","refs":["ako/mxcli#1397","mendixlabs/mxcli#1515"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-04-keep-9x-database-source-on-rewrite.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-04-keep-9x-database-source-on-rewrite.json new file mode 100644 index 0000000000..36be6b2002 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-04-keep-9x-database-source-on-rewrite.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-11-04","symptom":"on a 9.24 project, editing a data grid's sort rewrote its stored CustomWidgets$CustomWidgetDatabaseSource as a CustomWidgetXPathSource; an edit to a column only left it alone","cause":"describe prints both source types as `database from E where …`, and the pluggable builder writes a direct-entity database source as XPath on every version, so a rebuilt widget lost the stored type","file":"mdl/executor/pluggable_database_source.go (buildPluggableDataSourceV3 widgetName), mdl/executor/cmd_pages_pluggable_passthrough.go (storedUsesDatabaseSource)","insight":"Which edits rebuild what matters for reproduction: a column caption change left the grid's source untouched on main, so the bug only showed when the grid widget itself changed (its sort). Without a Studio Pro 9 sample, make the stored state with a variant build that forces the database-source path, then edit with base and fixed builds. mx check could not validate it on a fresh 9.24 app: every mxcli-written data grid 2 crashes mx 9.24 (NullReferenceException) and gallery does not build at all — the pluggable-on-9.x gap tracked in #1163; the written shape is the one #1399 measured from Studio Pro's FSM template.","refs":["ako/mxcli#1208"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-04-param-supplies-flow-arg-before-10-1.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-04-param-supplies-flow-arg-before-10-1.json new file mode 100644 index 0000000000..b61fb790b7 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-04-param-supplies-flow-arg-before-10-1.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-11-04","symptom":"check refused nanoflow data sources (and actions) whose argument came from the enclosing snippet's or page's parameter on a 9.24 project, which mx check 9.24 accepts at 0 errors","cause":"the CE1571 rule was measured on 11.13 only, where a page parameter does not supply the argument, and applied to every version","file":"mdl/executor/validate_datasource_args.go (paramsSupplyFlowArguments, validateFlowArgumentsScoped)","insight":"Bisect a behaviour change instead of assuming the major-version boundary: the issue assumed '9.x vs 10+', but 10.0.0 still accepts and 10.1.0 refuses. One script holding every shape (snippet param, page param, data source, button action) and a bisect.sh that downloads an mxbuild, makes a fresh app, writes the script, runs mx check and deletes the mxbuild gets each step to a few minutes and keeps disk bounded. The CDN listing for '10.1' prefix-matches 10.19.x; ask for three-part 10.1.0 to get 10.1's own builds.","refs":["ako/mxcli#1206"],"ce":["CE1571"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-04-system-module-assignee-by-version.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-04-system-module-assignee-by-version.json new file mode 100644 index 0000000000..3e39882be0 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-04-system-module-assignee-by-version.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-11-04","symptom":"on a Mendix 9.24 project MDL-WORKFLOW10 suggested `change $T (System.WorkflowUserTask_Assignees = …)`; check --references accepted it and the build failed CE1613, while the name 9.24 actually has (WorkflowUserTask_Assignee) was refused by exec as 'not a known association'","cause":"modelsdk/meta's System module described the 11.x platform for every project; check never resolved a one-dot `Module.Assoc` member at all (only that Module is a module), so check and exec disagreed","file":"modelsdk/meta/system_module.go (SystemAssocDef.Introduced/Removed, ModelerSystemAssociationsIn), mdl/backend/modelsdk/system_module_read.go (version-aware System model), mdl/executor/validate_task_claim.go, mdl/executor/validate_member_refs.go (moduleLacksAssociation)","insight":"The System module is not in the .mpr, so version facts about it must be measured from the platform itself: the modeler assembly embeds the System domain model as BSON, so grep it per version (mxcli setup mxbuild --version ; the CDN listing in the error gives build numbers) and read Name/Type/GUID around the hit. Assignee (9.24, Reference) and Assignees (10.0+, ReferenceSet) share GUID and ParentPointer — a rename, not a new member. Record only measured differences as Introduced/Removed and keep the version-blind list = current platform, so existing callers are unchanged. Disk: each extracted mxbuild is ~1-2 GB; clear old app copies first.","refs":["ako/mxcli#1810"],"ce":["CE1613"]} diff --git a/.claude/skills/fix-issue/findings/mdl-other/2026-11-03-mpr008-loop-overlap-blames-position.json b/.claude/skills/fix-issue/findings/mdl-other/2026-11-03-mpr008-loop-overlap-blames-position.json new file mode 100644 index 0000000000..af25e444fc --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-other/2026-11-03-mpr008-loop-overlap-blames-position.json @@ -0,0 +1 @@ +{"area": "mdl/linter", "date": "2026-11-03", "symptom": "MPR008 reported a loop overlapping the activity before/after it with \"Each MDL statement that creates a canvas activity needs its own @position annotation\", although every statement already had one (ako/mxcli#1839, mendixlabs/mxcli#1690)", "cause": "the rule worded every overlapping pair the same way; a LoopedActivity's box is sized by its body (stored Size, often 400px+ wide), so a loop at an ordinary 190px activity spacing reaches into its neighbour — a spacing problem, not a missing annotation", "file": "`mdl/linter/rules/mpr008_overlapping_activities.go` (`overlapViolationText`, `actInfo.loop`)", "insight": "A diagnostic that names one cause for a check with several sends users fixing the wrong thing. MPR008 already measured each box with its own stored size, so the loop case was detectable at no cost — the message just never asked which kind of box overlapped. Factoring the wording into a pure function made it unit-testable without an mpr.Reader (Check itself still needs one).", "refs": ["ako/mxcli#1839", "mendixlabs/mxcli#1690"], "rules": ["MPR008"]} diff --git a/.claude/skills/fix-issue/findings/mdl-visitor/2026-11-03-datasource-argument-error-at-keyword.json b/.claude/skills/fix-issue/findings/mdl-visitor/2026-11-03-datasource-argument-error-at-keyword.json new file mode 100644 index 0000000000..6417b4e0ba --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-visitor/2026-11-03-datasource-argument-error-at-keyword.json @@ -0,0 +1 @@ +{"area":"mdl/visitor","date":"2026-11-03","symptom":"`dataview (datasource: microflow M.DS_X(Param = ))` reported `line 3:19 no viable alternative at input 'datasource'` — the column of a correct keyword, 29 characters before the mistake","cause":"ANTLR predicts a widget property with full-LL lookahead, so a failure anywhere inside the property is reported at its first token; moving the report needs a grammar restructure or a custom error strategy","file":"mdl/visitor/datasource_args_hint.go (flowArgumentHint), mdl/visitor/visitor.go (errorSite.line)","insight":"When ANTLR mis-locates an error inside a long alternation, a source-line diagnosis in the error enhancer is far cheaper than an error strategy: scan the flow call's argument list on the offending line, name the first malformed argument with its own line:column, and stay silent when every argument looks well formed (the error is then elsewhere). Note the listener prefixes `line N:C` AFTER enhancement, so the line has to be passed in (errorSite.line), not parsed from msg. A trailing comma `M.F(P = $v,)` parses with no error to the same AST, so it was left alone.","refs":["ako/mxcli#533"]} diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-11-04-workflow-annotation-storage-name.json b/.claude/skills/fix-issue/findings/modelsdk/2026-11-04-workflow-annotation-storage-name.json new file mode 100644 index 0000000000..fd17768f30 --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-11-04-workflow-annotation-storage-name.json @@ -0,0 +1 @@ +{"area":"modelsdk/gen","date":"2026-11-04","symptom":"modelsdk/gen's initAnnotation named a workflow annotation Workflows$AnnotationActivity while the writer overrode it to Workflows$Annotation and the decoder registered both; anything built through the generated constructor would write the other name","cause":"the out-of-tree generator took the reflection data's storage alias, which no Mendix version stores; the generator cannot be re-run here, so gen is corrected by hand","file":"modelsdk/gen/workflows/types.go (initAnnotation override, registry), modelsdk/gen/workflows/annotation_storagename_test.go","insight":"Settle a storage-name dispute by measurement, not by the SDK or the codegen comment: write the same small workflow twice (a variant binary with the other name) into fresh 9.24.30 and 11.12.6 apps and run mx check — the wrong name is TypeCacheUnknownTypeException at load, the right one 0 errors. Do not generalise from one alias: CallMicroflowTask is ALSO a codegen alias and is correct before 11.9 (renamed after), so test each alias on the versions it spans. To triage the other 40 aliases cheaply, count alias vs SDK name in the Studio Pro-authored TestApp mxunits (alias present, SDK name absent = confirmed) and check which unconfirmed ones mxcli writes at all; the written ones (EntityEvent, StringExport, ImageViewer) are covered by the doctype mx-check suite.","refs":["ako/mxcli#1784"]} diff --git a/.claude/skills/fix-issue/findings/other/2026-11-03-check-syntax-skill-stale-forward-refs.json b/.claude/skills/fix-issue/findings/other/2026-11-03-check-syntax-skill-stale-forward-refs.json new file mode 100644 index 0000000000..a39c9deef7 --- /dev/null +++ b/.claude/skills/fix-issue/findings/other/2026-11-03-check-syntax-skill-stale-forward-refs.json @@ -0,0 +1 @@ +{"area":"skills","date":"2026-11-03","symptom":"check-syntax skill told agents every forward reference fails, a month after #1179 made plain exec resolve forward flow calls; the fix had been deferred because the skill was at the 700-line body limit","cause":"a behaviour change edited resolve-forward-references but not the second skill restating the same rule, and the line limit made the edit look expensive","file":".claude/skills/mendix/check-syntax/SKILL.md, reference/studio-pro-mcp-verification.md","insight":"Measure the skill limit with the test, not wc: TestLargeSkillsWereSplit excludes the generated skill-index block, so the 714-line file was 694 lines of body and the issue's 'over the limit' was already false; the correction itself pushed it to 703, so the split was still needed. Before writing a behaviour claim into a skill, run it — `check` flags a forward enumeration (MDL-ORDER01) but not a forward snippetcall, which the first draft over-claimed.","refs":["ako/mxcli#1211","mendixlabs/mxcli#1405"]} diff --git a/.claude/skills/mendix/bootstrap-app/SKILL.md b/.claude/skills/mendix/bootstrap-app/SKILL.md index 4e09151e68..35e812664f 100644 --- a/.claude/skills/mendix/bootstrap-app/SKILL.md +++ b/.claude/skills/mendix/bootstrap-app/SKILL.md @@ -14,8 +14,8 @@ description: "Provision a Mendix project in a repo that has none — interview, | 44-130 | Step 0 — interview, and WAIT for the answers | | 131-263 | Provision | | 264-333 | If this is a solution (several apps in one repo) | -| 334-391 | Then propose the model — do not build it yet | -| 392-455 | After bootstrap — the inner loop | +| 334-393 | Then propose the model — do not build it yet | +| 394-457 | After bootstrap — the inner loop | # Bootstrap a Mendix App in an Empty Repo @@ -363,7 +363,9 @@ flags it. The cost lands later, on someone else. `GROUP BY`, `SUM`/`COUNT` — returning rows a page binds to directly. The microflow version pulls every object into memory to produce one number, and it gets slower exactly as the app succeeds, which is the worst possible failure curve. Needs - **Mendix 10.18+** (`list features` confirms it). `create view entity Mod.Name (…) + **Mendix 10.21+** for mxcli to write one (view entities are 10.18, but the source + document mxcli writes is 10.21; `list features` and `mxcli syntax view-entity` + print the floor from the registry). `create view entity Mod.Name (…) as ( select … )` — see `mxcli syntax view-entity` for the shape, its `oql` and `association` subtopics for the rules that bite (every column needs an `AS` alias; `ORDER BY` needs a `LIMIT`; selecting an id under an alias makes an diff --git a/.claude/skills/mendix/check-syntax/SKILL.md b/.claude/skills/mendix/check-syntax/SKILL.md index a86679204f..bc51590115 100644 --- a/.claude/skills/mendix/check-syntax/SKILL.md +++ b/.claude/skills/mendix/check-syntax/SKILL.md @@ -17,9 +17,9 @@ description: "Validate MDL with `mxcli check` before presenting or executing it, | 468-514 | Validation Workflow | | 515-536 | Script Execution Behavior | | 537-597 | Script Organization | -| 598-650 | Troubleshooting Parse Errors | -| 651-708 | Studio Pro MCP — Verification Only | -| 709-714 | Related Skills | +| 598-659 | Troubleshooting Parse Errors | +| 660-667 | Studio Pro MCP — Verification Only | +| 668-673 | Related Skills | # MDL Syntax Validation Skill @@ -606,9 +606,18 @@ Error: snippet not found: MyModule.NavMenu Error: page not found: MyModule.Customer_NewEdit ``` -Script execution is sequential — each `CREATE` commits immediately. Forward references -fail because the target doesn't exist in the database at the moment the referencing -document is created. +Script execution is sequential, and most references are resolved when the referring +document is written — so a snippet, page or entity the script creates further down +does not exist yet, and the statement fails. `check` reports most of these before +anything is written (`MDL-ORDER01`; `MDL-PAGE01` for a widget's page) — not all +(a forward `snippetcall` still surfaces only at `exec`), so order the script anyway. + +**One exception: flow calls and navigation targets.** `call microflow` / `call +nanoflow` (and a menu or navigation profile's page, microflow or nanoflow) are stored +by name, so a plain `mxcli exec` — which rolls back on failure — resolves them against +every document the script declares; mutual recursion needs no stub. Only the modes +that cannot roll back (`--continue-on-error`, the REPL, `-c`) still refuse a forward +one (mendixlabs/mxcli#1405). **Fix options:** 1. **Reorder** — move the target document's `CREATE` earlier in the script (simplest fix) @@ -650,61 +659,11 @@ Extra tokens found: ## Studio Pro MCP — Verification Only -When Studio Pro's embedded MCP server is available **alongside** mxcli (i.e. you are -*not* using mxcli's own `--mcp` backend, but mxcli writes the `.mpr` directly while -Studio Pro is open), use Studio Pro MCP **only for reading and verification**. Never -author with `ped_create_document` / `ped_update_document` — mxcli owns the `.mpr`, and -mixing authoring tools corrupts intent and diverges UUIDs. - -### Role split - -| Task | Tool | -|------|------| -| Create/modify entities, microflows, pages, nanoflows, nav | mxcli MDL | -| Verify CE errors after exec | `ped_check_errors` | -| Inspect widget tree / microflow body detail | `ped_read_document` | -| Check if a document exists before creating | `ped_find_document` | -| Full model validation | `./mxcli docker check` | - -### Studio Pro reads its in-memory model, not the file — a flush IS needed - -This is the opposite of what you might expect. mxcli writes directly to the `.mpr` -SQLite file, but **Studio Pro serves `ped_*` reads from its in-memory model**, which -does not hot-reload when an external process changes the file. So after `mxcli exec`: - -- `ped_read_document` / `ped_check_errors` will show the **stale** pre-exec model until - Studio Pro re-scans — call `refresh_project` first (or reload the project in the UI). -- **Hazard:** if Studio Pro later saves on its own, it overwrites mxcli's disk write with - its in-memory copy, silently discarding your MDL changes. So a file-based write is - **refused** while Studio Pro's `.mpr.lock` is beside the `.mpr`; `exec --force` - (or `MXCLI_ALLOW_STUDIO_PRO_OPEN=1`) overrides it, e.g. for a lock left by a crash. - -**Safest practice:** don't keep the same project open-and-saving in Studio Pro while -mxcli writes it. Either close (or don't save in) Studio Pro during MDL authoring, or -`refresh_project` after every `mxcli exec` before verifying. If you need both writing -*and* a live Studio Pro, use mxcli's `--mcp` backend (which authors *through* Studio -Pro) instead of writing the file directly. - -### Step 6: Post-execution verification (add to the workflow above) - -After `./mxcli exec script.mdl -p app.mpr` succeeds: - -1. `refresh_project` (Studio Pro MCP) so the in-memory model reflects the new file. -2. `ped_check_errors` on each created/modified document for CE errors. The argument - depends on the Studio Pro release — read the tool's input schema: - - up to 11.14: `{"documents": [{"documentType": "Microflows$Microflow", "documentName": "Mod.Name"}]}` - - 11.15+: `{"filters": {"documentType": "Microflows$Microflow", "documentNamePrefix": "Mod.Name"}}`. - 11.15 **silently ignores** `documents` and checks the whole project. `documentNamePrefix` - is a prefix (`Mod.Order` also matches `Mod.OrderLine`), so read the `'Mod.Name' (Type):` - header above each problem. More than 100 problems come in pages: repeat the same - `filters` with `pagination: {checkId, offset, size}` using the `Check ID` from the first answer. - -> **Do not treat an empty `DESCRIBE` as proof of a dropped construct.** `DESCRIBE` -> renders from the MDL emitter, which does not yet render every activity/widget type -> (e.g. Java/JavaScript action calls, exclusive splits, some nanoflow buttons). The -> construct may be present in the model even when `DESCRIBE` omits it. To tell a real -> write-drop from an emitter gap, confirm with `ped_read_document` (the live model) or -> `./mxcli docker check` — only flag an engine bug once the live model is also missing it. +With Studio Pro's own MCP server open beside mxcli, use it only to **read and verify** +(`ped_check_errors`, `ped_read_document`), never to author — and `refresh_project` +after every `mxcli exec`, because Studio Pro serves its in-memory model, not the file. +Role split, the stale-read hazard and the post-exec check by Studio Pro release: +[reference/studio-pro-mcp-verification.md](reference/studio-pro-mcp-verification.md). ## Related Skills diff --git a/.claude/skills/mendix/check-syntax/reference/studio-pro-mcp-verification.md b/.claude/skills/mendix/check-syntax/reference/studio-pro-mcp-verification.md new file mode 100644 index 0000000000..d71fa08147 --- /dev/null +++ b/.claude/skills/mendix/check-syntax/reference/studio-pro-mcp-verification.md @@ -0,0 +1,57 @@ +# Studio Pro MCP — Verification Only + +When Studio Pro's embedded MCP server is available **alongside** mxcli (i.e. you are +*not* using mxcli's own `--mcp` backend, but mxcli writes the `.mpr` directly while +Studio Pro is open), use Studio Pro MCP **only for reading and verification**. Never +author with `ped_create_document` / `ped_update_document` — mxcli owns the `.mpr`, and +mixing authoring tools corrupts intent and diverges UUIDs. + +## Role split + +| Task | Tool | +|------|------| +| Create/modify entities, microflows, pages, nanoflows, nav | mxcli MDL | +| Verify CE errors after exec | `ped_check_errors` | +| Inspect widget tree / microflow body detail | `ped_read_document` | +| Check if a document exists before creating | `ped_find_document` | +| Full model validation | `./mxcli docker check` | + +## Studio Pro reads its in-memory model, not the file — a flush IS needed + +This is the opposite of what you might expect. mxcli writes directly to the `.mpr` +SQLite file, but **Studio Pro serves `ped_*` reads from its in-memory model**, which +does not hot-reload when an external process changes the file. So after `mxcli exec`: + +- `ped_read_document` / `ped_check_errors` will show the **stale** pre-exec model until + Studio Pro re-scans — call `refresh_project` first (or reload the project in the UI). +- **Hazard:** if Studio Pro later saves on its own, it overwrites mxcli's disk write with + its in-memory copy, silently discarding your MDL changes. So a file-based write is + **refused** while Studio Pro's `.mpr.lock` is beside the `.mpr`; `exec --force` + (or `MXCLI_ALLOW_STUDIO_PRO_OPEN=1`) overrides it, e.g. for a lock left by a crash. + +**Safest practice:** don't keep the same project open-and-saving in Studio Pro while +mxcli writes it. Either close (or don't save in) Studio Pro during MDL authoring, or +`refresh_project` after every `mxcli exec` before verifying. If you need both writing +*and* a live Studio Pro, use mxcli's `--mcp` backend (which authors *through* Studio +Pro) instead of writing the file directly. + +## Post-execution verification (step 6 of the validation workflow in SKILL.md) + +After `./mxcli exec script.mdl -p app.mpr` succeeds: + +1. `refresh_project` (Studio Pro MCP) so the in-memory model reflects the new file. +2. `ped_check_errors` on each created/modified document for CE errors. The argument + depends on the Studio Pro release — read the tool's input schema: + - up to 11.14: `{"documents": [{"documentType": "Microflows$Microflow", "documentName": "Mod.Name"}]}` + - 11.15+: `{"filters": {"documentType": "Microflows$Microflow", "documentNamePrefix": "Mod.Name"}}`. + 11.15 **silently ignores** `documents` and checks the whole project. `documentNamePrefix` + is a prefix (`Mod.Order` also matches `Mod.OrderLine`), so read the `'Mod.Name' (Type):` + header above each problem. More than 100 problems come in pages: repeat the same + `filters` with `pagination: {checkId, offset, size}` using the `Check ID` from the first answer. + +> **Do not treat an empty `DESCRIBE` as proof of a dropped construct.** `DESCRIBE` +> renders from the MDL emitter, which does not yet render every activity/widget type +> (e.g. Java/JavaScript action calls, exclusive splits, some nanoflow buttons). The +> construct may be present in the model even when `DESCRIBE` omits it. To tell a real +> write-drop from an emitter gap, confirm with `ped_read_document` (the live model) or +> `./mxcli docker check` — only flag an engine bug once the live model is also missing it. diff --git a/.claude/skills/mendix/create-page/reference/widgets.md b/.claude/skills/mendix/create-page/reference/widgets.md index 580c166a38..5be3ca6181 100644 --- a/.claude/skills/mendix/create-page/reference/widgets.md +++ b/.claude/skills/mendix/create-page/reference/widgets.md @@ -121,7 +121,8 @@ describe icon collection Atlas_Core.Atlas_Filled; -- every icon + its referenc - `action: save changes close page` - Save and close page - `action: cancel changes` - Cancel changes - `action: close page` - Close the page -- `action: delete` - Delete object +- `action: delete` - Delete the enclosing data container's object +- `action: delete $grid1` - Delete grid1's selection: the form for a button in a data grid's `controlbar`, which has no context object (bare `delete` there is CE7005; check reports MDL-PAGEDEL01). The grid needs `itemSelection: Single` or `Multi`, or mx check reports CE7001. `delete $Param` deletes an object parameter of the page or snippet - `action: call microflow Module.MicroflowName` - Call microflow - `action: call microflow Module.MicroflowName(Param = $value)` - Call microflow with parameters - `action: call nanoflow Module.NanoflowName` - Call nanoflow (client-side) @@ -422,6 +423,8 @@ column (caption: 'Actions') { > **A view entity's association is one-way for a page.** The association a view entity derives from its OQL (`select c.ID as Stats_Customer`) can be followed from the view entity only. A data source that follows it from the entity it points at, in any widget and at any hop (`database from $currentObject/M.Stats_Customer/M.Stats`, `$currentObject/M.Stats_Customer`), is CE8812 "Entity 'M.Customer' has no association 'M.Stats_Customer'". Query the view entity instead: `datasource: database M.Stats where [M.Stats_Customer = $currentObject]`. `check` reports it as MDL-ASSOCDS02. A microflow `retrieve … from $Cust/M.Stats_Customer` is accepted. +> **Each hop starts where the path is.** The first association must have the context entity (or a generalization of it) at one end, and each later one the entity the previous hop arrives at. `$Order/M.Flag_Account/M.Flag` from an Order names an association between Flag and Account, which cannot be followed from an Order; `check` reports it as MDL-ASSOCDS03 and `exec` refuses it. + **With WHERE and SORT BY (inline in DataSource):** ```sql datagrid dgActive ( diff --git a/.claude/skills/mendix/odata-data-sharing/SKILL.md b/.claude/skills/mendix/odata-data-sharing/SKILL.md index e4932f9825..605245b614 100644 --- a/.claude/skills/mendix/odata-data-sharing/SKILL.md +++ b/.claude/skills/mendix/odata-data-sharing/SKILL.md @@ -10,17 +10,18 @@ description: "Share data between Mendix apps over OData — published services, | lines | section | |---|---| -| 30-42 | Reference files | -| 43-52 | When to Use This Skill | -| 53-73 | MetadataUrl Formats | -| 74-125 | Refreshing the Cached Contract | -| 126-154 | ServiceUrl Must Be a Constant | -| 155-193 | Architecture Overview | -| 194-226 | API Versioning | -| 227-272 | Folder Organization | -| 273-284 | Module Organization Conventions | -| 285-306 | Checklist | -| 307-334 | Exploration Commands | +| 31-43 | Reference files | +| 44-53 | When to Use This Skill | +| 54-74 | MetadataUrl Formats | +| 75-126 | Refreshing the Cached Contract | +| 127-155 | ServiceUrl Must Be a Constant | +| 156-164 | Writing to an External Entity | +| 165-203 | Architecture Overview | +| 204-236 | API Versioning | +| 237-282 | Folder Organization | +| 283-294 | Module Organization Conventions | +| 295-316 | Checklist | +| 317-344 | Exploration Commands | # OData Data Sharing Between Mendix Apps @@ -152,6 +153,15 @@ CREATE CONSUMED ODATA SERVICE ProductClient.ProductDataApiClient ( This enforces Mendix best practice of externalizing configuration values for different environments. +## Writing to an External Entity + +A changed external object is saved with `send external object $Obj [refresh];` +and removed with `delete external object $Obj [refresh];` — Studio Pro's *Send +external object* / *Delete external object*. A plain `commit` or `delete` on an +external entity is a Studio Pro error. The entity set must allow it (Updatable / +Deletable in `$metadata`, else CE9435), and `on error continue` is refused +(CE6035); `mxcli syntax odata.write` has the rest. + ## Architecture Overview OData data sharing follows a **producer/consumer** pattern with three layers: diff --git a/.claude/skills/mendix/scheduled-events-and-queues/SKILL.md b/.claude/skills/mendix/scheduled-events-and-queues/SKILL.md index 94e0b95b0f..83a50eb55e 100644 --- a/.claude/skills/mendix/scheduled-events-and-queues/SKILL.md +++ b/.claude/skills/mendix/scheduled-events-and-queues/SKILL.md @@ -14,13 +14,13 @@ description: "Scheduled events (Mendix's cron) and task queues in MDL — the ei | 39-143 | Scheduled Events | | 144-168 | Task Queues | | 169-178 | Common Mistakes | -| 179-270 | Binding a Call to a Queue — `IN QUEUE` | -| 271-285 | Placing Them in Folders | -| 286-300 | Validation Checklist | -| 301-322 | Querying and Linting | -| 323-328 | Related | -| 329-332 | Scheduled events — Mendix's cron (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP). `Repeat:` names one of the eight `ScheduledEvents$*Schedule` variants and only that variant's fields are accepted; a field from another repeat is refused by `mxcli check` (MDL-SCHED01) and by exec, which call the same function. The document shape is pinned by re-serializing three whole Studio Pro-authored events (Workflow Commons 4.11.0, OIDC SSO 4.6.0, SAML 4.2.1) element by element — `modelsdk/gen` is **wrong** about two properties here | -| 333-335 | Task queues (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP TASK QUEUE). `Config.ParallelismExpression` is a **string** and the sibling int32 `Parallelism` is not written — matching all four Studio Pro queues in Business Events 3.12.1. Binding a *call* to a queue is not yet authorable, so `CREATE OR REPLACE|MODIFY MICROFLOW` is **refused** when the stored microflow has a queued call (guard-don't-drop, ADR-0005) | +| 179-282 | Binding a Call to a Queue — `IN QUEUE` | +| 283-297 | Placing Them in Folders | +| 298-312 | Validation Checklist | +| 313-334 | Querying and Linting | +| 335-340 | Related | +| 341-344 | Scheduled events — Mendix's cron (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP). `Repeat:` names one of the eight `ScheduledEvents$*Schedule` variants and only that variant's fields are accepted; a field from another repeat is refused by `mxcli check` (MDL-SCHED01) and by exec, which call the same function. The document shape is pinned by re-serializing three whole Studio Pro-authored events (Workflow Commons 4.11.0, OIDC SSO 4.6.0, SAML 4.2.1) element by element — `modelsdk/gen` is **wrong** about two properties here | +| 345-347 | Task queues (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP TASK QUEUE). `Config.ParallelismExpression` is a **string** and the sibling int32 `Parallelism` is not written — matching all four Studio Pro queues in Business Events 3.12.1. Binding a *call* to a queue is not yet authorable, so `CREATE OR REPLACE|MODIFY MICROFLOW` is **refused** when the stored microflow has a queued call (guard-don't-drop, ADR-0005) | # Scheduled Events and Task Queues @@ -249,15 +249,27 @@ refusal the binding was written back as null and the project then looked *healthier* than before — `mx check` stopped reporting CE1613, because the configuration the error was about had been deleted. -One thing is still refused: a **retry policy** on a queued call -(`Queues$QueueFixedRetry` / `Queues$QueueExponentialRetry`). MDL has no syntax -for it, so a rewrite cannot preserve it — change that microflow in Studio Pro. +A queued call's **retry policy** (`Queues$QueueFixedRetry` / +`Queues$QueueExponentialRetry`, Mendix 9.10+) follows the queue: + +```sql +call microflow Ops.POST_Message(Body = $Body) in queue Ops.SendQueue + retry exponential (Retries: 20, InitialInterval: 1, MaximumInterval: 60, IntervalType: Minutes); +call java action Ops.Refresh(Url = $Url) in queue Ops.SendQueue + retry fixed (Retries: 5, Interval: 10, IntervalType: Minutes); +``` + +Every property is required, `IntervalType` is `Seconds`, `Minutes` or `Hours`, +and `Retries` must be at least 1 (mxbuild: CE7044). `describe` prints a stored +retry, so a flow with one stays editable from MDL. A `create or modify` that +still calls into the queue but leaves the retry out is **refused**, not reset — +restate it (copy it from `describe`) or state the new one. ### When a Java Action Is the Better Tool -The activity property gives queueing and nothing else. The runtime API, reachable -from a Java action, gives queueing *and* retry, and can queue a Java action -directly with no wrapper microflow: +The activity property gives queueing and a retry policy. The runtime API, +reachable from a Java action, adds per-call control (a retry decided at run time) +and can queue a Java action directly with no wrapper microflow: ```java Core.userActionCall("Ops.RefreshData") diff --git a/.claude/skills/mendix/write-microflows/SKILL.md b/.claude/skills/mendix/write-microflows/SKILL.md index 97a212c52b..854488d0c0 100644 --- a/.claude/skills/mendix/write-microflows/SKILL.md +++ b/.claude/skills/mendix/write-microflows/SKILL.md @@ -72,7 +72,7 @@ Choose the mode by who owns the microflow ([choose-edit-mode](../choose-edit-mod patched in place (a move keeps the node's flows; an activity changed into one of the same kind — an expression, a `where`, a member value, an argument — keeps its `$ID` and its action's, reported `edited in place`), as are plain activities added, removed, reordered or changed in a loop body ([an insert needing room past a stated `@position`](reference/pitfalls.md#13-inserting-where-the-script-states-every-position)). A redrawn `@anchor`/`@curve`, a decision/loop/`return` added or changed inside a loop body, error handler or other `return` added/taken away rebuilds under mdl 0 (`MDL-V1-REBUILD`: IDs renumbered, merges and curves lost) and is refused under `mdl 1;`. **To put a - decision or loop in a loop body, `alter … replace body of loop $X in $Xs with begin … end;`** — the loop keeps its `$ID` ([pitfalls](reference/pitfalls.md#11-changing-something-inside-a-loop-body)); a decision `alter … insert` put in is taken out by a re-run, `drop` or `replace` ([pitfalls](reference/pitfalls.md#15-taking-out-a-decision-another-script-put-in)). **To rebuild deliberately, `drop` and `create` in ONE script** — the grants carry only within it ([pitfalls](reference/pitfalls.md#drop--create-is-still-a-new-document)). + decision or loop in a loop body, `alter … replace body of loop $X in $Xs with begin … end;`** — the loop keeps its `$ID` ([pitfalls](reference/pitfalls.md#11-changing-something-inside-a-loop-body)); a decision `alter … insert` put in is taken out by a re-run, `drop` or `replace` ([pitfalls](reference/pitfalls.md#15-taking-out-a-decision-another-script-put-in)). **To rebuild deliberately, `drop` and `create` in ONE script** — the grants carry only within it ([pitfalls](reference/pitfalls.md#drop--create-is-still-a-new-document)); a flow with an `-- Unsupported action` line cannot be rebuilt (the rebuild would delete it, so it is refused), so change it in place. ## When to Use a Microflow vs a Nanoflow diff --git a/CHANGELOG.md b/CHANGELOG.md index 13323aa8d4..7c9ca2456a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,10 +8,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added +- **`send external object` and `delete external object`** (ako/mxcli#1939, mendixlabs/mxcli#1735) — Studio Pro's *Send external object* and *Delete external object* activities, which save and remove an external (OData) object, now have MDL: `send external object $Obj [refresh] [on error …];` and `delete external object $Obj [refresh] [on error …];`. `describe` prints them, so a flow that holds one survives a drop + create from its description, which used to delete it silently. `check` refuses `on error continue` on either (CE6035, MDL076). See `mxcli syntax odata.write`. - **A module's version and module type from MDL** (ako/mxcli#1921) — the General tab of Studio Pro's *Module settings* dialog: `create or modify module Inventory (Version: '2.3.1', ModuleType: addon)` and `alter module Inventory set (Version: '2.4.0')`. `describe module` prints the settings that differ from a new module's (App, 1.0.0), and the location as a comment when it is *Marketplace modules*; `list modules` gains `Version` and `Type` columns. The version must be SemVer, the type `app` or `addon` (Mendix 9.12+), and a statement that restates the stored settings writes nothing. Measured on 11.16: the type is the settings' `ExportLevel` (`Source` / `Protected`), not `ProtectedModuleType`, which is `AddOn` on every module. ### Fixed +- **`describe microflow` no longer leaves out a note linked to the start event or to another note** (ako/mxcli#1933, mendixlabs/mxcli#1731) — such a note counted as linked, so it was not printed as a free note, and the element it is linked to prints no `@annotation`, so it was not printed at all; re-executing the description deleted it. It is now described as a free note, after a `-- WARNING` line saying the link itself has no MDL form and a rewrite drops it. +- **Rebuilding a described decision no longer adds a merge** (ako/mxcli#1929, mendixlabs/mxcli#1727) — a `case`, `split type` or `if … elsif` one of whose branches returns is described with its own merge as a label (`end case; join shared1; merge shared1;`). Executing that as a fresh create built the decision's merge and a second one for the label, wired merge to merge, and every further describe → drop + create added another (`join junctionN; merge junctionN;`). The label's merge is now the decision's merge when nothing else reaches it, so the round trip keeps one merge. +- **`describe microflow` prints a blank `call external action` argument as `= nothing`** (ako/mxcli#1915) — Studio Pro 11.16 stores an optional OData action parameter left blank as an empty argument, and `describe` printed it as `additional = )`, which does not parse, so every `mdl 1;` `create or modify` of such a microflow was refused (MDL-V1-REBUILD). `call external action` now takes `Param = nothing`, the blank-argument spelling `call microflow` and `call nanoflow` already use; `Param = empty` (what 11.14 stored) stays a different value, and each round-trips as itself. +- **The description of an AccessibilityHelper re-executes** (ako/mxcli#904) — an `attr (attribute: 'aria-label', …)` item was read by `check --references`, which `exec` runs first, as an attribute binding with no data container, so describe → exec refused every helper that sets an HTML attribute. An object-list item's properties are now the item's own. +- **A Delete button in a data grid's control bar deletes the grid's selection** (ako/mxcli#1932, mendixlabs/mxcli#1730) — `action: delete $grid1` stores the selection as Studio Pro does (`SourceVariable` = a page variable naming the grid), and `describe` prints a stored one back that way; it printed plain `delete`, so a describe → exec round trip dropped it. A bare `delete` with no object to delete — in a control bar, or outside every data widget — wrote a button mx check reports as CE7005; `check` and `exec` now refuse it (MDL-PAGEDEL01). `delete $Param` deletes a page or snippet parameter. +- **A script that would rebuild a flow holding unsupported activities is refused before it runs** — under `mdl 0`, a `create or modify` the splice cannot make is rebuilt as a whole, and a flow holding an activity MDL cannot express (`-- Unsupported action: …`) refuses that rebuild when `exec` reaches it, after the statements before it were written. `check -p` and `exec`'s preflight now refuse the script up front (`MDL-FLOW-UNSUPPORTED`), as they already do for a drop + create of such a flow. +- **`retrieve … offset n` without a limit keeps its offset** (ako/mxcli#1941, mendixlabs/mxcli#1737) — an offset with no `limit` was written as the "all" range, so the retrieve returned the first objects instead of skipping `n`, and `describe` printed no offset. It is now written as a Custom range with that offset and no limit, which `mx check` accepts. - **`marketplace update` no longer removes the module and then fails** (ako/mxcli#1934, mendixlabs/mxcli#1732) — every update failed with `open package target.mpk: no such file or directory` after it had already dropped the installed module, leaving the project half-updated: it looked for the downloaded package under a different name than it was saved under. It now uses the downloaded package, and checks the package and the new version's module before it changes anything, so an update that cannot complete stops with the project untouched and says so. - **`create or modify microflow` no longer inverts a decision whose condition changes** (ako/mxcli#1930, mendixlabs/mxcli#1728) — `describe` prints a decision whose true path goes straight on as `if not(X) then …` with its paths swapped. Fed back with only the condition changed, the in-place edit stored the printed `not(Y)` as the decision's expression while its paths stayed, so the decision did the opposite of what the script said, with `check`, `exec` and `diff` all clean. The edit now stores the condition the decision means. - **`test --attach` no longer breaks the hosting app's pages** (ako/mxcli#1917, mendixlabs/mxcli#1723) — after a green `mxcli test --attach` against `run --local --test-endpoint`, the app answered 404 on its page bundles and every page showed the generic error dialog until it was restarted. The attach's builds went to the app's own `mxbuild --serve`, where on Mendix 11.12.5 and later any build after the first leaves the client bundle without its pages. The attach now finishes such a build on a `mxbuild --serve` of its own, which it stops afterwards, and leaves the app's untouched; this costs a cold build each time it happens. @@ -60,6 +68,7 @@ Headline: **Catch before the build what fails at build time or in production, an - **`create or replace page` can clear a data grid column caption** (ako/mxcli#1877) — `Caption: ''` over a stored caption reported "Unchanged page" and kept it, because the translation carry added the stored text back to an empty header written with no translation. The header is now written as one empty translation, which the carry leaves alone; an empty translation where the stored text has none is dropped, so a Studio Pro empty caption still rewrites as unchanged. This also repairs pages that still carry a leaked column name (#535). - **A custom-content data grid column with an empty caption no longer shows its MDL name as the header** (ako/mxcli#535) — through the generic `pluggablewidget 'com.mendix.widget.web.datagrid.Datagrid'` form, `column colTriage (Caption: '') { … }` (or no caption) wrote `colTriage` as the header text, which end users saw; Mendix stores no name for a column. It now writes an empty header, as the keyword `datagrid` form already did; it builds clean (11.15.0) and Data grid 2 renders it blank. - **`--mcp http://localhost:/mcp` reaches Studio Pro on a Windows machine with Docker Desktop** (mendixlabs/mxcli#1684, ako/mxcli#1833) — without `--mcp-dial`, mxcli dialled `host.docker.internal` whenever that name resolved, taking it to mean "inside a container". Docker Desktop for Windows puts the name in the hosts file, pointing at the LAN address, and Studio Pro listens on loopback only, so the connection timed out (`dial tcp :7782: i/o timeout`) on the very machine Studio Pro runs on. mxcli now dials the URL's own host first and tries `host.docker.internal` only when that fails, so the same command still works from a devcontainer. `--mcp-dial` is unchanged: when given, it is the only address tried. +- **`--mcp http://localhost:/mcp` dials 127.0.0.1 first** (mendixlabs/mxcli#1684, ako/mxcli#1833) — a `localhost` URL went through the resolver, which can answer with a LAN address first, while Studio Pro on Windows listens on 127.0.0.1 only, so the connection timed out without `--mcp-dial 127.0.0.1:`. mxcli now tries 127.0.0.1, then whatever `localhost` resolves to, then `host.docker.internal`; the `Host: localhost` header Studio Pro requires is unchanged. - **`--mcp-trace` and `--mcp-verbose` list the PED calls** (mendixlabs/mxcli#1684, ako/mxcli#1833) — they printed each MDL statement but never a tool call, on `exec`, `-c`, the REPL and `mxcli mcp capabilities` alike. A run that sent every write to Studio Pro read as one that sent none, which is how a report concluded that `exec --mcp` wrote the `.mpr` instead. The backend the CLI builds for `--mcp` now gets the tracer too, so each statement is followed by its `→ ped_…` calls. - **A PED tool whose handler throws is reported as a failure** — Studio Pro answers `Error executing tool handler: …` with `isError:false` (measured on 11.16.0), which a caller trusting the flag read as success; the MCP client now classifies that text as an error. - **`describe` and the source catalog no longer decode every document of a type to find one** (ako/mxcli#1850) — describing a snippet decoded every snippet of the project, a nanoflow every microflow and nanoflow, a page every page, and so did resolving the snippets, pages, layouts and flows a description names. A source catalog build describes every document, so it paid that once per document: on FactoryManagement a first `refresh catalog full source` took 33 s and 145 s of CPU, and now takes 19 s and 38 s, with identical text for all 3,718 documents. `create nanoflow` likewise decodes only the nanoflows of its own name. diff --git a/cmd/mxcli/brain/check.go b/cmd/mxcli/brain/check.go index 6c78680d1d..168854fe0f 100644 --- a/cmd/mxcli/brain/check.go +++ b/cmd/mxcli/brain/check.go @@ -5,10 +5,11 @@ // Two independent questions, and conflating them is the mistake this file // exists to avoid: // -// - Does each anchor still resolve? Three states, and only one is a failure. +// - Does each anchor still resolve? Four states, and only one is a failure. // An anchor to a document type the catalog does not index resolves as // *missing* through the catalog alone, which is a false staleness signal — // so a second, type-agnostic lookup separates "gone" from "not indexed". +// With no model at all, nothing can be checked, which is its own state. // - Is the entry in the right shard? A separate axis, not a fourth state: an // anchor can resolve perfectly and the entry still sit in the wrong file. package brain @@ -31,6 +32,10 @@ const ( // view does not cover. Reported, never failed — treating it as missing // would make `check` demand edits to entries that are perfectly current. NotIndexable + // Unverifiable: there is no model to check against — the brain was started + // before the project (ako/mxcli#1948). Neither a failure nor "built": a + // requirement counts as planned, a decision is simply not checked. + Unverifiable ) // MarshalJSON writes the state's name. The zero value of an int is a state @@ -53,6 +58,8 @@ func (s *AnchorState) UnmarshalJSON(b []byte) error { *s = NotFound case "not indexable": *s = NotIndexable + case "unverifiable": + *s = Unverifiable default: return fmt.Errorf("unknown anchor state %q", name) } @@ -65,6 +72,8 @@ func (s AnchorState) String() string { return "resolved" case NotFound: return "not found" + case Unverifiable: + return "unverifiable" default: return "not indexable" } @@ -87,6 +96,17 @@ type Resolver interface { Resolve(Anchor) (Resolution, error) } +// NoModel is the Resolver for a brain whose project does not exist yet. It +// answers every anchor Unverifiable rather than NotFound: a null resolver that +// said NotFound would report the whole store as stale when the truth is that +// nothing could be checked. +type NoModel struct{} + +// Resolve reports that there is nothing to resolve against. +func (NoModel) Resolve(Anchor) (Resolution, error) { + return Resolution{State: Unverifiable}, nil +} + // AnchorFinding is one anchor's outcome. type AnchorFinding struct { Shard string `json:"shard"` @@ -143,14 +163,25 @@ type Report struct { Entries int `json:"entries"` Anchors int `json:"anchors"` ResolvedN int `json:"resolved"` - // Findings carries NotFound and NotIndexable anchors only; a resolved - // anchor is not a finding. + // Unverifiable counts anchors that could not be checked because there is + // no model. Counted rather than listed: with no model every anchor is in + // this state, and listing them all would bury the one line that matters. + Unverifiable int `json:"unverifiable"` + // ModelAbsent says the check ran with no project to resolve against, so a + // consumer does not have to infer it from a count. + ModelAbsent bool `json:"model_absent,omitempty"` + // Findings carries NotFound and NotIndexable anchors only; a resolved or + // unverifiable anchor is not a finding. Findings []AnchorFinding `json:"findings"` Misfiled []MisfiledFinding `json:"misfiled"` // Malformed names entry blocks whose metadata line could not be read. - Malformed []string `json:"malformed"` - Slices []SliceProgress `json:"slices"` - Open []OpenQuestion `json:"open"` + Malformed []string `json:"malformed"` + // MissingSources names entries whose source document is not there. It + // fails the check: provenance pointing at a file nobody has is a + // reference to nothing, the same defect as an anchor that names nothing. + MissingSources []SourceFinding `json:"missing_sources,omitempty"` + Slices []SliceProgress `json:"slices"` + Open []OpenQuestion `json:"open"` } // MarshalJSON adds a derived "failed" alongside the report's contents. A @@ -174,7 +205,7 @@ func (r Report) MarshalJSON() ([]byte, error) { // the wrong shard. A NotIndexable anchor is information, not a defect — see the // constant's comment. func (r Report) Failed() bool { - if len(r.Misfiled) > 0 || len(r.Malformed) > 0 { + if len(r.Misfiled) > 0 || len(r.Malformed) > 0 || len(r.MissingSources) > 0 { return true } for _, f := range r.Findings { @@ -188,7 +219,8 @@ func (r Report) Failed() bool { // Check validates the given shards. Passing a subset is how `--changed` avoids // paying for shards a diff did not touch. func Check(s *Store, r Resolver, shards []string) (Report, error) { - rep := Report{Shards: shards} + _, noModel := r.(NoModel) + rep := Report{Shards: shards, ModelAbsent: noModel} for _, shard := range shards { entries, malformed, err := s.LoadShard(shard) if err != nil { @@ -197,6 +229,7 @@ func Check(s *Store, r Resolver, shards []string) (Report, error) { for _, m := range malformed { rep.Malformed = append(rep.Malformed, shard+": "+m) } + rep.MissingSources = append(rep.MissingSources, s.missingSources(shard, entries)...) if IsPlanShard(shard) { progress, err := checkSlice(r, shard, entries) if err != nil { @@ -205,6 +238,7 @@ func Check(s *Store, r Resolver, shards []string) (Report, error) { rep.Entries += len(entries) rep.Anchors += progress.anchors rep.ResolvedN += progress.resolved + rep.Unverifiable += progress.unverifiable rep.Slices = append(rep.Slices, progress.SliceProgress) rep.Open = append(rep.Open, progress.open...) continue @@ -226,10 +260,14 @@ func Check(s *Store, r Resolver, shards []string) (Report, error) { if err != nil { return rep, err } - if res.State == Resolved { + switch res.State { + case Resolved: rep.ResolvedN++ resolvedModules = append(resolvedModules, res.Module) continue + case Unverifiable: + rep.Unverifiable++ + continue } rep.Findings = append(rep.Findings, AnchorFinding{ Shard: shard, EntryID: e.ID, Title: e.Title, @@ -249,8 +287,8 @@ func Check(s *Store, r Resolver, shards []string) (Report, error) { type sliceCounts struct { SliceProgress - anchors, resolved int - open []OpenQuestion + anchors, resolved, unverifiable int + open []OpenQuestion } // checkSlice counts a slice's requirements against the model. It records no @@ -285,8 +323,17 @@ func checkSlice(r Resolver, shard string, entries []Entry) (sliceCounts, error) // NotIndexable counts as built: the thing is there, the catalog // simply does not index its type. Calling it planned would report // finished work as outstanding. - if res.State == NotFound { + // + // Unverifiable does NOT: with no model nothing is built, and + // letting it fall through to the built branch with NotIndexable + // reports a project that does not exist yet as finished. + switch res.State { + case NotFound: + built = false + continue + case Unverifiable: built = false + out.unverifiable++ continue } out.resolved++ diff --git a/cmd/mxcli/brain/edit.go b/cmd/mxcli/brain/edit.go new file mode 100644 index 0000000000..ee6c7fcb06 --- /dev/null +++ b/cmd/mxcli/brain/edit.go @@ -0,0 +1,235 @@ +// SPDX-License-Identifier: Apache-2.0 + +// edit.go - changing an entry without changing which entry it is. +// +// An entry's id is a handle: `brain promote `, `brain resolve `, prose +// and other entries cite it. It is derived from the content when the entry is +// captured, but once it exists it is NOT re-derived — the rule rename.go and +// Resolve already follow (ako/mxcli#1950). Before this file there was no edit +// path at all, so every correction went drop + capture and orphaned the handle. +// +// The consequence the rest of the package has to absorb: a stored id need not +// match the text it sits beside. So duplicates are found by ContentKey — the +// hash of what the entry says now — never by comparing stored ids, which would +// miss the edited wording captured again and collide with the original wording. +package brain + +import ( + "crypto/sha256" + "encoding/hex" + "fmt" + "slices" + "strconv" +) + +// ContentKey is the hash of what the entry currently says: title, slice and +// anchors, exactly what computeID covers. For an entry never edited, resolved +// or re-anchored it equals the id; for one that was, it does not, and that +// difference is the point — the key identifies content, the id identifies the +// entry. +func (e Entry) ContentKey() string { return e.computeID() } + +// Edit returns the entry with new text and, when replaceAnchors is set, new +// anchors. Everything else — id, kind, slice, open, date — is kept. It is +// validated as capture validates, so an edit cannot produce an entry capture +// would have refused. +func (e Entry) Edit(text string, anchors []string, replaceAnchors bool) (Entry, error) { + title, body := splitTitle(text) + if title == "" { + return Entry{}, fmt.Errorf("an entry needs at least a title line") + } + out := e + out.Title, out.Body = title, body + if replaceAnchors { + out.Anchors = slices.Clone(anchors) + } + if _, err := ParseAnchors(out.Anchors); err != nil { + return Entry{}, err + } + if out.EntryKind() == KindRequirement && !out.Open { + if err := requirementAnchorsArePlannable(out.Anchors); err != nil { + return Entry{}, err + } + } + return out, nil +} + +// MoveToSlice returns a requirement (or a question filed against a slice) +// moved to another slice. A decision is refused: giving one a slice turns it +// into a requirement, whose anchors stop failing when they go stale — a change +// of meaning, not of place. +func (e Entry) MoveToSlice(slice string) (Entry, error) { + if e.EntryKind() != KindRequirement { + return Entry{}, fmt.Errorf("%s is a decision, which has no slice; a decision's file follows its anchors (re-anchor it with 'brain edit %s -a @Module.Element')", e.ID, e.ID) + } + if !sliceName.MatchString(slice) { + return Entry{}, fmt.Errorf("slice %q: use letters, digits, '-' and '_' (a leading number orders it: 01-accounts)", slice) + } + out := e + out.Slice = slice + return out, nil +} + +// TakenIDs is every id in use, committed or staged. +func TakenIDs(s *Store, q *Queue) (map[string]bool, error) { + taken := map[string]bool{} + staged, err := q.Load() + if err != nil { + return nil, err + } + for _, e := range staged { + taken[e.ID] = true + } + shards, err := s.ListShards() + if err != nil { + return nil, err + } + for _, sh := range shards { + entries, _, err := s.LoadShard(sh) + if err != nil { + return nil, err + } + for _, e := range entries { + taken[e.ID] = true + } + } + return taken, nil +} + +// AssignFreeID gives a newly captured entry an id no existing entry holds. +// +// Normally that is its content-derived id, unchanged. It differs only when the +// id is already a handle for an entry whose text has since changed — capture +// the ORIGINAL wording of an edited entry and it hashes straight to that +// entry's id. The caller has already established this is not a duplicate (the +// content keys differ), so it is a new fact, and it gets a new id derived from +// the same content plus a counter: deterministic, and still six hex digits. +func AssignFreeID(e Entry, taken map[string]bool) Entry { + for n := 1; taken[e.ID]; n++ { + h := sha256.Sum256([]byte(e.ContentKey() + "\x02" + strconv.Itoa(n))) + e.ID = hex.EncodeToString(h[:])[:6] + } + return e +} + +// Moved reports where an Update took an entry from and to; the same shard when +// it stayed put. +type Moved struct { + From string `json:"from"` + To string `json:"to"` +} + +// Update changes a committed entry in place, keeping its id. +// +// It stays at its position in its shard unless the change moves it: a +// requirement follows its slice, and a decision follows its anchors — but only +// if it was where its anchors put it. One placed elsewhere on purpose (promote +// --to project) stays where it was put. +// +// A change that would make the entry say the same as another committed entry +// is refused, naming that entry: two copies of one fact is what the store +// exists to prevent. +func (s *Store) Update(id string, change func(Entry) (Entry, error)) (Moved, error) { + old, from, err := s.Find(id) + if err != nil { + return Moved{}, err + } + if from == "" { + return Moved{}, fmt.Errorf("no committed entry with id %s", id) + } + next, err := change(old) + if err != nil { + return Moved{}, err + } + next.ID = old.ID // a handle: whatever the change did, this does not move + + if dup, err := s.findContentExcept(next.ContentKey(), id); err != nil { + return Moved{}, err + } else if dup.ID != "" { + return Moved{}, fmt.Errorf("that would make %s say the same as %s (%q)", id, dup.ID, dup.Title) + } + + to := from + switch { + case next.EntryKind() == KindRequirement: + to = next.Shard() + case from == old.Shard(): + to = next.Shard() + } + if to == from { + return Moved{From: from, To: to}, s.Replace(from, next) + } + // Into the new shard first, then out of the old: a failure in between + // leaves the entry in both files, visible to `check`, rather than in none. + if err := s.Promote(next, to); err != nil { + return Moved{}, err + } + return Moved{From: from, To: to}, s.removeFrom(from, id) +} + +// FindContent returns the committed entry whose content key is key, if any — +// the duplicate check for a capture, which has no id of its own to compare yet. +func (s *Store) FindContent(key string) (Entry, error) { return s.findContentExcept(key, "") } + +// findContentExcept returns a committed entry, other than except, whose content +// key is key. +func (s *Store) findContentExcept(key, except string) (Entry, error) { + shards, err := s.ListShards() + if err != nil { + return Entry{}, err + } + for _, sh := range shards { + entries, _, err := s.LoadShard(sh) + if err != nil { + return Entry{}, err + } + for _, e := range entries { + if e.ID != except && e.ContentKey() == key { + return e, nil + } + } + } + return Entry{}, nil +} + +// removeFrom drops one entry from one named shard. Drop searches every shard +// for the id, which during a move would find whichever copy comes first. +func (s *Store) removeFrom(shard, id string) error { + entries, _, err := s.LoadShard(shard) + if err != nil { + return err + } + return s.SaveShard(shard, slices.DeleteFunc(entries, func(e Entry) bool { return e.ID == id })) +} + +// Update changes a staged entry in place, keeping its id and its position in +// the queue — which matters, because the queue's order is the timeline +// `staged --since` reads. +func (q *Queue) Update(id string, change func(Entry) (Entry, error)) (bool, error) { + release, err := acquireQueueLock(q.Path) + if err != nil { + return false, err + } + defer release() + + entries, err := q.Load() + if err != nil { + return false, err + } + at := slices.IndexFunc(entries, func(e Entry) bool { return e.ID == id }) + if at < 0 { + return false, nil + } + next, err := change(entries[at]) + if err != nil { + return false, err + } + next.ID = id + for i, e := range entries { + if i != at && e.ContentKey() == next.ContentKey() { + return false, fmt.Errorf("that would make %s say the same as staged %s (%q)", id, e.ID, e.Title) + } + } + entries[at] = next + return true, q.write(entries) +} diff --git a/cmd/mxcli/brain/edit_test.go b/cmd/mxcli/brain/edit_test.go new file mode 100644 index 0000000000..054f0845a2 --- /dev/null +++ b/cmd/mxcli/brain/edit_test.go @@ -0,0 +1,266 @@ +// SPDX-License-Identifier: Apache-2.0 + +package brain + +import ( + "os" + "strings" + "testing" +) + +// ako/mxcli#1950: an entry's id is a handle, so changing its wording must not +// change its identity, orphan it, or leave the old text behind. + +func promoteAll(t *testing.T, s *Store, es ...Entry) { + t.Helper() + for _, e := range es { + if err := s.Promote(e, e.Shard()); err != nil { + t.Fatal(err) + } + } +} + +func editTo(text string) func(Entry) (Entry, error) { + return func(e Entry) (Entry, error) { return e.Edit(text, nil, false) } +} + +func TestEditKeepsTheIdAndReplacesTheTextOnce(t *testing.T) { + s := NewStore(t.TempDir()) + if _, err := s.Init(); err != nil { + t.Fatal(err) + } + const before = "A planner can create a visit for a site with a window and one engineer" + const after = "A planner can create a visit for a site with a window and one named engineer" + req := mustRequirement(t, before, "01-scheduling", "@Scheduling.Visit") + other := mustRequirement(t, "A planner can cancel a visit", "01-scheduling", "@Scheduling.ACT_Cancel") + promoteAll(t, s, req, other) + + res, err := s.Update(req.ID, editTo(after)) + if err != nil { + t.Fatal(err) + } + if res.From != req.Shard() || res.To != req.Shard() { + t.Errorf("an edit that changes only wording must not move: %+v", res) + } + + entries, _, err := s.LoadShard(req.Shard()) + if err != nil { + t.Fatal(err) + } + if len(entries) != 2 || entries[0].ID != req.ID || entries[0].Title != after { + t.Fatalf("after edit: %+v — want the same id, in the same position, with the new text", entries) + } + if entries[1].ID != other.ID { + t.Errorf("the other entry was disturbed: %+v", entries[1]) + } + raw, err := os.ReadFile(s.ShardPath(req.Shard())) + if err != nil { + t.Fatal(err) + } + if n := strings.Count(string(raw), after); n != 1 { + t.Errorf("new text appears %d times, want 1", n) + } + if strings.Contains(string(raw), "and one engineer") { + t.Error("the old text is still in the shard") + } + + rep, err := Check(s, stubResolver{}, []string{req.Shard()}) + if err != nil { + t.Fatal(err) + } + if rep.Failed() || rep.Slices[0].Planned != 2 { + t.Errorf("check after edit: %+v", rep) + } +} + +// Once edited, an entry's stored id no longer matches its text, so a duplicate +// check that compares stored ids is blind in both directions. Both are asserted. +func TestDuplicatesAreFoundByContentNotByStoredID(t *testing.T) { + dir := t.TempDir() + s := NewStore(dir) + if _, err := s.Init(); err != nil { + t.Fatal(err) + } + original := mustRequirement(t, "A planner can create a visit", "01-scheduling", "@Scheduling.Visit") + promoteAll(t, s, original) + if _, err := s.Update(original.ID, editTo("A planner can create a visit for one engineer")); err != nil { + t.Fatal(err) + } + + // 1. The edited wording, captured again, is a duplicate — though its + // natural id differs from the stored one. + again := mustRequirement(t, "A planner can create a visit for one engineer", "01-scheduling", "@Scheduling.Visit") + if again.ID == original.ID { + t.Fatal("test premise: the edited text must hash to a different id") + } + if err := s.Promote(again, again.Shard()); err == nil { + t.Error("promoting the edited wording a second time was accepted: duplicate check compares stored ids") + } + + // 2. The ORIGINAL wording hashes to the edited entry's handle. It is a + // different fact now, so it is not a duplicate — but it must not + // share the id either, or one handle names two entries. + old := mustRequirement(t, "A planner can create a visit", "01-scheduling", "@Scheduling.Visit") + if old.ID != original.ID { + t.Fatal("test premise: the original text must hash to the edited entry's id") + } + taken, err := TakenIDs(s, NewQueue(dir)) + if err != nil { + t.Fatal(err) + } + fresh := AssignFreeID(old, taken) + if fresh.ID == original.ID { + t.Fatal("the original wording was given the edited entry's id") + } + if fresh.ContentKey() != old.ContentKey() { + t.Error("assigning a free id must not change the content key") + } + if err := s.Promote(fresh, fresh.Shard()); err != nil { + t.Errorf("the original wording, re-captured, is a new fact and must promote: %v", err) + } + + // Control: with nothing edited, AssignFreeID is the identity — the + // content-derived id still IS the id for every entry never changed. + untouched := mustRequirement(t, "A planner can delete a visit", "01-scheduling", "@Scheduling.ACT_Delete") + if got := AssignFreeID(untouched, taken); got.ID != untouched.ID { + t.Errorf("an untaken id was changed: %s -> %s", untouched.ID, got.ID) + } +} + +func TestEditRefusesToTurnAnEntryIntoAnotherOne(t *testing.T) { + s := NewStore(t.TempDir()) + if _, err := s.Init(); err != nil { + t.Fatal(err) + } + a := mustRequirement(t, "A planner can create a visit", "01-scheduling", "@Scheduling.Visit") + b := mustRequirement(t, "A planner can cancel a visit", "01-scheduling", "@Scheduling.Visit") + promoteAll(t, s, a, b) + + _, err := s.Update(b.ID, editTo("A planner can create a visit")) + if err == nil || !strings.Contains(err.Error(), a.ID) { + t.Errorf("err = %v, want a refusal naming %s", err, a.ID) + } +} + +// A decision's anchors decide its file. Re-anchoring one into another module +// must move it there, or `check` reports it misfiled the moment it runs. +func TestReanchoringADecisionMovesItToItsShard(t *testing.T) { + s := NewStore(t.TempDir()) + if _, err := s.Init(); err != nil { + t.Fatal(err) + } + dec := mustEntry(t, "Orders are committed by Finance", "@Sales.Order") + promoteAll(t, s, dec) + + res, err := s.Update(dec.ID, func(e Entry) (Entry, error) { + return e.Edit(e.Title, []string{"@Finance.ACT_Post"}, true) + }) + if err != nil { + t.Fatal(err) + } + if res.From != "Sales" || res.To != "Finance" { + t.Errorf("moved %s -> %s, want Sales -> Finance", res.From, res.To) + } + if e, shard, _ := s.Find(dec.ID); shard != "Finance" || e.ID != dec.ID { + t.Errorf("Find after re-anchor: %q in %s", e.ID, shard) + } + rep, err := Check(s, stubResolver{"@Finance.ACT_Post": {State: Resolved, Module: "Finance"}}, []string{"Finance"}) + if err != nil { + t.Fatal(err) + } + if rep.Failed() { + t.Errorf("check after re-anchor: %+v", rep) + } + + // Control: a decision put somewhere on purpose (promote --to project) + // stays there when its anchors change. Following the anchors only applies + // to an entry that was where its anchors put it. + pinned := mustEntry(t, "Every module logs to the same node", "@Sales.Order") + if err := s.Promote(pinned, ProjectShard); err != nil { + t.Fatal(err) + } + res, err = s.Update(pinned.ID, func(e Entry) (Entry, error) { + return e.Edit(e.Title, []string{"@Finance.Order"}, true) + }) + if err != nil { + t.Fatal(err) + } + if res.To != ProjectShard { + t.Errorf("a decision placed in project.md on purpose moved to %s", res.To) + } +} + +func TestMoveChangesARequirementsSliceAndKeepsItsID(t *testing.T) { + s := NewStore(t.TempDir()) + if _, err := s.Init(); err != nil { + t.Fatal(err) + } + req := mustRequirement(t, "A planner can create a visit", "01-scheduling", "@Scheduling.Visit") + promoteAll(t, s, req) + + res, err := s.Update(req.ID, func(e Entry) (Entry, error) { return e.MoveToSlice("02-dispatch") }) + if err != nil { + t.Fatal(err) + } + if res.To != PlanShard("02-dispatch") { + t.Errorf("moved to %s", res.To) + } + e, shard, _ := s.Find(req.ID) + if shard != PlanShard("02-dispatch") || e.Slice != "02-dispatch" { + t.Errorf("after move: %+v in %s", e, shard) + } + if _, err := os.Stat(s.ShardPath(req.Shard())); !os.IsNotExist(err) { + t.Error("the emptied slice file was left behind") + } + + // A decision has no slice; turning one into a requirement is a change of + // meaning (its anchors would stop failing when they go stale), not a move. + dec := mustEntry(t, "Orders are committed by Finance", "@Sales.Order") + if _, err := dec.MoveToSlice("01-scheduling"); err == nil { + t.Error("moving a decision into a slice must be refused") + } +} + +func TestEditValidatesLikeCapture(t *testing.T) { + req := mustRequirement(t, "A planner can create a visit", "01-scheduling", "@Scheduling.Visit") + if _, err := req.Edit(" ", nil, false); err == nil { + t.Error("an empty title must be refused") + } + if _, err := req.Edit(req.Title, []string{"@Scheduling"}, true); err == nil { + t.Error("a requirement re-anchored at a bare module must be refused, as capture refuses it") + } + got, err := req.Edit("New wording\n\nWith a body.", nil, false) + if err != nil { + t.Fatal(err) + } + if got.ID != req.ID || got.Slice != req.Slice || got.Kind != req.Kind || got.Date != req.Date || + got.Title != "New wording" || got.Body != "With a body." || len(got.Anchors) != 1 { + t.Errorf("edit changed more than the text: %+v", got) + } +} + +func TestQueueUpdateEditsAStagedEntryInPlace(t *testing.T) { + q := NewQueue(t.TempDir()) + a := mustEntry(t, "Orders are committed by Finance", "@Sales.Order") + b := mustEntry(t, "Invoices are numbered per year", "@Sales.Invoice") + for _, e := range []Entry{a, b} { + if _, err := q.Append(e); err != nil { + t.Fatal(err) + } + } + ok, err := q.Update(a.ID, editTo("Orders are committed by Finance, never by Sales")) + if err != nil || !ok { + t.Fatalf("Update = %v, %v", ok, err) + } + got, err := q.Load() + if err != nil { + t.Fatal(err) + } + if len(got) != 2 || got[0].ID != a.ID || got[0].Title != "Orders are committed by Finance, never by Sales" || got[1].ID != b.ID { + t.Errorf("queue after edit: %+v", got) + } + // The queue's own duplicate check is by content too. + if added, _ := q.Append(mustEntry(t, "Orders are committed by Finance, never by Sales", "@Sales.Order")); added { + t.Error("the edited wording was queued a second time") + } +} diff --git a/cmd/mxcli/brain/entry.go b/cmd/mxcli/brain/entry.go index 47a484c05a..91ae5844a4 100644 --- a/cmd/mxcli/brain/entry.go +++ b/cmd/mxcli/brain/entry.go @@ -53,7 +53,12 @@ type Entry struct { Title string `json:"title"` Body string `json:"body,omitempty"` Anchors []string `json:"anchors,omitempty"` - Date string `json:"date"` + // Source is where the entry came from: a document path relative to the + // project directory, with / separators, plus an optional #locator + // ("docs/prd.md#L18-21"). Provenance, not content — it is not part of the + // content key, so the same fact found in two places is still one fact. + Source string `json:"source,omitempty"` + Date string `json:"date"` } // EntryKind normalises the zero value: an entry with no kind is a decision. diff --git a/cmd/mxcli/brain/nomodel_test.go b/cmd/mxcli/brain/nomodel_test.go new file mode 100644 index 0000000000..cba727a624 --- /dev/null +++ b/cmd/mxcli/brain/nomodel_test.go @@ -0,0 +1,102 @@ +// SPDX-License-Identifier: Apache-2.0 + +package brain + +import ( + "encoding/json" + "testing" +) + +// With no model there is nothing to check an anchor against. The honest +// answers are "nothing is built" for a requirement and "could not check" for a +// decision — not "built" and not "stale" (ako/mxcli#1948). +// +// The requirement half is the trap: checkSlice counts every state other than +// NotFound as built, so that a NotIndexable anchor does not report finished +// work as outstanding. A new state added without touching that branch reports +// a project that does not exist yet as 100% built. +func TestNoModelReportsNothingBuiltAndFailsNothing(t *testing.T) { + s := NewStore(t.TempDir()) + if _, err := s.Init(); err != nil { + t.Fatal(err) + } + req := mustRequirement(t, "A planner can create a visit", "01-scheduling", "@Scheduling.Visit") + unanchored := mustRequirement(t, "It should feel fast", "01-scheduling") + dec := mustEntry(t, "Visits are committed by the planner", "@Scheduling.Visit") + for _, e := range []Entry{req, unanchored} { + if err := s.Promote(e, e.Shard()); err != nil { + t.Fatal(err) + } + } + if err := s.Promote(dec, "Scheduling"); err != nil { + t.Fatal(err) + } + shards := []string{"Scheduling", req.Shard()} + + rep, err := Check(s, NoModel{}, shards) + if err != nil { + t.Fatal(err) + } + if len(rep.Slices) != 1 { + t.Fatalf("slices = %+v", rep.Slices) + } + if got := rep.Slices[0]; got.Built != 0 || got.Planned != 1 || got.Unanchored != 1 { + t.Errorf("no model: %+v, want Built 0, Planned 1, Unanchored 1", got) + } + if rep.Failed() { + t.Errorf("an unverifiable anchor must not fail the check: %+v", rep) + } + if rep.Unverifiable != 2 { + t.Errorf("unverifiable = %d, want 2 (one requirement anchor, one decision anchor)", rep.Unverifiable) + } + for _, f := range rep.Findings { + if f.State == NotFound { + t.Errorf("no model must not be reported as NOT FOUND: %+v", f) + } + } + if rep.ResolvedN != 0 { + t.Errorf("resolved = %d with no model, want 0", rep.ResolvedN) + } + + // Control: the same store against a model that lacks the element. The + // decision is stale and fails — so the pass above is the new state's + // doing, not a check that has stopped failing anything. + rep, err = Check(s, stubResolver{}, shards) + if err != nil { + t.Fatal(err) + } + if !rep.Failed() { + t.Error("control: a decision anchored at a missing element must still fail when there is a model") + } + if rep.Unverifiable != 0 { + t.Errorf("control: unverifiable = %d with a model, want 0", rep.Unverifiable) + } + + // And the requirement flips to built once the model has the element. + rep, err = Check(s, stubResolver{"@Scheduling.Visit": {State: Resolved, Module: "Scheduling", Kind: "entity"}}, shards) + if err != nil { + t.Fatal(err) + } + if got := rep.Slices[0]; got.Built != 1 || got.Planned != 0 { + t.Errorf("with the element present: %+v, want Built 1", got) + } +} + +// The state names itself and round-trips, rather than falling through to +// "not indexable" in String() and being rejected by UnmarshalJSON. +func TestUnverifiableStateRoundTrips(t *testing.T) { + if got := Unverifiable.String(); got != "unverifiable" { + t.Errorf("String() = %q", got) + } + b, err := json.Marshal(Unverifiable) + if err != nil { + t.Fatal(err) + } + var back AnchorState + if err := json.Unmarshal(b, &back); err != nil { + t.Fatalf("unmarshal %s: %v", b, err) + } + if back != Unverifiable { + t.Errorf("round trip gave %v", back) + } +} diff --git a/cmd/mxcli/brain/shard.go b/cmd/mxcli/brain/shard.go index e76364cc48..a99ff79a62 100644 --- a/cmd/mxcli/brain/shard.go +++ b/cmd/mxcli/brain/shard.go @@ -50,6 +50,15 @@ func renderEntry(e Entry) string { open = " · OPEN" } fmt.Fprintf(&b, "Anchors: %s · id `%s` · %s%s\n", renderAnchors(e.Anchors), e.ID, e.Date, open) + if e.Source != "" { + // Its own line, directly after the metadata line (ako/mxcli#1951). + // Not ON that line: older builds match it with an exact regex and + // would report every entry carrying a source as malformed. Not + // before it: older builds discard lines between title and metadata + // on their next rewrite. Here, an older build reads it as the first + // line of the body and keeps it. + fmt.Fprintf(&b, "Source: `%s`\n", e.Source) + } if e.Body != "" { fmt.Fprintf(&b, "\n%s\n", e.Body) } @@ -137,10 +146,28 @@ func parseEntry(block string) (Entry, bool) { if metaAt < 0 { return Entry{}, false } - e.Body = strings.TrimSpace(strings.Join(lines[metaAt+1:], "\n")) + rest := lines[metaAt+1:] + // The first non-blank line after the metadata may be the source. An + // older build that rewrote the shard put a blank line before it, having + // taken it for body text, so blank lines are skipped — but only the FIRST + // line is read: a body that mentions a source further down is prose. + for i, ln := range rest { + if strings.TrimSpace(ln) == "" { + continue + } + if m := sourceLine.FindStringSubmatch(strings.TrimSpace(ln)); m != nil { + e.Source = m[1] + rest = rest[i+1:] + } + break + } + e.Body = strings.TrimSpace(strings.Join(rest, "\n")) return e, true } +// sourceLine matches an entry's provenance line. +var sourceLine = regexp.MustCompile("^Source: `([^`]+)`$") + func firstLine(s string) string { if i := strings.IndexByte(s, '\n'); i >= 0 { return strings.TrimSpace(s[:i]) diff --git a/cmd/mxcli/brain/source.go b/cmd/mxcli/brain/source.go new file mode 100644 index 0000000000..560dae1c23 --- /dev/null +++ b/cmd/mxcli/brain/source.go @@ -0,0 +1,97 @@ +// SPDX-License-Identifier: Apache-2.0 + +// source.go - which document, and which part of it, an entry came from. +// +// A requirements tool is believable when clicking a requirement shows the line +// it came from, and useful when it can say which parts of a document produced +// no requirement at all (ako/mxcli#1951). Both need the source recorded on the +// entry, in a form that survives a checkout on another machine — so a path +// relative to the project directory, never an absolute one — and pointing at a +// file the repository actually has. +package brain + +import ( + "fmt" + "os" + "path/filepath" + "strings" +) + +// NormalizeSource turns what a user typed — a path, absolute or relative to +// base, optionally followed by #locator — into the stored form: relative to +// base, / separated. It refuses a file that does not exist or lies outside +// base: provenance pointing at a file nobody else has is not provenance. +// +// The locator is free text ("L18-21", "Goals", "section 3"); only a backtick or +// a line break is refused, since either would break the shard line. +func NormalizeSource(base, raw string) (string, error) { + path, locator, hasLocator := strings.Cut(strings.TrimSpace(raw), "#") + if path == "" { + return "", fmt.Errorf("--source needs a document path, optionally followed by #") + } + if strings.ContainsAny(locator, "`\n\r") { + return "", fmt.Errorf("source locator %q: no backticks or line breaks", locator) + } + abs := path + if !filepath.IsAbs(abs) { + abs = filepath.Join(base, abs) + } + absBase, err := filepath.Abs(base) + if err != nil { + return "", err + } + abs, err = filepath.Abs(abs) + if err != nil { + return "", err + } + rel, err := filepath.Rel(absBase, abs) + if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) { + return "", fmt.Errorf("source %s is outside the project (%s); keep the document in the repository so the provenance travels with it", path, absBase) + } + info, err := os.Stat(abs) + if err != nil { + return "", fmt.Errorf("source %s: %w", path, err) + } + if info.IsDir() { + return "", fmt.Errorf("source %s is a directory, not a document", path) + } + out := filepath.ToSlash(rel) + if hasLocator && locator != "" { + out += "#" + locator + } + return out, nil +} + +// SourcePath is the document part of a stored source, without the locator. +func SourcePath(source string) string { + path, _, _ := strings.Cut(source, "#") + return path +} + +// SourceFinding is an entry whose source document is missing. +type SourceFinding struct { + Shard string `json:"shard"` + EntryID string `json:"entry_id"` + Title string `json:"title"` + Source string `json:"source"` +} + +// ProjectDir is the directory the store belongs to — what sources are +// relative to. +func (s *Store) ProjectDir() string { + return filepath.Clean(strings.TrimSuffix(s.Root, filepath.FromSlash(StoreDir))) +} + +func (s *Store) missingSources(shard string, entries []Entry) []SourceFinding { + var out []SourceFinding + for _, e := range entries { + if e.Source == "" { + continue + } + p := filepath.Join(s.ProjectDir(), filepath.FromSlash(SourcePath(e.Source))) + if _, err := os.Stat(p); err != nil { + out = append(out, SourceFinding{Shard: shard, EntryID: e.ID, Title: e.Title, Source: e.Source}) + } + } + return out +} diff --git a/cmd/mxcli/brain/source_test.go b/cmd/mxcli/brain/source_test.go new file mode 100644 index 0000000000..3cc715d90c --- /dev/null +++ b/cmd/mxcli/brain/source_test.go @@ -0,0 +1,164 @@ +// SPDX-License-Identifier: Apache-2.0 + +package brain + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// ako/mxcli#1951: which sentence of which document an entry came from. + +func writeDoc(t *testing.T, base, rel string) { + t.Helper() + p := filepath.Join(base, filepath.FromSlash(rel)) + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte("# PRD\n\nA planner can create a visit.\n"), 0o644); err != nil { + t.Fatal(err) + } +} + +func TestSourceRoundTripsThroughAShard(t *testing.T) { + req := mustRequirement(t, "A planner can create a visit\n\nWith a window.", "01-scheduling", "@Scheduling.Visit") + req.Source = "docs/sample-prd.md#L18-21" + + text := RenderShard(req.Shard(), []Entry{req}) + // Its own line, IMMEDIATELY after the metadata line: never on it (an + // older parser's exact regex would call every such entry malformed and + // fail `check`), never before it (an older parser drops those lines on + // its next rewrite). After it, an older build reads it as body and keeps it. + lines := strings.Split(text, "\n") + for i, ln := range lines { + if strings.HasPrefix(ln, "Anchors: ") { + if strings.Contains(ln, "sample-prd") { + t.Errorf("source rendered on the metadata line: %q", ln) + } + if i+1 >= len(lines) || lines[i+1] != "Source: `docs/sample-prd.md#L18-21`" { + t.Errorf("line after metadata = %q, want the Source line", lines[i+1]) + } + } + } + + got, malformed, err := ParseShard(req.Shard(), text) + if err != nil || len(malformed) != 0 || len(got) != 1 { + t.Fatalf("parse: %v %v %+v", err, malformed, got) + } + if got[0].Source != req.Source || got[0].Body != "With a window." { + t.Errorf("round trip: source %q, body %q", got[0].Source, got[0].Body) + } + + // Control: an entry without a source renders exactly as before, so + // existing shards are byte-for-byte unchanged by this feature. + req.Source = "" + if strings.Contains(RenderShard(req.Shard(), []Entry{req}), "Source:") { + t.Error("an entry with no source must not render a Source line") + } +} + +// An older mxcli reads the Source line as the first line of the body, and on +// its next rewrite renders that body after a blank line. The new parser must +// take the line back out of the body in that shape too, or one round trip +// through an old build turns provenance into prose. +func TestSourceSurvivesARewriteByAnOlderBuild(t *testing.T) { + const oldRewrite = "## A planner can create a visit\n\n" + + "Anchors: `@Scheduling.Visit` · id `abc123` · 2026-09-03\n\n" + + "Source: `docs/sample-prd.md#L18-21`\n\nWith a window.\n" + got, malformed, err := ParseShard(PlanShard("01-scheduling"), oldRewrite) + if err != nil || len(malformed) != 0 || len(got) != 1 { + t.Fatalf("parse: %v %v %+v", err, malformed, got) + } + if got[0].Source != "docs/sample-prd.md#L18-21" || got[0].Body != "With a window." { + t.Errorf("source %q, body %q", got[0].Source, got[0].Body) + } + + // Control: a body that merely MENTIONS a source further down is prose, + // not provenance — only the first line after the metadata is read. + const prose = "## X\n\nAnchors: none · id `abc124` · 2026-09-03\n\nSee below.\n\nSource: `docs/other.md`\n" + got, _, _ = ParseShard(ProjectShard, prose) + if len(got) != 1 || got[0].Source != "" || !strings.Contains(got[0].Body, "Source: `docs/other.md`") { + t.Errorf("a later Source line was taken as provenance: %+v", got) + } +} + +// The same requirement found in two places in a document is still one fact. +func TestContentKeyIgnoresSource(t *testing.T) { + a := mustRequirement(t, "A planner can create a visit", "01-scheduling", "@Scheduling.Visit") + b := a + b.Source = "docs/sample-prd.md#L18" + if a.ContentKey() != b.ContentKey() { + t.Error("source changed the content key") + } +} + +func TestNormalizeSource(t *testing.T) { + base := t.TempDir() + writeDoc(t, base, "docs/sample-prd.md") + + for _, tc := range []struct{ in, want string }{ + {filepath.Join(base, "docs", "sample-prd.md"), "docs/sample-prd.md"}, + {filepath.Join(base, "docs", "sample-prd.md") + "#L18-21", "docs/sample-prd.md#L18-21"}, + {"docs/sample-prd.md#Goals", "docs/sample-prd.md#Goals"}, // relative to base + } { + got, err := NormalizeSource(base, tc.in) + if err != nil || got != tc.want { + t.Errorf("NormalizeSource(%q) = %q, %v; want %q", tc.in, got, err, tc.want) + } + } + for _, bad := range []string{ + filepath.Join(base, "docs", "missing.md"), // provenance nobody has + filepath.Join(filepath.Dir(base), "outside.md"), // not in the repository + filepath.Join(base, "docs", "sample-prd.md") + "#a`b", // would break the line + "", + } { + if bad == filepath.Join(filepath.Dir(base), "outside.md") { + writeDoc(t, filepath.Dir(base), "outside.md") + } + if got, err := NormalizeSource(base, bad); err == nil { + t.Errorf("NormalizeSource(%q) = %q, want an error", bad, got) + } + } +} + +// A source file that has gone is a reference to nothing — the same defect as an +// anchor that names nothing, and it fails the check the same way. +func TestCheckFailsWhenASourceFileIsGone(t *testing.T) { + base := t.TempDir() + s := NewStore(base) + if _, err := s.Init(); err != nil { + t.Fatal(err) + } + writeDoc(t, base, "docs/sample-prd.md") + req := mustRequirement(t, "A planner can create a visit", "01-scheduling", "@Scheduling.Visit") + req.Source = "docs/sample-prd.md#L18-21" + dec := mustEntry(t, "Visits are committed by the planner", "@Scheduling.Visit") + dec.Source = "docs/sample-prd.md#L30" + promoteAll(t, s, req) + if err := s.Promote(dec, "Scheduling"); err != nil { + t.Fatal(err) + } + r := stubResolver{"@Scheduling.Visit": {State: Resolved, Module: "Scheduling"}} + shards := []string{"Scheduling", req.Shard()} + + rep, err := Check(s, r, shards) + if err != nil { + t.Fatal(err) + } + if rep.Failed() || len(rep.MissingSources) != 0 { + t.Fatalf("control: the source exists, so nothing fails: %+v", rep) + } + + if err := os.Remove(filepath.Join(base, "docs", "sample-prd.md")); err != nil { + t.Fatal(err) + } + rep, err = Check(s, r, shards) + if err != nil { + t.Fatal(err) + } + if !rep.Failed() || len(rep.MissingSources) != 2 { + t.Errorf("a missing source must fail, once per entry: %+v", rep.MissingSources) + } +} diff --git a/cmd/mxcli/brain/staged.go b/cmd/mxcli/brain/staged.go index 3517d3eb33..8ad9ed2cf1 100644 --- a/cmd/mxcli/brain/staged.go +++ b/cmd/mxcli/brain/staged.go @@ -78,8 +78,10 @@ func (q *Queue) Append(e Entry) (added bool, err error) { if err != nil { return false, err } + // By content, not by stored id: an edited entry keeps an id its text no + // longer hashes to (edit.go). for _, existing := range entries { - if existing.ID == e.ID { + if existing.ID == e.ID || existing.ContentKey() == e.ContentKey() { return false, nil } } diff --git a/cmd/mxcli/brain/store.go b/cmd/mxcli/brain/store.go index 558674f36f..4680385567 100644 --- a/cmd/mxcli/brain/store.go +++ b/cmd/mxcli/brain/store.go @@ -212,6 +212,11 @@ func (s *Store) Promote(e Entry, shard string) error { if existing.ID == e.ID { return fmt.Errorf("%s already carries entry %s (%q)", shard, e.ID, existing.Title) } + // By content as well: an edited entry keeps an id its text no longer + // hashes to, so the same words captured again arrive with a new id. + if existing.ContentKey() == e.ContentKey() { + return fmt.Errorf("%s already says this, as entry %s (%q)", shard, existing.ID, existing.Title) + } } next := append(entries, e) // Counted WITH frontmatter: the cap is what a session pays to load the diff --git a/cmd/mxcli/brain_before_project_test.go b/cmd/mxcli/brain_before_project_test.go new file mode 100644 index 0000000000..1aeaed71a1 --- /dev/null +++ b/cmd/mxcli/brain_before_project_test.go @@ -0,0 +1,290 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/cmd/mxcli/brain" +) + +// A brain can be born before its project (ako/mxcli#1949). These tests pin +// down the three halves that make that official rather than accidental: +// -C names the directory, `mxcli new` adopts a brain already there, and the +// move that installs the project never touches it. + +// seedBrain makes a directory hold exactly what a brain-first workflow leaves: +// an initialised store with one promoted requirement, plus a queued capture. +func seedBrain(t *testing.T, dir string) (promoted brain.Entry) { + t.Helper() + store := brain.NewStore(dir) + if _, err := store.Init(); err != nil { + t.Fatal(err) + } + req, err := brain.NewRequirement("Accounts can be administered", []string{"@Administration.Account"}, "01-accounts", brainTestDay()) + if err != nil { + t.Fatal(err) + } + if err := store.Promote(req, req.Shard()); err != nil { + t.Fatal(err) + } + queued, err := brain.NewEntry("Accounts are created by an admin only", []string{"@Administration.Account"}, brainTestDay()) + if err != nil { + t.Fatal(err) + } + if _, err := brain.NewQueue(dir).Append(queued); err != nil { + t.Fatal(err) + } + return req +} + +func TestNewDestinationAcceptsOnlyAnEmptyDirOrABrain(t *testing.T) { + write := func(t *testing.T, dir, rel, content string) { + t.Helper() + p := filepath.Join(dir, rel) + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + cases := []struct { + name string + setup func(t *testing.T, dir string) + wantAdopt bool + wantErr string // substring; empty means accepted + }{ + {"missing", func(t *testing.T, dir string) { _ = os.Remove(dir) }, false, ""}, + {"empty", func(t *testing.T, dir string) {}, false, ""}, + {"brain and queue", func(t *testing.T, dir string) { seedBrain(t, dir) }, true, ""}, + {"store only", func(t *testing.T, dir string) { + if _, err := brain.NewStore(dir).Init(); err != nil { + t.Fatal(err) + } + }, true, ""}, + // Controls: each is a brain plus ONE thing that is not, or a + // docs/brain mxcli did not write. Adoption must stay this narrow, or + // `new` starts merging a project into someone's files. + {"foreign docs/brain", func(t *testing.T, dir string) { + write(t, dir, "docs/brain/README.md", "my own notes\n") + }, false, "not written by mxcli"}, + {"brain plus a stray file", func(t *testing.T, dir string) { + seedBrain(t, dir) + write(t, dir, "notes.txt", "x") + }, false, "not empty"}, + {"brain plus other docs", func(t *testing.T, dir string) { + seedBrain(t, dir) + write(t, dir, "docs/design.md", "x") + }, false, "not empty"}, + {"brain plus other .mxcli state", func(t *testing.T, dir string) { + seedBrain(t, dir) + write(t, dir, ".mxcli/catalog.db", "x") + }, false, "not empty"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + dir := filepath.Join(t.TempDir(), "App") + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + tc.setup(t, dir) + adopt, err := checkNewProjectDestination(dir) + if tc.wantErr == "" { + if err != nil { + t.Fatalf("refused: %v", err) + } + if adopt != tc.wantAdopt { + t.Errorf("adopt = %v, want %v", adopt, tc.wantAdopt) + } + return + } + if err == nil || !strings.Contains(err.Error(), tc.wantErr) { + t.Errorf("err = %v, want one containing %q", err, tc.wantErr) + } + }) + } +} + +// The move must merge the project around the brain and leave the brain's bytes +// alone. Before this, moveProject's rename failed on the non-empty directory +// and its os.CopyFS fallback merged by accident — and stopped half way on the +// first file that already existed. +func TestMoveProjectMergesAroundAnAdoptedBrain(t *testing.T) { + dest := filepath.Join(t.TempDir(), "App") + seedBrain(t, dest) + before := snapshot(t, dest) + + stage := fakeStage(t) + if err := moveProject(stage, dest); err != nil { + t.Fatalf("moveProject: %v", err) + } + for _, f := range []string{"App.mpr", filepath.Join("mprcontents", "aa", "unit.mxunit"), ".gitignore"} { + if _, err := os.Stat(filepath.Join(dest, f)); err != nil { + t.Errorf("%s did not arrive: %v", f, err) + } + } + after := snapshot(t, dest) + for path, content := range before { + if after[path] != content { + t.Errorf("brain file %s changed by the move", path) + } + } +} + +// A conflict is found before anything moves, so a refusal leaves both sides as +// they were — the guarantee `mxcli new` makes for every other failure. +func TestMoveProjectRefusesAConflictBeforeMovingAnything(t *testing.T) { + dest := filepath.Join(t.TempDir(), "App") + seedBrain(t, dest) + before := snapshot(t, dest) + + stage := fakeStage(t) + clash := filepath.Join(stage, "docs", "brain", "README.md") + if err := os.MkdirAll(filepath.Dir(clash), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(clash, []byte("template"), 0o644); err != nil { + t.Fatal(err) + } + + err := moveProject(stage, dest) + if err == nil || !strings.Contains(err.Error(), filepath.Join("docs", "brain", "README.md")) { + t.Fatalf("err = %v, want a conflict naming docs/brain/README.md", err) + } + if _, err := os.Stat(filepath.Join(dest, "App.mpr")); !os.IsNotExist(err) { + t.Error("App.mpr was moved although the move was refused") + } + if _, err := os.Stat(filepath.Join(stage, "App.mpr")); err != nil { + t.Errorf("the staged project lost App.mpr on a refused move: %v", err) + } + if got := snapshot(t, dest); len(got) != len(before) { + t.Errorf("destination changed: %d files, was %d", len(got), len(before)) + } +} + +// The issue's proof, minus `mx create-project` (which needs mxbuild): a brain +// born in an empty directory, a project installed into that directory the way +// `mxcli new` installs one, and the same -C command going from "no model, 0 of +// 1" to counting the requirement against the model. +func TestBrainBornBeforeTheProjectIsCountedAfterIt(t *testing.T) { + dir := filepath.Join(t.TempDir(), "PedApp") + + // Through the CLI, by directory: no .mpr is ever named. + runBrainForTest(t, []string{"brain", "init", "-C", dir}) + out := runBrainForTest(t, []string{"brain", "capture", "Accounts can be administered", + "-a", "@Administration.Account", "-s", "01-accounts", "-C", dir}) + id := strings.Fields(out)[1] + runBrainForTest(t, []string{"brain", "promote", id, "-C", dir}) + + out = runBrainForTest(t, []string{"brain", "plan", "-C", dir}) + if !strings.Contains(out, "0 of 1 requirements built") || !strings.Contains(out, "No model") { + t.Fatalf("before the project:\n%s", out) + } + + adopt, err := checkNewProjectDestination(dir) + if err != nil || !adopt { + t.Fatalf("checkNewProjectDestination = %v, %v; want adopt", adopt, err) + } + stage := filepath.Join(t.TempDir(), "stage") + if err := os.CopyFS(stage, os.DirFS(filepath.Join("..", "..", "testdata", "pedapp"))); err != nil { + t.Fatal(err) + } + if err := moveProject(stage, dir); err != nil { + t.Fatal(err) + } + + out = runBrainForTest(t, []string{"brain", "plan", "-C", dir}) + if !strings.Contains(out, "1 of 1 requirements built") || strings.Contains(out, "No model") { + t.Errorf("after the project was installed:\n%s", out) + } +} + +func TestResolveBrainLocation(t *testing.T) { + dir := t.TempDir() + if _, err := resolveBrainLocation("", ""); err == nil { + t.Error("neither -p nor -C must be refused") + } + if _, err := resolveBrainLocation(filepath.Join(dir, "App.mpr"), dir); err == nil { + t.Error("-p and -C together must be refused, not silently resolved to one") + } + + // -p is the shorthand: the directory of a path that need not exist. + loc, err := resolveBrainLocation(filepath.Join(dir, "Nothing.mpr"), "") + if err != nil || loc.Dir != dir || loc.MPR != filepath.Join(dir, "Nothing.mpr") { + t.Errorf("-p: %+v, %v", loc, err) + } + + // -C with no project yet: a directory and no model. + loc, err = resolveBrainLocation("", dir) + if err != nil || loc.Dir != dir || loc.MPR != "" { + t.Errorf("-C, no .mpr: %+v, %v", loc, err) + } + + // -C after `mxcli new`: the one .mpr there is the model. + if err := os.WriteFile(filepath.Join(dir, "App.mpr"), nil, 0o644); err != nil { + t.Fatal(err) + } + loc, err = resolveBrainLocation("", dir) + if err != nil || loc.MPR != filepath.Join(dir, "App.mpr") { + t.Errorf("-C, one .mpr: %+v, %v", loc, err) + } + + // Two is ambiguous, and guessing would check the brain against the wrong app. + if err := os.WriteFile(filepath.Join(dir, "Other.mpr"), nil, 0o644); err != nil { + t.Fatal(err) + } + if _, err := resolveBrainLocation("", dir); err == nil || !strings.Contains(err.Error(), "Other.mpr") { + t.Errorf("-C with two .mpr files: err = %v, want one naming them", err) + } +} + +// `brain init ` used to accept the directory and ignore it, writing the +// store wherever -p pointed instead. +func TestBrainInitRejectsAPositionalDirectory(t *testing.T) { + dir := t.TempDir() + resetBrainFlagsForTest(t) + rootCmd.SetArgs([]string{"brain", "init", filepath.Join(dir, "elsewhere"), "-C", dir}) + if err := rootCmd.Execute(); err == nil { + t.Error("a positional argument to init must be an error, not ignored") + } + if brain.NewStore(dir).Exists() { + t.Error("init ran despite the stray argument") + } +} + +func fakeStage(t *testing.T) string { + t.Helper() + stage := filepath.Join(t.TempDir(), "stage") + for _, f := range []string{"App.mpr", filepath.Join("mprcontents", "aa", "unit.mxunit"), ".gitignore"} { + p := filepath.Join(stage, f) + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte("content"), 0o644); err != nil { + t.Fatal(err) + } + } + return stage +} + +func snapshot(t *testing.T, root string) map[string]string { + t.Helper() + out := map[string]string{} + err := filepath.WalkDir(root, func(p string, d os.DirEntry, err error) error { + if err != nil || d.IsDir() { + return err + } + b, err := os.ReadFile(p) + rel, _ := filepath.Rel(root, p) + out[rel] = string(b) + return err + }) + if err != nil { + t.Fatal(err) + } + return out +} diff --git a/cmd/mxcli/cmd_brain.go b/cmd/mxcli/cmd_brain.go index 47af863c3d..31f2afb5cd 100644 --- a/cmd/mxcli/cmd_brain.go +++ b/cmd/mxcli/cmd_brain.go @@ -49,12 +49,20 @@ which is why 'brain plan' can report progress derived from the model instead of from a status column that goes stale the moment someone builds something. An agent captures; a person promotes. The queue is in .mxcli/ and is git-ignored, -so nothing reaches a pull request until someone has looked at it.`, +so nothing reaches a pull request until someone has looked at it. + +The brain can exist before the project. Name its directory with -C , or with +-p for the directory an .mpr is (or will be) in; neither needs the +project to exist. 'mxcli new' into that directory adopts the brain, and from then +on 'plan' and 'check' with the same -C count against the model.`, Example: ` mxcli brain init -p app.mpr + mxcli brain init -C ./workspace # before the project exists mxcli brain capture "Orders are committed by Finance, not Sales" -a @Sales.Order -a @Finance.ACT_Post -p app.mpr mxcli brain staged -p app.mpr mxcli brain promote a1b2c3 -p app.mpr + mxcli brain edit a1b2c3 "Orders are committed by Finance, never by Sales" -p app.mpr mxcli brain capture "Orders must be approvable by a manager" --slice 02-approvals -a @Sales.ACT_Order_Approve -p app.mpr + mxcli brain capture "A planner can create a visit" -s 01-scheduling --source docs/prd.md#L18-21 -p app.mpr mxcli brain brief --slice 02-approvals -p app.mpr mxcli brain plan -p app.mpr mxcli brain check -p app.mpr @@ -64,6 +72,9 @@ so nothing reaches a pull request until someone has looked at it.`, var brainInitCmd = &cobra.Command{ Use: "init", Short: "Create docs/brain/ (refuses a docs/brain/ it did not write)", + // No arguments: the directory is -C. Accepting one and ignoring it wrote + // the store wherever -p pointed instead of where the caller asked. + Args: cobra.NoArgs, Run: func(cmd *cobra.Command, args []string) { dir := brainProjectDir(cmd) written, err := brain.NewStore(dir).Init() @@ -108,7 +119,30 @@ var brainCaptureCmd = &cobra.Command{ if err != nil { brainFatal(err) } - added, err := brain.NewQueue(dir).Append(e) + if raw, _ := cmd.Flags().GetString("source"); raw != "" { + if e.Source, err = brainSource(dir, raw); err != nil { + brainFatal(err) + } + } + store, queue := brain.NewStore(dir), brain.NewQueue(dir) + // Already committed is a duplicate too, found by content: an edited + // entry's id no longer matches its text (ako/mxcli#1950). + if store.Exists() { + if dup, err := store.FindContent(e.ContentKey()); err != nil { + brainFatal(err) + } else if dup.ID != "" { + fmt.Printf("Already committed as %s — not added again.\n", dup.ID) + return + } + } + // The content-derived id may already be the handle of an entry whose + // text has since changed; a new fact gets an id of its own. + taken, err := brain.TakenIDs(store, queue) + if err != nil { + brainFatal(err) + } + e = brain.AssignFreeID(e, taken) + added, err := queue.Append(e) if err != nil { brainFatal(err) } @@ -244,6 +278,135 @@ var brainDropCmd = &cobra.Command{ }, } +var brainEditCmd = &cobra.Command{ + Use: "edit [] [-a @Anchor]…", + Short: "Change an entry's wording (and with -a its anchors), keeping its id", + Long: `Rewrite an entry in place. The id does not change. + +An id is a handle — 'brain promote ', 'brain resolve ' and prose cite +it — so correcting a word must not turn the entry into a different one. The text +replaces the title (first line) and body (the rest); -a, if given, replaces the +anchors, and is checked as 'capture' checks it. With -a the text may be left +out, to re-anchor an entry without retyping it. + +A staged entry is edited in the queue, keeping its place there. A committed one +is edited in its shard, keeping its position — unless re-anchoring a decision +points it at another module, in which case it moves to that module's file, so +'check' does not report it misfiled. A decision put in project.md on purpose +('promote --to project') stays there. + +An edit that would make the entry say the same as another is refused.`, + Example: ` mxcli brain edit 66dcf4 "A planner can create a visit for one named engineer" -C ./App + mxcli brain edit a1b2c3 -a @Finance.ACT_Post -p app.mpr # re-anchor, keep the text`, + Args: cobra.RangeArgs(1, 2), + Run: func(cmd *cobra.Command, args []string) { + anchors, _ := cmd.Flags().GetStringSlice("anchor") + replace := cmd.Flags().Changed("anchor") + var source string + if raw, _ := cmd.Flags().GetString("source"); raw != "" { + var err error + if source, err = brainSource(brainProjectDir(cmd), raw); err != nil { + brainFatal(err) + } + } + if len(args) == 1 && !replace && source == "" { + brainFatal(fmt.Errorf("nothing to change: give the new text, -a anchors, --source, or a combination")) + } + runBrainChange(cmd, args[0], "Edited", func(e brain.Entry) (brain.Entry, error) { + if source != "" { + e.Source = source + } + text := e.Title + if e.Body != "" { + text += "\n\n" + e.Body + } + if len(args) == 2 { + text = args[1] + } + return e.Edit(text, anchors, replace) + }) + }, +} + +var brainMoveCmd = &cobra.Command{ + Use: "move --slice ", + Short: "Move a requirement to another slice, keeping its id", + Long: `Move a requirement (or a question filed against a slice) to another slice. + +The id does not change. A decision has no slice — its file follows its anchors, +so re-anchor it with 'brain edit -a …' instead; giving it a slice +would turn it into a requirement, which is a change of meaning, not of place.`, + Example: ` mxcli brain move 66dcf4 --slice 02-dispatch -C ./App`, + Args: cobra.ExactArgs(1), + Run: func(cmd *cobra.Command, args []string) { + slice, _ := cmd.Flags().GetString("slice") + if slice == "" { + brainFatal(fmt.Errorf("say where to: --slice ")) + } + runBrainChange(cmd, args[0], "Moved", func(e brain.Entry) (brain.Entry, error) { + return e.MoveToSlice(slice) + }) + }, +} + +// brainSource resolves a --source the way a shell user means it — a relative +// path is relative to where they ran the command — and stores it relative to +// the project directory, so it means the same on every checkout. +func brainSource(projectDir, raw string) (string, error) { + path, locator, hasLocator := strings.Cut(raw, "#") + if path != "" && !filepath.IsAbs(path) { + abs, err := filepath.Abs(path) + if err != nil { + return "", err + } + path = abs + } + if hasLocator { + path += "#" + locator + } + return brain.NormalizeSource(projectDir, path) +} + +// runBrainChange applies a change to an entry wherever it is — the queue first, +// as 'drop' looks, then the store — and reports where it ended up. +func runBrainChange(cmd *cobra.Command, id, verb string, change func(brain.Entry) (brain.Entry, error)) { + dir := brainProjectDir(cmd) + queue := brain.NewQueue(dir) + ok, err := queue.Update(id, change) + if err != nil { + brainFatal(err) + } + if ok { + e, _, err := queue.Get(id) + if err != nil { + brainFatal(err) + } + if globalJSONFlag { + brainJSON(map[string]any{"id": id, "staged": true, "shard": e.Shard()}) + return + } + fmt.Printf("%s staged %s -> would promote into %s\n", verb, id, shardLabel(e.Shard())) + return + } + store := brain.NewStore(dir) + if !store.Exists() { + brainFatal(fmt.Errorf("no entry with id %s, staged or committed", id)) + } + moved, err := store.Update(id, change) + if err != nil { + brainFatal(err) + } + if globalJSONFlag { + brainJSON(map[string]any{"id": id, "staged": false, "shard": moved.To, "from": moved.From}) + return + } + if moved.From != moved.To { + fmt.Printf("%s %s, and moved it from %s to %s\n", verb, id, shardLabel(moved.From), shardLabel(moved.To)) + return + } + fmt.Printf("%s %s in %s\n", verb, id, shardLabel(moved.To)) +} + var brainShowCmd = &cobra.Command{ Use: "show [shard]", Short: "Size and headroom per shard (computed, never written down)", @@ -328,10 +491,14 @@ maintain or to go stale. Building the thing is what moves the number. Slices are listed in name order, so a numeric prefix is how a roadmap is sequenced: 01-accounts, 02-approvals. That is the user's choice, not a field -mxcli maintains.`, +mxcli maintains. + +The brain can exist before the project does. With no .mpr at the -p path, +nothing is built: every anchored requirement counts as planned, and the output +says the model is absent.`, Run: func(cmd *cobra.Command, args []string) { - projectPath := brainProjectPath(cmd) - store := brain.NewStore(filepath.Dir(projectPath)) + loc := brainLocationOf(cmd) + store := brain.NewStore(loc.Dir) sliceShards, err := store.ListSlices() if err != nil { brainFatal(err) @@ -348,7 +515,7 @@ mxcli maintains.`, fmt.Println(" mxcli brain capture \"\" --slice 01- -a @Module.Element") return } - resolver, closeFn, err := openBrainResolver(projectPath) + resolver, closeFn, err := openBrainResolverOrNoModel(loc.MPR) if err != nil { brainFatal(err) } @@ -359,9 +526,13 @@ mxcli maintains.`, brainFatal(err) } if globalJSONFlag { - brainJSON(planReport(rep.Slices)) + brainJSON(planReport(rep)) return } + if rep.ModelAbsent { + printNoModel(rep, loc) + fmt.Println() + } printBrainPlan(rep.Slices) }, } @@ -465,10 +636,16 @@ perfectly current. Separately, each entry is checked for being in the right shard. That is a second axis, not a fourth state — an anchor can resolve perfectly and the entry still sit in the wrong file. At least one anchor must belong to the entry's shard; -anchors into other modules are fine, because a fact can genuinely span two.`, +anchors into other modules are fine, because a fact can genuinely span two. + +An entry recorded with --source must still find that document in the project; +one whose source file is gone fails the check, like an anchor that names nothing. + +With no .mpr at the -p path there is nothing to resolve against: every anchor +is UNVERIFIABLE, which is counted and reported once and does not fail.`, Run: func(cmd *cobra.Command, args []string) { - projectPath := brainProjectPath(cmd) - store := brain.NewStore(filepath.Dir(projectPath)) + loc := brainLocationOf(cmd) + store := brain.NewStore(loc.Dir) if !store.Exists() { fmt.Println("No store yet. Create one with 'mxcli brain init'.") return @@ -478,7 +655,7 @@ anchors into other modules are fine, because a fact can genuinely span two.`, brainFatal(err) } if changed, _ := cmd.Flags().GetBool("changed"); changed { - shards, err = changedShards(filepath.Dir(projectPath), shards) + shards, err = changedShards(loc.Dir, shards) if err != nil { brainFatal(err) } @@ -488,7 +665,7 @@ anchors into other modules are fine, because a fact can genuinely span two.`, } } - resolver, closeFn, err := openBrainResolver(projectPath) + resolver, closeFn, err := openBrainResolverOrNoModel(loc.MPR) if err != nil { brainFatal(err) } @@ -506,7 +683,7 @@ anchors into other modules are fine, because a fact can genuinely span two.`, case ci: printBrainReportCI(rep) default: - printBrainReport(rep) + printBrainReport(rep, loc) } if rep.Failed() { os.Exit(1) @@ -519,6 +696,11 @@ anchors into other modules are fine, because a fact can genuinely span two.`, // (a not-indexable anchor) are omitted rather than printed, because a line in a // CI log reads as something to fix. func printBrainReportCI(rep brain.Report) { + if rep.ModelAbsent { + // stderr: in CI an absent model is worth seeing, but it is not a + // problem line, and stdout stays empty for a clean store. + fmt.Fprintf(os.Stderr, "brain: no model; %d anchor(s) not checked\n", rep.Unverifiable) + } for _, m := range rep.Malformed { fmt.Printf("brain: malformed entry: %s\n", m) } @@ -531,9 +713,28 @@ func printBrainReportCI(rep brain.Report) { for _, m := range rep.Misfiled { fmt.Printf("brain: %s: %s: misfiled, resolves in %s\n", shardLabel(m.Shard), m.EntryID, m.Belongs) } + for _, m := range rep.MissingSources { + fmt.Printf("brain: %s: %s: source %s not found\n", shardLabel(m.Shard), m.EntryID, brain.SourcePath(m.Source)) + } +} + +// printNoModel says, once, that nothing could be checked. Without it a plan of +// "0 built" reads as "nothing has been done" when the truth is "there is no +// project yet", and the two call for different actions. +func printNoModel(rep brain.Report, loc brainLocation) { + if !rep.ModelAbsent { + return + } + where := "at " + loc.MPR + if loc.MPR == "" { + where = "in " + loc.Dir + } + fmt.Printf("No model %s yet: %d anchor(s) not checked, so nothing counts as built.\n", + where, rep.Unverifiable) } -func printBrainReport(rep brain.Report) { +func printBrainReport(rep brain.Report, loc brainLocation) { + printNoModel(rep, loc) for _, m := range rep.Malformed { fmt.Printf("MALFORMED %s\n", m) } @@ -552,6 +753,10 @@ func printBrainReport(rep brain.Report) { fmt.Printf("MISFILED %s: %q is in %s but resolves in %s\n", m.EntryID, m.Title, shardLabel(m.Shard), belongs) } + for _, m := range rep.MissingSources { + fmt.Printf("NO SOURCE %s: %q cites %s, which is not in the project\n", + m.EntryID, m.Title, brain.SourcePath(m.Source)) + } if len(rep.Slices) > 0 { fmt.Println() printBrainPlan(rep.Slices) @@ -720,6 +925,20 @@ func brainCell(v any) string { return fmt.Sprint(v) } +// openBrainResolverOrNoModel opens the project, or answers for a project that +// does not exist yet. Only a missing file means "no model": an .mpr that is +// there but will not open is still an error, or a corrupt project would be +// reported as merely unbuilt. +func openBrainResolverOrNoModel(projectPath string) (brain.Resolver, func(), error) { + if projectPath == "" { + return brain.NoModel{}, func() {}, nil // -C on a directory with no .mpr yet + } + if _, err := os.Stat(projectPath); os.IsNotExist(err) { + return brain.NoModel{}, func() {}, nil + } + return openBrainResolver(projectPath) +} + func openBrainResolver(projectPath string) (brain.Resolver, func(), error) { // Catalog progress goes to stderr so stdout carries only the report — the // same split cmd_lint.go makes, and what lets `brain check` be piped. @@ -765,28 +984,86 @@ func shardLabel(shard string) string { } } -func brainProjectPath(cmd *cobra.Command) string { - p, _ := cmd.Flags().GetString("project") - if p == "" { - fmt.Fprintln(os.Stderr, "Error: --project (-p) is required") - os.Exit(1) +// brainLocation is where a brain lives and which model, if any, it is checked +// against. +// +// The store is in Dir; the .mpr is only ever needed by plan and check. That +// split is deliberate and load-bearing (ako/mxcli#1949): a brain is routinely +// started before its project exists — requirements first, app second — so no +// store command may open, or even stat, the .mpr. MPR is empty when -C names a +// directory that holds no project yet. +type brainLocation struct { + Dir string + MPR string +} + +// resolveBrainLocation turns -p / -C into a location. +// +// - -p is the shorthand: the directory that .mpr is (or will be) +// in. The file need not exist. +// - -C names the directory. The model is the one .mpr in it, if there +// is one, so the same command keeps working after `mxcli new` creates the +// project there; more than one is refused rather than guessed between. +func resolveBrainLocation(project, dir string) (brainLocation, error) { + switch { + case project != "" && dir != "": + return brainLocation{}, fmt.Errorf("pass -p or -C , not both") + case project != "": + return brainLocation{Dir: filepath.Dir(project), MPR: project}, nil + case dir != "": + matches, err := filepath.Glob(filepath.Join(dir, "*.mpr")) + if err != nil { + return brainLocation{}, err + } + switch len(matches) { + case 0: + return brainLocation{Dir: dir}, nil + case 1: + return brainLocation{Dir: dir, MPR: matches[0]}, nil + default: + names := make([]string, len(matches)) + for i, m := range matches { + names[i] = filepath.Base(m) + } + return brainLocation{}, fmt.Errorf("%s holds more than one project (%s); name one with -p", + dir, strings.Join(names, ", ")) + } + default: + return brainLocation{}, fmt.Errorf("say where the brain is: -p or -C ") } - return p } -func brainProjectDir(cmd *cobra.Command) string { return filepath.Dir(brainProjectPath(cmd)) } +func brainLocationOf(cmd *cobra.Command) brainLocation { + project, _ := cmd.Flags().GetString("project") + dir, _ := cmd.Flags().GetString("dir") + loc, err := resolveBrainLocation(project, dir) + if err != nil { + brainFatal(err) + } + return loc +} + +func brainProjectDir(cmd *cobra.Command) string { return brainLocationOf(cmd).Dir } func brainFatal(err error) { fmt.Fprintf(os.Stderr, "Error: %v\n", err) os.Exit(1) } -func init() { - for _, c := range []*cobra.Command{ +// brainLocatedCommands is every brain command that needs to know where the +// brain is — all of them, which is why -p and -C are registered from one list. +func brainLocatedCommands() []*cobra.Command { + return []*cobra.Command{ brainInitCmd, brainCaptureCmd, brainStagedCmd, brainPromoteCmd, - brainDropCmd, brainShowCmd, brainCheckCmd, - } { - c.Flags().StringP("project", "p", "", "Path to the .mpr file") + brainDropCmd, brainShowCmd, brainCheckCmd, brainPlanCmd, + brainResolveCmd, brainBriefCmd, brainEditCmd, brainMoveCmd, + } +} + +func init() { + for _, c := range brainLocatedCommands() { + c.Flags().StringP("project", "p", "", "Path to the .mpr file (need not exist yet; the brain lives beside it)") + c.Flags().StringP("dir", "C", "", "Directory holding the brain; its one .mpr, if any, is the model") } brainCaptureCmd.Flags().StringSliceP("anchor", "a", nil, "Anchor into the model (@Module, @Module.Element, @Module.Entity.Attribute); repeatable") @@ -794,6 +1071,13 @@ func init() { "Record this as an OPEN QUESTION — something not decided yet, so its anchors are not checked") brainCaptureCmd.Flags().StringP("slice", "s", "", "Record this as a requirement of the named slice (plan/.md) instead of a decision") + brainCaptureCmd.Flags().String("source", "", + "Where this came from: a document in the project, optionally # (docs/prd.md#L18-21)") + brainEditCmd.Flags().String("source", "", + "Set where this came from: a document in the project, optionally #") + brainEditCmd.Flags().StringSliceP("anchor", "a", nil, + "Replace the anchors with these (repeatable); without -a the anchors are kept") + brainMoveCmd.Flags().StringP("slice", "s", "", "The slice to move the requirement to") brainPromoteCmd.Flags().String("to", "", "Override the derived shard (use 'project' for a cross-cutting fact)") brainStagedCmd.Flags().String("since", "", @@ -805,18 +1089,15 @@ func init() { brainCheckCmd.Flags().Bool("changed", false, "Only check shards touched by the working tree") brainCheckCmd.Flags().Bool("ci", false, "Machine-friendly output for CI") - brainPlanCmd.Flags().StringP("project", "p", "", "Path to the .mpr file") brainPlanCmd.Flags().String("slice", "", "Report only this slice, instead of every slice in the plan") - brainBriefCmd.Flags().StringP("project", "p", "", "Path to the .mpr file") brainBriefCmd.Flags().String("slice", "", "The slice being worked; its modules are derived from its requirements' anchors") brainBriefCmd.Flags().StringSlice("module", nil, "Modules being worked, for a session with no slice; repeatable") - brainResolveCmd.Flags().StringP("project", "p", "", "Path to the .mpr file") brainCmd.AddCommand(brainInitCmd, brainCaptureCmd, brainStagedCmd, brainPromoteCmd, brainDropCmd, brainShowCmd, brainCheckCmd, brainPlanCmd, - brainResolveCmd, brainBriefCmd) + brainResolveCmd, brainBriefCmd, brainEditCmd, brainMoveCmd) rootCmd.AddCommand(brainCmd) } @@ -881,15 +1162,19 @@ func stagedReport(all, shown []brain.Entry, filter brain.StagedFilter) map[strin // planReport carries the totals as well as the slices. A dispatcher's question // is usually "is this slice done", and the answer is a comparison it should not // have to assemble from four counters. -func planReport(slices []brain.SliceProgress) map[string]any { +func planReport(rep brain.Report) map[string]any { var built, total int - for _, sl := range slices { + for _, sl := range rep.Slices { built += sl.Built total += sl.Total() } - return map[string]any{ - "slices": slices, + out := map[string]any{ + "slices": rep.Slices, "built": built, "total": total, } + if rep.ModelAbsent { + out["model_absent"] = true + } + return out } diff --git a/cmd/mxcli/cmd_brain_edit_test.go b/cmd/mxcli/cmd_brain_edit_test.go new file mode 100644 index 0000000000..e6bdc4e90f --- /dev/null +++ b/cmd/mxcli/cmd_brain_edit_test.go @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "os" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/cmd/mxcli/brain" +) + +// The issue's proof, through the CLI (ako/mxcli#1950): edit a promoted +// requirement, and the id is unchanged, the shard holds the new text once and +// the old nowhere, and check still passes. +func TestBrainEditKeepsTheHandleEndToEnd(t *testing.T) { + dir := t.TempDir() + runBrainForTest(t, []string{"brain", "init", "-C", dir}) + const before = "A planner can create a visit for a site with a window and one engineer" + const after = "A planner can create a visit for a site with a window and one named engineer" + + out := runBrainForTest(t, []string{"brain", "capture", before, "-a", "@Scheduling.Visit", "-s", "01-scheduling", "-C", dir}) + id := strings.Fields(out)[1] + runBrainForTest(t, []string{"brain", "promote", id, "-C", dir}) + + var edited struct { + ID string `json:"id"` + Shard string `json:"shard"` + } + out = runBrainForTest(t, []string{"brain", "edit", id, after, "-C", dir, "--json"}) + if err := json.Unmarshal([]byte(out), &edited); err != nil { + t.Fatalf("edit --json: %v\n%s", err, out) + } + if edited.ID != id || edited.Shard != brain.PlanShard("01-scheduling") { + t.Errorf("edit --json = %+v", edited) + } + + raw, err := os.ReadFile(brain.NewStore(dir).ShardPath(brain.PlanShard("01-scheduling"))) + if err != nil { + t.Fatal(err) + } + if strings.Count(string(raw), after) != 1 || strings.Contains(string(raw), "and one engineer") || !strings.Contains(string(raw), id) { + t.Errorf("shard after edit:\n%s", raw) + } + + out = runBrainForTest(t, []string{"brain", "check", "-C", dir}) + if !strings.Contains(out, "OK") { + t.Errorf("check after edit:\n%s", out) + } + + // The edited wording again is a duplicate of what is committed... + out = runBrainForTest(t, []string{"brain", "capture", after, "-a", "@Scheduling.Visit", "-s", "01-scheduling", "-C", dir}) + if !strings.Contains(out, "Already committed as "+id) { + t.Errorf("re-capturing the edited wording:\n%s", out) + } + // ...and the original wording is a new fact with an id of its own, not + // the edited entry's handle, which it hashes to. + out = runBrainForTest(t, []string{"brain", "capture", before, "-a", "@Scheduling.Visit", "-s", "01-scheduling", "-C", dir}) + fresh := strings.Fields(out)[1] + if !strings.HasPrefix(out, "Queued") || fresh == id { + t.Errorf("re-capturing the original wording:\n%s", out) + } + runBrainForTest(t, []string{"brain", "promote", fresh, "-C", dir}) + + // A staged entry is edited in the queue, in place. + out = runBrainForTest(t, []string{"brain", "capture", "Visits are planned by the planner", "-a", "@Scheduling.Visit", "-C", dir}) + staged := strings.Fields(out)[1] + out = runBrainForTest(t, []string{"brain", "edit", staged, "Visits are planned by the planner, never the engineer", "-C", dir}) + if !strings.Contains(out, "Edited staged "+staged) { + t.Errorf("editing a staged entry:\n%s", out) + } + + // -a with no text re-anchors and keeps the wording. + out = runBrainForTest(t, []string{"brain", "capture", "Visits are committed by the planner", "-a", "@Scheduling.Visit", "-C", dir}) + dec := strings.Fields(out)[1] + runBrainForTest(t, []string{"brain", "promote", dec, "-C", dir}) + out = runBrainForTest(t, []string{"brain", "edit", dec, "-a", "@Dispatch.ACT_Plan", "-C", dir}) + if !strings.Contains(out, "moved it from Scheduling.md to Dispatch.md") { + t.Errorf("re-anchor without text:\n%s", out) + } + if e, _, _ := brain.NewStore(dir).Find(dec); e.Title != "Visits are committed by the planner" { + t.Errorf("re-anchoring changed the wording to %q", e.Title) + } + + // move keeps the id and changes the file. + out = runBrainForTest(t, []string{"brain", "move", id, "--slice", "02-dispatch", "-C", dir}) + if !strings.Contains(out, "moved it from plan/01-scheduling.md to plan/02-dispatch.md") { + t.Errorf("move:\n%s", out) + } + if _, shard, _ := brain.NewStore(dir).Find(id); shard != brain.PlanShard("02-dispatch") { + t.Errorf("after move, %s is in %q", id, shard) + } +} diff --git a/cmd/mxcli/cmd_brain_json_test.go b/cmd/mxcli/cmd_brain_json_test.go index 0109788457..4c3844a022 100644 --- a/cmd/mxcli/cmd_brain_json_test.go +++ b/cmd/mxcli/cmd_brain_json_test.go @@ -11,6 +11,7 @@ import ( "time" "github.com/mendixlabs/mxcli/cmd/mxcli/brain" + "github.com/spf13/pflag" ) // The types marshalling correctly is only half of it. The other half is that @@ -105,10 +106,7 @@ func TestBrainStagedJSONIsStillJSONWhenEmpty(t *testing.T) { // and the control below (human output is NOT JSON) passes for the wrong reason. func runBrainForTest(t *testing.T, args []string) string { t.Helper() - if err := rootCmd.PersistentFlags().Set("json", "false"); err != nil { - t.Fatal(err) - } - globalJSONFlag = false + resetBrainFlagsForTest(t) out, err := captureStdout(t, func() error { rootCmd.SetArgs(args) @@ -120,6 +118,30 @@ func runBrainForTest(t *testing.T, args []string) string { return out } +// resetBrainFlagsForTest restores every brain flag, and --json, to its default. +// cobra keeps flag values between runs of the shared rootCmd, and two of the +// leftovers change the answer rather than just the noise: a -C from one run +// makes the next run's -p look like both were passed, and a StringSlice flag +// APPENDS on each Set — so a second `capture -a X` captures [X, X], a +// different entry from the first. +func resetBrainFlagsForTest(t *testing.T) { + t.Helper() + if err := rootCmd.PersistentFlags().Set("json", "false"); err != nil { + t.Fatal(err) + } + globalJSONFlag = false + for _, c := range brainLocatedCommands() { + c.Flags().VisitAll(func(f *pflag.Flag) { + if sv, ok := f.Value.(pflag.SliceValue); ok { + _ = sv.Replace(nil) + } else if err := f.Value.Set(f.DefValue); err != nil { + t.Fatalf("reset --%s: %v", f.Name, err) + } + f.Changed = false + }) + } +} + func brainTestDay() time.Time { return time.Date(2026, 9, 3, 0, 0, 0, 0, time.UTC) } // The dispatcher's loop, end to end: note the queue's last id, run the slice, diff --git a/cmd/mxcli/cmd_brain_nomodel_test.go b/cmd/mxcli/cmd_brain_nomodel_test.go new file mode 100644 index 0000000000..8b4bbca376 --- /dev/null +++ b/cmd/mxcli/cmd_brain_nomodel_test.go @@ -0,0 +1,66 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/cmd/mxcli/brain" +) + +// A brain can be filled before the project exists, and `plan` must then report +// the truth — nothing built — instead of failing to connect (ako/mxcli#1948). +// The .mpr is deliberately never created. +func TestPlanAndCheckAnswerWhenTheModelDoesNotExistYet(t *testing.T) { + dir := t.TempDir() + mpr := filepath.Join(dir, "Nothing.mpr") + + store := brain.NewStore(dir) + if _, err := store.Init(); err != nil { + t.Fatal(err) + } + req, err := brain.NewRequirement("A planner can create a visit", []string{"@Scheduling.Visit"}, "01-scheduling", brainTestDay()) + if err != nil { + t.Fatal(err) + } + if err := store.Promote(req, req.Shard()); err != nil { + t.Fatal(err) + } + + out := runBrainForTest(t, []string{"brain", "plan", "-p", mpr}) + if !strings.Contains(out, "0 of 1 requirements built") { + t.Errorf("plan without a model:\n%s", out) + } + if !strings.Contains(out, "No model") { + t.Errorf("plan must say the model is absent, not leave it to be inferred:\n%s", out) + } + + var plan struct { + Built int `json:"built"` + Total int `json:"total"` + ModelAbsent bool `json:"model_absent"` + } + out = runBrainForTest(t, []string{"brain", "plan", "-p", mpr, "--json"}) + if err := json.Unmarshal([]byte(out), &plan); err != nil { + t.Fatalf("plan --json: %v\n%s", err, out) + } + if plan.Built != 0 || plan.Total != 1 || !plan.ModelAbsent { + t.Errorf("plan --json = %+v, want built 0 of 1 with model_absent", plan) + } + + var check struct { + Failed bool `json:"failed"` + Unverifiable int `json:"unverifiable"` + ModelAbsent bool `json:"model_absent"` + } + out = runBrainForTest(t, []string{"brain", "check", "-p", mpr, "--json"}) + if err := json.Unmarshal([]byte(out), &check); err != nil { + t.Fatalf("check --json: %v\n%s", err, out) + } + if check.Failed || check.Unverifiable != 1 || !check.ModelAbsent { + t.Errorf("check --json = %+v, want not failed, 1 unverifiable, model_absent", check) + } +} diff --git a/cmd/mxcli/cmd_brain_source_test.go b/cmd/mxcli/cmd_brain_source_test.go new file mode 100644 index 0000000000..54c9a03cc5 --- /dev/null +++ b/cmd/mxcli/cmd_brain_source_test.go @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/cmd/mxcli/brain" +) + +// ako/mxcli#1951 through the CLI: --source is stored relative to the project, +// rendered into the shard, present in the JSON, and kept by an edit. +func TestCaptureRecordsWhereARequirementCameFrom(t *testing.T) { + dir := t.TempDir() + prd := filepath.Join(dir, "docs", "sample-prd.md") + if err := os.MkdirAll(filepath.Dir(prd), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(prd, []byte("# PRD\n"), 0o644); err != nil { + t.Fatal(err) + } + runBrainForTest(t, []string{"brain", "init", "-C", dir}) + + // An absolute path in, a project-relative one stored: the shard must mean + // the same thing on another machine. + out := runBrainForTest(t, []string{"brain", "capture", "A planner can create a visit", + "-a", "@Scheduling.Visit", "-s", "01-scheduling", "--source", prd + "#L18-21", "-C", dir}) + id := strings.Fields(out)[1] + + var staged struct { + Staged []struct { + ID string `json:"id"` + Source string `json:"source"` + } `json:"staged"` + } + out = runBrainForTest(t, []string{"brain", "staged", "-C", dir, "--json"}) + if err := json.Unmarshal([]byte(out), &staged); err != nil { + t.Fatal(err) + } + if len(staged.Staged) != 1 || staged.Staged[0].Source != "docs/sample-prd.md#L18-21" { + t.Fatalf("staged --json: %s", out) + } + + runBrainForTest(t, []string{"brain", "promote", id, "-C", dir}) + shard := brain.NewStore(dir).ShardPath(brain.PlanShard("01-scheduling")) + raw, err := os.ReadFile(shard) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(raw), "Source: `docs/sample-prd.md#L18-21`") || strings.Contains(string(raw), dir) { + t.Errorf("shard:\n%s", raw) + } + if out := runBrainForTest(t, []string{"brain", "check", "-C", dir}); !strings.Contains(out, "OK") { + t.Errorf("check with the source present:\n%s", out) + } + + // An edit of the wording keeps the source; --source replaces it. + runBrainForTest(t, []string{"brain", "edit", id, "A planner can create a visit for one engineer", "-C", dir}) + if e, _, _ := brain.NewStore(dir).Find(id); e.Source != "docs/sample-prd.md#L18-21" { + t.Errorf("edit dropped the source: %q", e.Source) + } + runBrainForTest(t, []string{"brain", "edit", id, "--source", prd + "#Goals", "-C", dir}) + if e, _, _ := brain.NewStore(dir).Find(id); e.Source != "docs/sample-prd.md#Goals" || e.Title != "A planner can create a visit for one engineer" { + t.Errorf("edit --source: %+v", e) + } +} diff --git a/cmd/mxcli/cmd_new.go b/cmd/mxcli/cmd_new.go index 08d5312cae..180432c2ab 100644 --- a/cmd/mxcli/cmd_new.go +++ b/cmd/mxcli/cmd_new.go @@ -36,6 +36,10 @@ Step 4 exists because Atlas's layouts live in a Marketplace module, and Mendix's own guidance is not to edit them: an update replaces the module and the edit is gone. A generated app therefore starts with a layout it can change. +The target directory must be empty, or hold only a brain (docs/brain/ and +.mxcli/brain/, from 'mxcli brain init -C '). A brain is adopted: the project +is created around it and its requirements are counted against the new model. + The project is created in a short temporary directory and moved into place, so --output-dir may be as deep as your filesystem allows and a failure never leaves partial output behind. Mendix tooling itself refuses paths over 259 characters, @@ -85,11 +89,16 @@ Examples: os.Exit(1) } - // Check if directory already exists and has content - if entries, err := os.ReadDir(absDir); err == nil && len(entries) > 0 { - fmt.Fprintf(os.Stderr, "Error: directory %s already exists and is not empty\n", absDir) + // The directory must be empty, or hold only a brain started before the + // project (requirements first, app second), which is adopted in place. + adoptBrain, err := checkNewProjectDestination(absDir) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) os.Exit(1) } + if adoptBrain { + fmt.Printf("Adopting the brain already in %s; the project is created around it.\n", absDir) + } // A hand-typed version may name no CDN artifact: Mendix 9 and 10 publish a // build number the release notes never mention, so the release called diff --git a/cmd/mxcli/help.go b/cmd/mxcli/help.go index 6916d3ad19..6542c82e7a 100644 --- a/cmd/mxcli/help.go +++ b/cmd/mxcli/help.go @@ -39,6 +39,7 @@ Top-level topics: structure - SHOW STRUCTURE command move - MOVE command for relocating documents search - Full-text SEARCH command + features - LIST FEATURES: what a Mendix version supports, and since when Examples: mxcli syntax --json # Full index (LLM: cache this) diff --git a/cmd/mxcli/mcp_exec_no_file_write_test.go b/cmd/mxcli/mcp_exec_no_file_write_test.go new file mode 100644 index 0000000000..06985605a5 --- /dev/null +++ b/cmd/mxcli/mcp_exec_no_file_write_test.go @@ -0,0 +1,174 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "crypto/sha256" + "encoding/hex" + "io/fs" + "os" + "path/filepath" + "sort" + "strings" + "testing" +) + +// mendixlabs/mxcli#1684 (ako/mxcli#1833): "`mxcli exec --mcp ` +// and `mxcli -c \"...\" --mcp ` do not send the writes through Studio Pro's +// MCP API. The command exits 0, prints `connect local '.mpr'` and +// `Created constant: ...`, makes no `ped_*` tool call ... and writes the `.mpr` +// and `mprcontents/` on disk while Studio Pro has the app open." and "With +// `--mcp` given, mxcli also skips its own refusal while `.mpr.lock` exists." +// +// These run the real main() in a child process (runMainStreams), with the flags +// exactly as the report gives them, against a fake Studio Pro and a project +// whose .mpr.lock is present — Studio Pro has it open. The contract: +// a statement with a PED route reaches Studio Pro, one without is refused, +// and in neither case does a byte of the .mpr or mprcontents/ change. The +// controls at the bottom show the hash detects a write and the lock guard is +// live in the same setup, so a pass here is not an inert comparison. + +const mcpProbeConstant = "create constant MyFirstModule.LiveProbe (Type: String, DefaultValue: 'x');" + +// mcpLockedProject copies the v2 fixture into a fresh directory and puts +// Studio Pro's lock file beside it (lower-cased, as Studio Pro names it). +func mcpLockedProject(t *testing.T, lock bool) string { + t.Helper() + dst := t.TempDir() + if err := os.CopyFS(dst, os.DirFS("../../testdata/expr-checker")); err != nil { + t.Fatalf("copy fixture: %v", err) + } + if lock { + if err := os.WriteFile(filepath.Join(dst, "minimal.mpr.lock"), nil, 0o644); err != nil { + t.Fatal(err) + } + } + return filepath.Join(dst, "minimal.mpr") +} + +// modelFingerprint hashes the .mpr and every file under mprcontents/ — the +// files Studio Pro owns while the project is open. +func modelFingerprint(t *testing.T, mpr string) string { + t.Helper() + files := []string{mpr} + root := filepath.Join(filepath.Dir(mpr), "mprcontents") + _ = filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error { + if err == nil && !d.IsDir() { + files = append(files, p) + } + return nil + }) + sort.Strings(files) + h := sha256.New() + for _, f := range files { + data, err := os.ReadFile(f) + if err != nil { + t.Fatalf("read %s: %v", f, err) + } + sum := sha256.Sum256(data) + h.Write([]byte(f + "\x00" + hex.EncodeToString(sum[:]) + "\n")) + } + return hex.EncodeToString(h.Sum(nil)) +} + +func TestMCP_ExecAndOneLinerNeverWriteTheProjectStudioProHasOpen(t *testing.T) { + if testing.Short() { + t.Skip("spawns mxcli against a project") + } + t.Setenv("MXCLI_ALLOW_STUDIO_PRO_OPEN", "") + + cases := []struct { + name string + script bool // exec rather than -c + stmt string // the one statement run + routed bool // true: has a PED route; false: must be refused + }{ + {"exec, create constant goes through PED", true, mcpProbeConstant, true}, + {"-c, create constant goes through PED", false, mcpProbeConstant, true}, + {"exec, a statement with no PED route is refused", true, "create image collection MyFirstModule.LiveProbeImages;", false}, + {"-c, a statement with no PED route is refused", false, "create image collection MyFirstModule.LiveProbeImages;", false}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + srv, calls := fakeStudioPro(t) + mpr := mcpLockedProject(t, true) + dir := filepath.Dir(mpr) + before := modelFingerprint(t, mpr) + + args := []string{"-p", mpr} + if c.script { + script := filepath.Join(t.TempDir(), "probe.mdl") + if err := os.WriteFile(script, []byte(c.stmt+"\n"), 0o644); err != nil { + t.Fatal(err) + } + args = append([]string{"exec", script}, args...) + } else { + args = append(args, "-c", c.stmt) + } + args = append(args, "--mcp", srv.URL+"/mcp", "--mcp-dial", strings.TrimPrefix(srv.URL, "http://"), "--mcp-trace") + res := runMainStreams(t, dir, args...) + got := calls() + + if after := modelFingerprint(t, mpr); after != before { + t.Errorf("the .mpr / mprcontents/ changed on disk with --mcp while Studio Pro has the project open "+ + "(exit %d, PED calls %v)\nstdout:\n%s\nstderr:\n%s", res.exitCode, got, res.stdout, res.stderr) + } + if c.routed { + if res.exitCode != 0 { + t.Fatalf("exit %d, want 0\nstdout:\n%s\nstderr:\n%s", res.exitCode, res.stdout, res.stderr) + } + if !contains(got, "ped_create_document") { + t.Errorf("Studio Pro received %v, want a ped_create_document: the write did not go through PED", got) + } + if !strings.Contains(res.stderr, "→ ped_create_document") { + t.Errorf("--mcp-trace lists no ped_create_document call:\n%s", res.stderr) + } + return + } + if res.exitCode == 0 { + t.Errorf("a statement with no PED route exited 0 under --mcp; it must be refused\nstdout:\n%s\nstderr:\n%s", + res.stdout, res.stderr) + } + if !strings.Contains(res.stderr, "not supported by the MCP backend") { + t.Errorf("the refusal does not say the statement has no MCP route:\n%s", res.stderr) + } + if contains(got, "ped_create_document") { + t.Errorf("Studio Pro received a create for a statement that was refused: %v", got) + } + }) + } +} + +// Controls for the test above, in the same setup without --mcp. The first shows +// the fingerprint does detect this write (else "unchanged" proves nothing); the +// second that the lock guard fires here, so the --mcp runs above reached the +// project with the lock in place. +func TestMCP_ControlsWithoutMCP(t *testing.T) { + if testing.Short() { + t.Skip("spawns mxcli against a project") + } + t.Setenv("MXCLI_ALLOW_STUDIO_PRO_OPEN", "") + + t.Run("no lock: the write lands and the fingerprint sees it", func(t *testing.T) { + mpr := mcpLockedProject(t, false) + before := modelFingerprint(t, mpr) + res := runMainStreams(t, filepath.Dir(mpr), "-p", mpr, "-c", mcpProbeConstant) + if res.exitCode != 0 { + t.Fatalf("exit %d\nstdout:\n%s\nstderr:\n%s", res.exitCode, res.stdout, res.stderr) + } + if modelFingerprint(t, mpr) == before { + t.Fatal("a local write left the fingerprint unchanged — the --mcp test above could not detect a file write") + } + }) + t.Run("lock present: the local write is refused", func(t *testing.T) { + mpr := mcpLockedProject(t, true) + before := modelFingerprint(t, mpr) + res := runMainStreams(t, filepath.Dir(mpr), "-p", mpr, "-c", mcpProbeConstant) + if res.exitCode == 0 || !strings.Contains(res.stderr, "Studio Pro has this project open") { + t.Fatalf("the .mpr.lock refusal did not fire (exit %d):\n%s", res.exitCode, res.stderr) + } + if modelFingerprint(t, mpr) != before { + t.Fatal("the refused write changed the project") + } + }) +} diff --git a/cmd/mxcli/newproject_paths.go b/cmd/mxcli/newproject_paths.go index 43ac77f496..4177d3e93c 100644 --- a/cmd/mxcli/newproject_paths.go +++ b/cmd/mxcli/newproject_paths.go @@ -3,10 +3,13 @@ package main import ( + "errors" "fmt" "io" "os" "path/filepath" + + "github.com/mendixlabs/mxcli/cmd/mxcli/brain" ) // MxToolset refuses to extract a file whose full destination path exceeds this, @@ -101,7 +104,15 @@ func warnIfPathTooLongForStudioPro(w io.Writer, dest string, longest int, longes // moveProject moves a created project from the staging directory into dest. // Falls back to copy-then-remove when the two are on different filesystems, // which os.Rename cannot cross. +// +// dest is empty, or holds only an adopted brain (checkNewProjectDestination). +// In the second case the project is merged around the brain, and every +// conflict is found before anything moves, so a refusal leaves both sides as +// they were. func moveProject(stage, dest string) error { + if entries, err := os.ReadDir(dest); err == nil && len(entries) > 0 { + return mergeProject(stage, dest) + } if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil { return fmt.Errorf("create parent directory: %w", err) } @@ -139,3 +150,162 @@ func describeCreateProjectFailure(stage string, runErr error) error { } return runErr } + +// checkNewProjectDestination decides whether `mxcli new` may create a project +// in dir. An empty or missing directory is fine. So is one holding only a brain +// — docs/brain/ written by mxcli, and its queue in .mxcli/brain/ — which is +// adopted: requirements come first and the app second, so the brain is born +// before the project and the project moves in around it (ako/mxcli#1949). +// +// Deliberately that narrow. Anything else in the directory is somebody's files, +// and merging a generated project into them is not what "new" means. +func checkNewProjectDestination(dir string) (adopt bool, err error) { + entries, err := os.ReadDir(dir) + if os.IsNotExist(err) { + return false, nil + } + if err != nil { + return false, err + } + if len(entries) == 0 { + return false, nil + } + notEmpty := func(what string) error { + return fmt.Errorf("directory %s already exists and is not empty (%s); "+ + "only a brain — docs/brain/ and .mxcli/brain/ — is adopted", dir, what) + } + for _, e := range entries { + if e.Name() != "docs" && e.Name() != ".mxcli" { + return false, notEmpty(e.Name()) + } + if !e.IsDir() { + return false, notEmpty(e.Name()) + } + inner, err := os.ReadDir(filepath.Join(dir, e.Name())) + if err != nil { + return false, err + } + for _, in := range inner { + if in.Name() != "brain" || !in.IsDir() { + return false, notEmpty(filepath.Join(e.Name(), in.Name())) + } + } + } + // The same refusal `brain init` makes: a docs/brain/ without mxcli's + // marker is someone's own notes, not a brain to adopt. + store := brain.NewStore(dir) + if info, err := os.Stat(store.Root); err == nil && info.IsDir() && !store.Exists() { + return false, fmt.Errorf("%w: %s", brain.ErrForeignStore, store.Root) + } + if store.Exists() { + readme, err := os.ReadFile(filepath.Join(store.Root, "README.md")) + if err != nil { + return false, err + } + if !brain.HasMarker(string(readme)) { + return false, fmt.Errorf("%w: %s", brain.ErrForeignStore, store.Root) + } + } + return true, nil +} + +// mergeProject moves stage's tree into a dest that already has content. +// Directories present on both sides are merged; any FILE present on both sides +// is a conflict, and all of them are reported before anything has moved. +func mergeProject(stage, dest string) error { + var conflicts []string + err := filepath.WalkDir(stage, func(p string, d os.DirEntry, err error) error { + if err != nil || p == stage { + return err + } + rel, _ := filepath.Rel(stage, p) + existing, statErr := os.Lstat(filepath.Join(dest, rel)) + if statErr != nil { + return nil // not in dest: no conflict here or below + } + if d.IsDir() && existing.IsDir() { + return nil // merge, keep looking below + } + conflicts = append(conflicts, rel) + if d.IsDir() { + return filepath.SkipDir + } + return nil + }) + if err != nil { + return err + } + if len(conflicts) > 0 { + return fmt.Errorf("the project would overwrite %s already in %s", joinPaths(conflicts), dest) + } + if err := moveInto(stage, dest); err != nil { + return err + } + return os.RemoveAll(stage) +} + +// moveInto moves each entry of src into dst, descending only into directories +// that exist on both sides — everything else moves whole. +func moveInto(src, dst string) error { + entries, err := os.ReadDir(src) + if err != nil { + return err + } + for _, e := range entries { + s, d := filepath.Join(src, e.Name()), filepath.Join(dst, e.Name()) + if info, err := os.Lstat(d); err == nil && info.IsDir() { + if err := moveInto(s, d); err != nil { + return err + } + continue + } + if err := os.Rename(s, d); err == nil { + continue + } + // Cross-device: copy this entry, then let the caller drop the stage. + if e.IsDir() { + if err := os.CopyFS(d, os.DirFS(s)); err != nil { + return fmt.Errorf("copy %s into place: %w", e.Name(), err) + } + continue + } + if err := copyFileNoClobber(s, d); err != nil { + return fmt.Errorf("copy %s into place: %w", e.Name(), err) + } + } + return nil +} + +func copyFileNoClobber(src, dst string) (err error) { + in, err := os.Open(src) + if err != nil { + return err + } + defer in.Close() + info, err := in.Stat() + if err != nil { + return err + } + out, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_EXCL, info.Mode().Perm()) + if err != nil { + return err + } + defer func() { err = errors.Join(err, out.Close()) }() + _, err = io.Copy(out, in) + return err +} + +func joinPaths(paths []string) string { + const show = 5 + out := "" + for i, p := range paths { + if i == show { + return fmt.Sprintf("%s and %d more", out, len(paths)-show) + } + if i > 0 { + out += ", " + } + out += p + } + return out +} diff --git a/cmd/mxcli/syntax/drop_statements_drift_test.go b/cmd/mxcli/syntax/drop_statements_drift_test.go new file mode 100644 index 0000000000..1f3f949570 --- /dev/null +++ b/cmd/mxcli/syntax/drop_statements_drift_test.go @@ -0,0 +1,118 @@ +// SPDX-License-Identifier: Apache-2.0 + +package syntax + +import ( + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +// mendixlabs/mxcli#1716 (ako/mxcli#1911): `DROP ASSOCIATION Module.Name;` parses +// and executes, but a reader looking for it found nothing: `mxcli syntax` lists no +// drop topic, and `mxcli syntax drop` answered with three unrelated topics that +// merely had "drop" in a path segment. DROP ENTITY, DROP MICROFLOW, DROP NANOFLOW +// and DROP SNIPPET were documented nowhere at all. +// +// The grammar's dropStatement rule is the authority: every statement it accepts +// must be spelled out in the `drop` topic, so a DROP added to the parser cannot +// ship undocumented. Same shape as widget_keywords_drift_test.go. + +// dropKeywordRules maps the grammar's shared keyword sub-rules to the canonical +// spelling — the first alternative; the others are deprecated aliases. +var dropKeywordRules = map[string]string{ + "taskQueueKw": "TASK QUEUE", + "consumedODataServiceKw": "CONSUMED ODATA SERVICE", + "publishedODataServiceKw": "PUBLISHED ODATA SERVICE", + "consumedRestServiceKw": "CONSUMED REST SERVICE", + "aiModelKw": "AI MODEL", +} + +// dropStatementForms reads the DROP head of every alternative of the +// dropStatement rule in the committed grammar. +func dropStatementForms(t *testing.T) []string { + t.Helper() + path := filepath.Join("..", "..", "..", "mdl", "grammar", "MDLParser.g4") + b, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + rule := regexp.MustCompile(`(?s)\ndropStatement\s*\n\s*:(.*?)\n\s*;`).FindStringSubmatch(string(b)) + if rule == nil { + t.Fatal("dropStatement rule not found — if it was renamed, re-point this guard rather than deleting it") + } + var out []string + for _, line := range strings.Split(rule[1], "\n") { + if i := strings.Index(line, "//"); i >= 0 { + line = line[:i] + } + fields := strings.Fields(strings.TrimPrefix(strings.TrimSpace(line), "|")) + if len(fields) == 0 || fields[0] != "DROP" { + continue + } + words := []string{"DROP"} + for _, f := range fields[1:] { + if kw, ok := dropKeywordRules[f]; ok { + words = append(words, kw) + continue + } + if f != strings.ToUpper(f) || strings.Contains(f, "LITERAL") || f == "AT_KW" { + break // ifExists?, qualifiedName, a literal: the head is complete + } + words = append(words, strings.TrimSuffix(f, "_KW")) + } + if len(words) == 1 { + t.Fatalf("dropStatement alternative with an unknown keyword rule: %q — add it to dropKeywordRules", line) + } + out = append(out, strings.Join(words, " ")) + } + if len(out) < 30 { + t.Fatalf("read only %d dropStatement alternatives — the parse of the rule is broken", len(out)) + } + return out +} + +func TestDropTopicListsEveryDropStatement(t *testing.T) { + f := ByPath("drop") + if f == nil { + t.Fatal(`no "drop" topic: DROP statements are not discoverable from mxcli syntax (#1716)`) + } + syntax := strings.ToUpper(f.Syntax) + for _, form := range dropStatementForms(t) { + if !strings.Contains(syntax, form+" ") { + t.Errorf("the grammar accepts %q but the drop topic does not list it", form) + } + } +} + +// The reported lookups: `mxcli syntax drop` must land on the topic, and +// `mxcli syntax drop association` must reach a page that spells the statement. +func TestDropLookupFindsTheStatement(t *testing.T) { + m := Lookup([]string{"drop"}) + if !m.Exact || len(m.Features) == 0 || m.Features[0].Path != "drop" { + t.Fatalf("syntax drop: got exact=%v %v, want the drop topic", m.Exact, paths(m.Features)) + } + for _, q := range [][]string{{"drop", "association"}, {"drop", "entity"}, {"drop", "microflow"}} { + m := Lookup(q) + want := "DROP " + strings.ToUpper(q[1]) + " " + found := false + for _, f := range m.Features { + if strings.Contains(strings.ToUpper(f.Syntax), want) { + found = true + } + } + if !found { + t.Errorf("syntax %s: no topic among %v spells %q", strings.Join(q, " "), paths(m.Features), want) + } + } +} + +func paths(fs []SyntaxFeature) []string { + var out []string + for _, f := range fs { + out = append(out, f.Path) + } + return out +} diff --git a/cmd/mxcli/syntax/expression_functions_test.go b/cmd/mxcli/syntax/expression_functions_test.go new file mode 100644 index 0000000000..5a90b76820 --- /dev/null +++ b/cmd/mxcli/syntax/expression_functions_test.go @@ -0,0 +1,56 @@ +// SPDX-License-Identifier: Apache-2.0 + +package syntax + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/exprcheck" +) + +// mendixlabs/mxcli#1708 (ako/mxcli#1866): "the expression reference has no +// entries for isNew, the addDays/addMonths family or trimToDays/trimToMonths". +// `mxcli syntax expression` must reach a page that names them, with types. +func TestExpressionReferenceNamesReportedFunctions(t *testing.T) { + m := Lookup([]string{"expression"}) + var text strings.Builder + for _, f := range m.Features { + text.WriteString(f.Syntax) + } + for _, want := range []string{ + "isNew(Any) -> Boolean", + "addDays(DateTime, Integer) -> DateTime", + "addMonths(DateTime, Integer) -> DateTime", + "trimToDays(DateTime) -> DateTime", + "trimToMonths(DateTime) -> DateTime", + } { + if !strings.Contains(text.String(), want) { + t.Errorf("`mxcli syntax expression` (topics %v) does not list %q", paths(m.Features), want) + } + } +} + +// Every function the type checker accepts is in the reference, and nothing it +// does not accept: the page is rendered from exprcheck's table, so this guards +// the rendering, not a hand-kept list. +func TestExpressionFunctionReferenceMatchesChecker(t *testing.T) { + f := ByPath("microflow.expressions.functions") + if f == nil { + t.Fatal("no microflow.expressions.functions topic") + } + fns := exprcheck.Functions() + if len(fns) < 80 { + t.Fatalf("exprcheck.Functions() returned %d rows — expected the full built-in table", len(fns)) + } + for _, fn := range fns { + if !strings.Contains(f.Syntax, " "+fn.Name+"(") { + t.Errorf("checker accepts %s() but the reference does not list it", fn.Name) + } + } + // Rows nobody has built against mxbuild are CE0117 when measured; listing + // them as functions would send readers straight into a build failure. + if strings.Contains(f.Syntax, " dayOfYear(") || strings.Contains(f.Syntax, " hour(") { + t.Error("reference lists an unverified, build-rejected function (dayOfYear/hour)") + } +} diff --git a/cmd/mxcli/syntax/features_domain_model.go b/cmd/mxcli/syntax/features_domain_model.go index 08768d22a1..1094697294 100644 --- a/cmd/mxcli/syntax/features_domain_model.go +++ b/cmd/mxcli/syntax/features_domain_model.go @@ -63,8 +63,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "domain-model.entity", - Summary: "Entity creation: persistent, non-persistent, generalization, event handlers", + Path: "domain-model.entity", + Features: []string{"domain_model/entities", "domain_model/non_persistent_entities", "domain_model/entity_generalization"}, + Summary: "Entity creation: persistent, non-persistent, generalization, event handlers", Keywords: []string{ "entity", "create entity", "persistent", "non-persistent", "generalization", "extends", "event handler", "attribute", @@ -82,14 +83,15 @@ func init() { "non-persistent", "extends", "generalization", "index", "event handler", "before commit", "after commit", }, - Syntax: "CREATE PERSISTENT ENTITY Module.Name (\n Attr: Type [NOT NULL [ERROR MESSAGE 'msg']] [UNIQUE [ERROR MESSAGE 'msg']] [DEFAULT val],\n ...\n)\n[INDEX (attr1, attr2)]\n[ON BEFORE|AFTER CREATE|COMMIT|DELETE|ROLLBACK CALL Module.MF [RAISE ERROR]]\n[EXPORT LEVEL API|HIDDEN]; -- also per attribute: `Attr: Type EXPORT LEVEL API`\n\n-- An AutoNumber attribute needs its start value: `Nr: AutoNumber DEFAULT 1`.\n-- Without the DEFAULT seed, check reports MDL023 and the build fails CE7247.\n\n-- Documentation: the /** … */ doc comment before the statement, or\n-- ALTER ENTITY Module.Name SET DOCUMENTATION 'text' on an existing entity.\n\nCREATE NON-PERSISTENT ENTITY Module.Name (...);\nCREATE PERSISTENT ENTITY Module.Name EXTENDS Module.Parent (...);\n\n-- INDEX goes AFTER the closing parenthesis, never inside the attribute list.\n-- On an existing entity, either spelling works:\nALTER ENTITY Module.Name ADD INDEX [IF NOT EXISTS] [name] [ON] (attr1 [ASC|DESC], ...);\nCREATE INDEX IdxName ON Module.Name (attr1 [ASC|DESC], ...);\n\n-- Re-runnable script: IF NOT EXISTS skips instead of erroring, and leaves an\n-- existing element untouched (unlike OR MODIFY, which rebuilds it from the\n-- statement and drops any attribute the statement omits).\nCREATE ENTITY IF NOT EXISTS Module.Name (...);\nALTER ENTITY Module.Name ADD ATTRIBUTE IF NOT EXISTS Attr: Type;\nALTER ENTITY Module.Name DROP INDEX IF EXISTS (attr1, ...);", + Syntax: "CREATE PERSISTENT ENTITY Module.Name (\n Attr: Type [NOT NULL [ERROR MESSAGE 'msg']] [UNIQUE [ERROR MESSAGE 'msg']] [DEFAULT val],\n ...\n)\n[INDEX (attr1, attr2)]\n[ON BEFORE|AFTER CREATE|COMMIT|DELETE|ROLLBACK CALL Module.MF [RAISE ERROR]]\n[EXPORT LEVEL API|HIDDEN]; -- also per attribute: `Attr: Type EXPORT LEVEL API`\n\n-- An AutoNumber attribute needs its start value: `Nr: AutoNumber DEFAULT 1`.\n-- Without the DEFAULT seed, check reports MDL023 and the build fails CE7247.\n\n-- Documentation: the /** … */ doc comment before the statement, or\n-- ALTER ENTITY Module.Name SET DOCUMENTATION 'text' on an existing entity.\n\nCREATE NON-PERSISTENT ENTITY Module.Name (...);\nCREATE PERSISTENT ENTITY Module.Name EXTENDS Module.Parent (...);\n\nDROP ENTITY [IF EXISTS] Module.Name; -- every DROP: see `mxcli syntax drop`\n\n-- INDEX goes AFTER the closing parenthesis, never inside the attribute list.\n-- On an existing entity, either spelling works:\nALTER ENTITY Module.Name ADD INDEX [IF NOT EXISTS] [name] [ON] (attr1 [ASC|DESC], ...);\nCREATE INDEX IdxName ON Module.Name (attr1 [ASC|DESC], ...);\n\n-- Re-runnable script: IF NOT EXISTS skips instead of erroring, and leaves an\n-- existing element untouched (unlike OR MODIFY, which rebuilds it from the\n-- statement and drops any attribute the statement omits).\nCREATE ENTITY IF NOT EXISTS Module.Name (...);\nALTER ENTITY Module.Name ADD ATTRIBUTE IF NOT EXISTS Attr: Type;\nALTER ENTITY Module.Name DROP INDEX IF EXISTS (attr1, ...);", Example: "mdl 1;\n-- Persistent with constraints and index\nCREATE PERSISTENT ENTITY Shop.Order (\n OrderNumber: String(20) NOT NULL,\n Total: Decimal DEFAULT 0,\n CreatedAt: DateTime\n)\nINDEX (OrderNumber)\nON BEFORE COMMIT CALL Shop.ValidateOrder($currentObject) RAISE ERROR;\n\n-- With generalization\nCREATE PERSISTENT ENTITY Shop.ProductImage EXTENDS System.Image (\n Caption: String(200)\n);", SeeAlso: []string{"domain-model.entity.alter", "domain-model.entity.attributes", "domain-model.export-level"}, }) Register(SyntaxFeature{ - Path: "domain-model.entity.alter", - Summary: "ALTER ENTITY: add/rename/modify/drop attributes, indexes, documentation, event handlers; ALTER ENTITIES for the bulk add", + Path: "domain-model.entity.alter", + Features: []string{"domain_model/alter_entity"}, + Summary: "ALTER ENTITY: add/rename/modify/drop attributes, indexes, documentation, event handlers; ALTER ENTITIES for the bulk add", Keywords: []string{ "alter entity", "modify entity", "add attribute", "drop attribute", "rename attribute", "add index", @@ -115,8 +117,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "domain-model.entity.attributes", - Summary: "Attribute data types, constraints, system attributes, and calculated attributes", + Path: "domain-model.entity.attributes", + Features: []string{"domain_model/calculated_attributes"}, + Summary: "Attribute data types, constraints, system attributes, and calculated attributes", Keywords: []string{ "attribute", "data type", "string", "integer", "decimal", "boolean", "datetime", "autonumber", "binary", "hashedstring", @@ -146,7 +149,7 @@ func init() { "group by", "sum", "count", "report", "dashboard", "totals", "create view entity", "query performance", "read model", }, - MinVersion: "10.18.0", + Features: []string{"domain_model/view_entities"}, Syntax: "CREATE VIEW ENTITY [IF NOT EXISTS] Module.Name (\n" + " Attr: Type,\n" + " ...\n" + @@ -185,7 +188,7 @@ func init() { "MDL030", "MDL031", "MDL072", "CE0174", "CE6770", "string_agg", "aggregate functions", }, - MinVersion: "10.18.0", + Features: []string{"domain_model/view_entities"}, Syntax: "BOTH CLAUSE ORDERS ARE ACCEPTED — `select … from …` and Mendix's own\n" + "from-first `from … join … group by … select …`.\n\n" + "An association is walked with SLASHES, never dots:\n" + @@ -247,7 +250,7 @@ func init() { "view entity association", "oql association", "select id as", "OqlViewAssociationSource", "CE6771", "CE6770", "MDL080", }, - MinVersion: "10.18.0", + Features: []string{"domain_model/view_entities"}, Syntax: "SELECTING A PERSISTENT ENTITY'S ID UNDER AN ALIAS CREATES AN ASSOCIATION,\n" + "named after the alias. There is no second statement, and the id column is\n" + "NOT one of the view entity's attributes:\n\n" + @@ -417,8 +420,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "domain-model.export-level", - Summary: "EXPORT LEVEL API|HIDDEN on entities, attributes, associations and enumerations (Protected / add-on modules)", + Path: "domain-model.export-level", + Features: []string{"domain_model/export_levels"}, + Summary: "EXPORT LEVEL API|HIDDEN on entities, attributes, associations and enumerations (Protected / add-on modules)", Keywords: []string{ "export level", "usable", "hidden", "api", "protected module", "add-on module", "addon", "CE5400", "CE1406", "public surface", diff --git a/cmd/mxcli/syntax/features_expression_functions.go b/cmd/mxcli/syntax/features_expression_functions.go new file mode 100644 index 0000000000..cc9b3d6509 --- /dev/null +++ b/cmd/mxcli/syntax/features_expression_functions.go @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: Apache-2.0 + +package syntax + +import ( + "strings" + + "github.com/mendixlabs/mxcli/mdl/exprcheck" +) + +// The function reference is rendered from the type checker's own table rather +// than written by hand. A hand-kept list had drifted: isNew, the add*/subtract* +// family and the trimTo* functions were all accepted by `check` and named +// nowhere in `mxcli syntax` (mendixlabs/mxcli#1708). Rendering it means a row +// added to exprcheck is documented in the same change, with the types the +// checker enforces. +func init() { + Register(SyntaxFeature{ + Path: "microflow.expressions.functions", + Summary: "Every built-in expression function, with argument and result types", + Keywords: []string{ + "function", "functions", "built-in", "builtin", "expression function", + "isNew", "addDays", "addMonths", "addYears", "subtractDays", "trimToDays", + "trimToMonths", "beginOfDay", "endOfMonth", "dateTime", "formatDateTime", + "parseDateTime", "formatDecimal", "parseInteger", "toString", "getCaption", + "date functions", "string functions", "math functions", + }, + Syntax: renderFunctionReference(exprcheck.Functions()), + Example: "mdl 1;\ncreate microflow Mod.DueDate ($Order: Mod.Order) returns DateTime\nbegin\n if isNew($Order) then\n return addMonths(trimToDays([%CurrentDateTime%]), 1);\n end if;\n return endOfMonth($Order/OrderDate);\nend;", + SeeAlso: []string{"microflow.expressions", "microflow.variables"}, + }) +} + +var functionGroups = []struct { + title string + match func(exprcheck.FunctionSignature) bool +}{ + {"Date and time", func(f exprcheck.FunctionSignature) bool { + return f.Return == "DateTime" || (len(f.Params) > 0 && f.Params[0] == "DateTime") + }}, + {"Objects and enumerations", func(f exprcheck.FunctionSignature) bool { + return len(f.Params) > 0 && f.Params[0] == "Any" && f.Name != "toString" + }}, + {"Strings and conversion", func(f exprcheck.FunctionSignature) bool { + return f.Return == "String" || (len(f.Params) > 0 && f.Params[0] == "String") + }}, + {"Numbers", func(f exprcheck.FunctionSignature) bool { return true }}, +} + +func renderFunctionReference(fns []exprcheck.FunctionSignature) string { + var b strings.Builder + b.WriteString("-- Built-in functions accepted by `mxcli check`, with the types it\n" + + "-- enforces: a wrong argument kind is E005 (CE0117 at build), a wrong\n" + + "-- argument count E006, an unknown name MDL044. [x] marks an optional\n" + + "-- argument; *UTC variants use the UTC calendar instead of the user's.\n" + + "-- The current time is the token [%CurrentDateTime%], not a function.\n" + + "-- There are no hour(), dayOfWeek(), year() or month() functions: format the\n" + + "-- date instead, e.g. parseInteger(formatDateTime($d, 'yyyy')).\n") + done := map[string]bool{} + for _, g := range functionGroups { + var lines []string + for _, f := range fns { + if done[f.Name] || !g.match(f) { + continue + } + done[f.Name] = true + line := " " + f.String() + if f.ReturnNote != "" { + line += " -- " + f.ReturnNote + } + lines = append(lines, line) + } + if len(lines) == 0 { + continue + } + b.WriteString("\n-- " + g.title + "\n") + b.WriteString(strings.Join(lines, "\n")) + b.WriteByte('\n') + } + b.WriteString("\n-- *Between returns the ABSOLUTE difference, as a Decimal (calendar*Between\n" + + "-- an Integer, millisecondsBetween a Long): see microflow.expressions.\n") + return b.String() +} diff --git a/cmd/mxcli/syntax/features_integration.go b/cmd/mxcli/syntax/features_integration.go index a931925057..2892fb51ae 100644 --- a/cmd/mxcli/syntax/features_integration.go +++ b/cmd/mxcli/syntax/features_integration.go @@ -32,8 +32,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "odata.consume", - Summary: "Create consumed OData services and external entities", + Path: "odata.consume", + Features: []string{"integration/odata_client", "microflows/call_external_action", "microflows/call_external_action_arguments"}, + Summary: "Create consumed OData services and external entities", Keywords: []string{ "create consumed odata service", "create odata client", "odata client", "consume odata", "external entity", "metadata url", "odata4", "headers", "proxy", @@ -72,8 +73,29 @@ func init() { }) Register(SyntaxFeature{ - Path: "odata.publish", - Summary: "Publish entities as OData services with KEY, options, and navigation properties", + Path: "odata.write", + Summary: "Save or delete an external (OData) object from a microflow: SEND / DELETE EXTERNAL OBJECT", + Keywords: []string{ + "send external object", "delete external object", "external object", "save external entity", + "odata write", "odata update", "odata delete", "commit external entity", + }, + Syntax: "SEND EXTERNAL OBJECT $Obj [REFRESH] [ON ERROR …];\n" + + "DELETE EXTERNAL OBJECT $Obj [REFRESH] [ON ERROR …];\n\n" + + "-- Studio Pro's \"Send external object\" and \"Delete external object\"\n" + + "-- activities. An external entity is saved through its consumed service:\n" + + "-- a plain COMMIT or DELETE on one is a Studio Pro error. The entity set must\n" + + "-- allow the operation (Updatable / Deletable in the service's $metadata),\n" + + "-- or mxbuild reports CE9435.\n" + + "-- ON ERROR CONTINUE is CE6035 on both (MDL076); the default and a custom\n" + + "-- handler, with or without rollback, are accepted.", + Example: "mdl 1;\nCREATE OR MODIFY MICROFLOW MyModule.ACT_SaveCustomer ($Customer: MyModule.Customer)\nBEGIN\n CHANGE $Customer (Name = 'Renamed');\n SEND EXTERNAL OBJECT $Customer REFRESH;\n RETURN;\nEND;", + SeeAlso: []string{"odata.consume", "microflow.object-operations", "microflow.error-handling"}, + }) + + Register(SyntaxFeature{ + Path: "odata.publish", + Features: []string{"integration/published_odata_service", "integration/odata_graphql"}, + Summary: "Publish entities as OData services with KEY, options, and navigation properties", Keywords: []string{ "create published odata service", "create odata service", "publish entity", "publish odata", "expose", "key", "navigation property", "association exposure", @@ -210,8 +232,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "rest.call", - Summary: "CALL REST SERVICE activity inside a microflow, and its five RETURNS forms", + Path: "rest.call", + Features: []string{"microflows/send_rest_request", "microflows/send_rest_request_parameters", "microflows/send_rest_request_output", "microflows/send_rest_request_body", "microflows/send_rest_query_params"}, + Summary: "CALL REST SERVICE activity inside a microflow, and its five RETURNS forms", Keywords: []string{ "rest call", "call rest service", "http get", "http post", "returns response", "returns string", "returns mapping", @@ -334,8 +357,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "rest.consumed", - Summary: "Create consumed REST clients with operations, mappings, and authentication", + Path: "rest.consumed", + Features: []string{"integration/rest_client_basic", "integration/rest_client_authentication", "integration/rest_client_headers", "integration/rest_client_operations", "integration/rest_client_query_params", "integration/rest_client_openapi"}, + Summary: "Create consumed REST clients with operations, mappings, and authentication", Keywords: []string{ "create consumed rest service", "create rest client", "rest client", "consume rest", "rest operation", "alter consumed rest service", "add operation", "modify operation", "drop operation", @@ -349,8 +373,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "rest.published", - Summary: "Create and manage published REST services with resources and operations", + Path: "rest.published", + Features: []string{"integration/published_rest_service", "integration/published_rest_alter", "integration/published_rest_grant_revoke"}, + Summary: "Create and manage published REST services with resources and operations", Keywords: []string{ "create published rest", "publish rest", "rest resource", "rest operation", "microflow", "path parameter", @@ -459,8 +484,9 @@ func init() { // ── OQL ─────────────────────────────────────────────────────────── Register(SyntaxFeature{ - Path: "oql", - Summary: "OQL query execution against a running Mendix runtime", + Path: "oql", + Features: []string{"oql_functions/basic_select", "oql_functions/aggregate_functions", "oql_functions/join_types", "oql_functions/subqueries", "oql_functions/comments", "oql_functions/string_agg"}, + Summary: "OQL query execution against a running Mendix runtime", Keywords: []string{ "oql", "query", "runtime", "m2ee", "aggregate", "view entity", "mxcli oql", @@ -473,8 +499,9 @@ func init() { // ── External database connector ─────────────────────────────────── Register(SyntaxFeature{ - Path: "database-connection", - Summary: "External Database Connector — query another database from a microflow", + Path: "database-connection", + Features: []string{"integration/database_connector_basic", "integration/database_connector_queries", "microflows/execute_database_query", "microflows/execute_database_query_runtime_connection", "integration/database_connector_execute"}, + Summary: "External Database Connector — query another database from a microflow", Keywords: []string{ "database connection", "database connections", "external database", "create database connection", "drop database connection", @@ -587,8 +614,9 @@ DESCRIBE DATABASE CONNECTION Ops.Erp;`, // ── Business Events ─────────────────────────────────────────────── Register(SyntaxFeature{ - Path: "business-events", - Summary: "Business event services — publish and subscribe to events via Kafka", + Path: "business-events", + Features: []string{"integration/business_events"}, + Summary: "Business event services — publish and subscribe to events via Kafka", Keywords: []string{ "business event", "business events", "kafka", "publish", "subscribe", "message", "event channel", @@ -662,7 +690,7 @@ DESCRIBE DATABASE CONNECTION Ops.Erp;`, "exposed as", "returns", "generics", "drop java action", "extra code", "helper method", "import", }, - Syntax: "CREATE [OR MODIFY] JAVA ACTION Module.Name[] [FOLDER 'path'](\n [/** description */] Param: Type [NOT NULL],\n EntityType: ENTITY NOT NULL,\n Obj: pEntity\n) RETURNS ReturnType [AS $ReturnName]\n[EXPOSED AS 'Label' IN 'Category'\n [ICON 'icon.png'] [ICON DARK 'icon-dark.png']\n [IMAGE 'image.png'] [IMAGE DARK 'image-dark.png']]\n[NOT EXPOSED]\nAS $$\n// Java code — AS $$ ... $$ is mandatory, cannot be omitted\n$$\n[EXTRA CODE $$\n// helper methods, constants, nested classes beside executeAction()\n$$];\n\nOR MODIFY: updates signature/body in-place, preserves UUID.\n\nEXPOSED AS puts the action in Studio Pro's toolbox. The icon is a 64x64 PNG\nand the image a 256x192 PNG, read from disk relative to the .mdl file's own directory.\nAn OMITTED clause preserves what is stored — including bitmaps set in Studio\nPro — so removing an entry is NOT EXPOSED, and clearing one bitmap is\nDROP ICON / DROP IMAGE [DARK].\n\nA parameter's description is a /** ... */ doc comment before it, as on an\nentity attribute; AS $ReturnName names the return value (the default name of\nthe variable a microflow call stores it in), as on a microflow. Leaving either\nout keeps what is stored; a new action gets no description and ReturnValueName.\n\nThe .java file is written as mxbuild writes it, so a build leaves it unchanged:\nthe action's own /** ... */ (before CREATE) is the class javadoc, and a rewrite\nwithout one keeps the stored documentation.\n\nThe file has two more sections a regeneration keeps, and the statement states\nboth: an import line in the body goes to the import list, and EXTRA CODE $$ ... $$\nis the EXTRA CODE section (helpers the body calls). An OMITTED EXTRA CODE keeps\nthe stored section; EXTRA CODE $$ $$ clears it. DESCRIBE prints the imports a\ndeveloper added and the section, so its output rebuilds the same file\n(ako/mxcli#1702).\n\nType parameters: an ENTITY parameter declares T. A type parameter no\nENTITY parameter names is declared in the header, Module.Name; an\nunqualified type that is neither is refused (MDL-JA01), since it would be\nwritten as an entity with no module (CE1613).\n\nA bare Module.Name type is an entity or an enumeration, whichever the project\n(or an earlier statement of the script) has; ENUM Module.Name says enumeration\nexplicitly. A name that is neither is refused by check --references and exec,\nsince it would be written as an entity that does not exist (CE1613).\n\nRETURNS takes Boolean, Integer, Long, Decimal, String, DateTime, an enumeration,\nan entity, a list, a type parameter or Void; leaving it out returns Boolean.\nAny other type is refused (MDL-JA03) — StringTemplate(...) is a parameter\ntype only.", + Syntax: "CREATE [OR MODIFY] JAVA ACTION Module.Name[] [FOLDER 'path'](\n [/** description */] Param: Type [NOT NULL],\n EntityType: ENTITY NOT NULL,\n Obj: pEntity\n) RETURNS ReturnType [AS $ReturnName]\n[EXPOSED AS 'Label' IN 'Category'\n [ICON 'icon.png'] [ICON DARK 'icon-dark.png']\n [IMAGE 'image.png'] [IMAGE DARK 'image-dark.png']]\n[NOT EXPOSED]\nAS $$\n// Java code — AS $$ ... $$ is mandatory, cannot be omitted\n$$\n[EXTRA CODE $$\n// helper methods, constants, nested classes beside executeAction()\n$$];\n\nOR MODIFY: updates signature/body in-place, preserves UUID.\n\nEXPOSED AS puts the action in Studio Pro's toolbox. The icon is a 64x64 PNG\nand the image a 256x192 PNG, read from disk relative to the .mdl file's own directory.\nAn OMITTED clause preserves what is stored — including bitmaps set in Studio\nPro — so removing an entry is NOT EXPOSED, and clearing one bitmap is\nDROP ICON / DROP IMAGE [DARK].\n\nA parameter's description is a /** ... */ doc comment before it, as on an\nentity attribute; AS $ReturnName names the return value (the default name of\nthe variable a microflow call stores it in), as on a microflow. Leaving either\nout keeps what is stored; a new action gets no description and ReturnValueName.\n\nThe .java file is written as mxbuild writes it, so a build leaves it unchanged:\nthe action's own /** ... */ (before CREATE) is the class javadoc, and a rewrite\nwithout one keeps the stored documentation.\n\nThe file has two more sections a regeneration keeps, and the statement states\nboth: an import line in the body goes to the import list, and EXTRA CODE $$ ... $$\nis the EXTRA CODE section (helpers the body calls). An OMITTED EXTRA CODE keeps\nthe stored section; EXTRA CODE $$ $$ clears it. DESCRIBE prints the imports a\ndeveloper added and the section, so its output rebuilds the same file\n(ako/mxcli#1702).\n\nA $$ … $$ literal ends at the first $$ it contains, so a body or EXTRA CODE\nholding $$ is written as a string literal instead (AS '…', apostrophes doubled);\nDESCRIBE does this itself (ako/mxcli#1003).\n\nType parameters: an ENTITY parameter declares T. A type parameter no\nENTITY parameter names is declared in the header, Module.Name; an\nunqualified type that is neither is refused (MDL-JA01), since it would be\nwritten as an entity with no module (CE1613).\n\nA bare Module.Name type is an entity or an enumeration, whichever the project\n(or an earlier statement of the script) has; ENUM Module.Name says enumeration\nexplicitly. A name that is neither is refused by check --references and exec,\nsince it would be written as an entity that does not exist (CE1613).\n\nRETURNS takes Boolean, Integer, Long, Decimal, String, DateTime, an enumeration,\nan entity, a list, a type parameter or Void; leaving it out returns Boolean.\nAny other type is refused (MDL-JA03) — StringTemplate(...) is a parameter\ntype only.", Example: "mdl 1;\nCREATE JAVA ACTION Utils.FormatCurrency(\n /** The amount to format. */\n Amount: Decimal NOT NULL\n) RETURNS String AS $FormattedAmount\nEXPOSED AS 'Format Currency' IN 'Formatting'\n ICON 'assets/currency-64.png'\n IMAGE 'assets/currency-256.png'\nAS $$\nreturn String.format(\"%.2f\", Amount);\n$$;\n\n-- Generic entity validator with type parameter\nCREATE JAVA ACTION Utils.IsValid(\n EntityType: ENTITY NOT NULL,\n Obj: pEntity NOT NULL\n) RETURNS Boolean\nAS $$\nreturn Obj != null;\n$$;\n\n-- Type parameter with no entity selector: the argument's entity binds it\nCREATE JAVA ACTION Utils.Describe(\n Obj: T NOT NULL\n) RETURNS String\nAS $$\nreturn Obj.toString();\n$$;\n\n-- A helper in the EXTRA CODE section, and the import it needs\nCREATE JAVA ACTION Utils.Twice(N: Integer NOT NULL) RETURNS Integer\nAS $$\nimport java.util.Objects;\nreturn twice(Objects.requireNonNull(N));\n$$\nEXTRA CODE $$\nprivate static long twice(long n) {\n\treturn n * 2L;\n}\n$$;\n\n-- Idempotent update (preserves UUID)\nCREATE OR MODIFY JAVA ACTION Utils.FormatCurrency(\n Amount: Decimal NOT NULL,\n Decimals: Integer NOT NULL\n) RETURNS String\nAS $$\nreturn String.format(\"%.\" + Decimals + \"f\", Amount);\n$$;", SeeAlso: []string{"java-action", "javascript-action"}, }) @@ -682,8 +710,9 @@ DESCRIBE DATABASE CONNECTION Ops.Erp;`, // ── JSON Structures ─────────────────────────────────────────────── Register(SyntaxFeature{ - Path: "message-definition", - Summary: "Message definition collections — a mapping source built from the domain model", + Path: "message-definition", + Features: []string{"integration/message_definition_collection", "integration/message_definition_document"}, + Summary: "Message definition collections — a mapping source built from the domain model", Keywords: []string{ "message definition", "message definition collection", "create message definition", "exposed entity", "exposed attribute", "exposed association", @@ -889,8 +918,9 @@ DESCRIBE DATABASE CONNECTION Ops.Erp;`, // ── Data Transformers ───────────────────────────────────────────── Register(SyntaxFeature{ - Path: "data-transformer", - Summary: "Data transformers — JSLT/XSLT document transformations (Mendix 11.9+)", + Path: "data-transformer", + Features: []string{"integration/data_transformer", "microflows/transform_json"}, + Summary: "Data transformers — JSLT/XSLT document transformations (Mendix 11.9+)", Keywords: []string{ "data transformer", "create data transformer", "drop data transformer", "list data transformers", "jslt", "xslt", "transform", @@ -915,8 +945,9 @@ DESCRIBE DATABASE CONNECTION Ops.Erp;`, }) Register(SyntaxFeature{ - Path: "agents.model", - Summary: "CREATE/DROP AI MODEL documents for AI agents", + Path: "agents.model", + Features: []string{"agent_documents/agent_model"}, + Summary: "CREATE/DROP AI MODEL documents for AI agents", Keywords: []string{"create ai model", "drop ai model", "describe ai model", "list ai models", "create model", "drop model", "describe model", "list models", "provider", "mxcloudgenai"}, Syntax: "CREATE [OR MODIFY] AI MODEL Module.Name [FOLDER 'path'] (\n Provider: MxCloudGenAI,\n Key: @Module.ApiKeyConstant\n);\nDESCRIBE AI MODEL Module.Name;\nLIST AI MODELS [IN Module];\nDROP AI MODEL Module.Name;", @@ -926,6 +957,7 @@ DESCRIBE DATABASE CONNECTION Ops.Erp;`, Register(SyntaxFeature{ Path: "agents.knowledge-base", + Features: []string{"agent_documents/agent_knowledge_base"}, Summary: "CREATE/DROP KNOWLEDGE BASE documents for AI agents", Keywords: []string{"create knowledge base", "drop knowledge base", "knowledge base", "kb", "rag"}, Syntax: "CREATE [OR MODIFY] KNOWLEDGE BASE Module.Name [FOLDER 'path'] (\n Provider: MxCloudGenAI,\n Key: @Module.KBApiKeyConstant\n);\nDESCRIBE KNOWLEDGE BASE Module.Name;\nLIST KNOWLEDGE BASES [IN Module];\nDROP KNOWLEDGE BASE Module.Name;", @@ -935,6 +967,7 @@ DESCRIBE DATABASE CONNECTION Ops.Erp;`, Register(SyntaxFeature{ Path: "agents.mcp-service", + Features: []string{"agent_documents/agent_consumed_mcp_service"}, Summary: "CREATE/DROP CONSUMED MCP SERVICE documents for AI agents", Keywords: []string{"consumed mcp service", "mcp", "mcp service", "protocol version"}, Syntax: "CREATE [OR MODIFY] CONSUMED MCP SERVICE Module.Name [FOLDER 'path'] (\n ProtocolVersion: v2025_03_26,\n Version: '1.0',\n ConnectionTimeoutSeconds: 30,\n Documentation: 'description'\n);\nDESCRIBE CONSUMED MCP SERVICE Module.Name;\nLIST CONSUMED MCP SERVICES [IN Module];\nDROP CONSUMED MCP SERVICE Module.Name;", @@ -943,8 +976,9 @@ DESCRIBE DATABASE CONNECTION Ops.Erp;`, }) Register(SyntaxFeature{ - Path: "agents.agent", - Summary: "CREATE/DROP AGENT documents with variables, tools, KB tools, and MCP service tools", + Path: "agents.agent", + Features: []string{"agent_documents/agent"}, + Summary: "CREATE/DROP AGENT documents with variables, tools, KB tools, and MCP service tools", Keywords: []string{ "create agent", "drop agent", "usagetype", "systemprompt", "userprompt", "variables", "toolchoice", "temperature", "topp", "maxtokens", diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index 23b3e9699a..cc888cf348 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -4,8 +4,9 @@ package syntax func init() { Register(SyntaxFeature{ - Path: "microflow", - Summary: "Programmatic logic — variables, object operations, control flow, and integrations", + Path: "microflow", + Features: []string{"microflows/basic"}, + Summary: "Programmatic logic — variables, object operations, control flow, and integrations", Keywords: []string{ "microflow", "nanoflow", "logic", "automation", "action", "activity", "flow", @@ -16,8 +17,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "microflow.synchronize", - Summary: "SYNCHRONIZE — offline data synchronization (nanoflow only)", + Path: "microflow.synchronize", + Features: []string{"microflows/synchronize", "microflows/synchronize_unsynchronized"}, + Summary: "SYNCHRONIZE — offline data synchronization (nanoflow only)", Keywords: []string{ "synchronize", "sync", "offline", "unsynchronized", "specific", "offline first", "nanoflow", @@ -28,8 +30,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "microflow.send-email", - Summary: "SEND EMAIL — the built-in Send Email activity (SMTP, Mendix 11.13+, beta)", + Path: "microflow.send-email", + Features: []string{"microflows/send_email"}, + Summary: "SEND EMAIL — the built-in Send Email activity (SMTP, Mendix 11.13+, beta)", Keywords: []string{ "send email", "email", "mail", "smtp", "send mail", "e-mail", "cc", "bcc", "subject", "attachment", "email connector", @@ -93,8 +96,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "microflow.create", - Summary: "Create a microflow with parameters, return type, and body", + Path: "microflow.create", + Features: []string{"microflows/deep_link_url"}, + Summary: "Create a microflow with parameters, return type, and body", Keywords: []string{ "create microflow", "new microflow", "define microflow", "parameters", "returns", "folder", @@ -119,7 +123,8 @@ func init() { " MDL-MF03 DISALLOW needs an error message or microflow (CE4899)\n" + "and with a project, a URL another microflow already owns (CE0570).\n\n" + "`Mark as used` still has no clause and is carried, as all of these were\n" + - "before they were authorable (mendixlabs/mxcli#1120).", + "before they were authorable (mendixlabs/mxcli#1120).\n\n" + + "DROP MICROFLOW [IF EXISTS] Module.Name; -- every DROP: see `mxcli syntax drop`", Example: "CREATE MICROFLOW MyModule.ACT_CreateOrder (\n $CustomerCode: String,\n $Quantity: Integer\n)\nRETURNS MyModule.Order AS $NewOrder\nFOLDER 'Orders'\nEXPOSED AS MICROFLOW ACTION 'Create order' IN 'Orders'\n ICON 'assets/order-64.png'\nBEGIN\n $NewOrder = CREATE MyModule.Order (\n OrderNumber = 'ORD-001',\n Quantity = $Quantity\n );\n COMMIT $NewOrder;\n RETURN $NewOrder;\nEND;", SeeAlso: []string{"microflow.nanoflow", "microflow.variables"}, }) @@ -218,8 +223,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "microflow.control-flow", - Summary: "IF/ELSIF/ELSE, LOOP, WHILE, BREAK, CONTINUE, RETURN", + Path: "microflow.control-flow", + Features: []string{"microflows/while_loop", "microflows/loop_over_list", "microflows/loop_in_branch"}, + Summary: "IF/ELSIF/ELSE, LOOP, WHILE, BREAK, CONTINUE, RETURN", Keywords: []string{ "if", "elsif", "else", "then", "end if", "loop", "while", "break", "continue", "return", @@ -343,7 +349,7 @@ func init() { "-- (MDL-FLOW01). NORMALIZED instead folds the branch guards into a single\n" + "-- condition, which is equivalent and duplicates nothing.\n" + "--\n" + - "-- The graph from mxcli #923:\n" + + "-- The graph from mendixlabs/mxcli#923:\n" + "-- split1: true -> split2 false -> merge1\n" + "-- split2: true -> merge1 false -> merge2\n" + "-- The activity on merge1 runs on `not(c1) or c2`. Default DESCRIBE renders\n" + @@ -465,8 +471,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "microflow.object-operations", - Summary: "CREATE, CHANGE, COMMIT, ROLLBACK, and DELETE objects", + Path: "microflow.object-operations", + Features: []string{"microflows/send_external_object", "microflows/delete_external_object"}, + Summary: "CREATE, CHANGE, COMMIT, ROLLBACK, and DELETE objects", Keywords: []string{ "create object", "change object", "commit", "rollback", "delete", "save", "persist", "modify object", @@ -508,8 +515,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "microflow.list-operations", - Summary: "List manipulation — HEAD, TAIL, FIND, FILTER, SORT, UNION, RANGE, aggregates", + Path: "microflow.list-operations", + Features: []string{"microflows/list_find_filter_by_expression", "microflows/list_range"}, + Summary: "List manipulation — HEAD, TAIL, FIND, FILTER, SORT, UNION, RANGE, aggregates", Keywords: []string{ "list", "head", "tail", "find", "filter", "sort", "union", "intersect", "subtract", "count", "sum", @@ -663,7 +671,7 @@ func init() { " -- absolute, whichever date comes first; rounded for an Integer\n" + " return round(minutesBetween($Start, $End));\n" + "end;", - SeeAlso: []string{"microflow.logging", "microflow.variables", "regular-expression"}, + SeeAlso: []string{"microflow.expressions.functions", "microflow.logging", "microflow.variables", "regular-expression"}, }) Register(SyntaxFeature{ @@ -743,8 +751,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "microflow.show-page", - Summary: "Open and close pages from microflows", + Path: "microflow.show-page", + Features: []string{"microflows/show_page_with_params"}, + Summary: "Open and close pages from microflows", Keywords: []string{ "show page", "open page", "close page", "display page", "navigate", "page action", "number of pages to close", "title override", @@ -761,14 +770,19 @@ func init() { }) Register(SyntaxFeature{ - Path: "microflow.call", - Summary: "Call microflows and Java actions with parameters", + Path: "microflow.call", + Features: []string{"microflows/queued_calls"}, + Summary: "Call microflows and Java actions with parameters", Keywords: []string{ "call microflow", "call java action", "invoke", "in queue", "queued call", "background execution", "sub-microflow", "java action", "parameter passing", "blank argument", "nothing", "text template parameter", "stringtemplate", "document parameter", "agent parameter", }, - Syntax: "$Result = CALL MICROFLOW Module.Name (Param = value) [IN QUEUE Module.Queue];\n$Result = CALL JAVA ACTION Module.Name (Param = value) [IN QUEUE Module.Queue];\n\n" + + Syntax: "$Result = CALL MICROFLOW Module.Name (Param = value) [IN QUEUE Module.Queue [retry]];\n$Result = CALL JAVA ACTION Module.Name (Param = value) [IN QUEUE Module.Queue [retry]];\n\n" + + "-- retry (Mendix 9.10+; every property required, Retries >= 1):\n" + + "-- RETRY FIXED (Retries: n, Interval: n, IntervalType: Seconds|Minutes|Hours)\n" + + "-- RETRY EXPONENTIAL (Retries: n, InitialInterval: n, MaximumInterval: n, IntervalType: …)\n" + + "-- A rewrite that keeps a queued call but drops its stored retry is refused.\n\n" + "-- A Java action parameter of type StringTemplate(Text|Sql) takes a template, filled\n" + "-- like a log message: Param = 'text {1}' WITH ({1} = expression, ...). A plain string\n" + "-- is the text; any other expression is the template '{1}' with it as the argument,\n" + @@ -777,13 +791,14 @@ func init() { "-- DateTime); an enumeration is refused — pass getKey(...) or getCaption(...).\n" + "-- A document parameter (DESCRIBE JAVA ACTION shows Document(), e.g. an\n" + "-- agent-editor Agent) takes the document's name: Param = Module.Document.", - Example: "$IsValid = CALL MICROFLOW MyModule.ValidateOrder (\n Order = $NewOrder\n);\n\n$Token = CALL JAVA ACTION MyModule.GenerateToken (\n UserId = $User/Id\n);\n\n-- Run the call on a task queue (background execution). The queue must\n-- exist; a queued CALL JAVA ACTION must return Nothing, or the build fails\n-- with CE7038.\nCALL MICROFLOW MyModule.ACT_Refresh () IN QUEUE MyModule.RefreshQueue;\nCALL JAVA ACTION MyModule.RefreshData (Url = $Url) IN QUEUE MyModule.RefreshQueue;\n\n-- A blank argument: the mapping is kept, its expression left empty, as\n-- Studio Pro stores an argument field left blank. Not the same as `empty`\n-- (a Mendix value); only CALL MICROFLOW and CALL NANOFLOW take it. Mendix\n-- reports it as CE0127 \"Missing argument\": it exists so DESCRIBE round-trips.\nCALL MICROFLOW MyModule.ACT_Log (Message = 'x', TraceId = nothing);\n\n-- A text-template parameter, and a document parameter.\nCALL JAVA ACTION BusinessEvents.PublishBusinessEvent_V2 (\n Subject = 'Order {1} shipped' WITH ({1} = $Order/OrderNumber),\n BusinessEvent = $Event\n);\n$Response = CALL JAVA ACTION AgentEditorCommons.Agent_CallDocument_WithoutHistory (\n Agent = MyModule.PaymentReminderAgent, OptionalContextObject = $Invoice\n);", + Example: "$IsValid = CALL MICROFLOW MyModule.ValidateOrder (\n Order = $NewOrder\n);\n\n$Token = CALL JAVA ACTION MyModule.GenerateToken (\n UserId = $User/Id\n);\n\n-- Run the call on a task queue (background execution). The queue must\n-- exist; a queued CALL JAVA ACTION must return Nothing, or the build fails\n-- with CE7038.\nCALL MICROFLOW MyModule.ACT_Refresh () IN QUEUE MyModule.RefreshQueue;\nCALL JAVA ACTION MyModule.RefreshData (Url = $Url) IN QUEUE MyModule.RefreshQueue;\n\n-- Retried with back-off when the receiver is down.\nCALL MICROFLOW MyModule.POST_Message (Body = $Body) IN QUEUE MyModule.RefreshQueue\n RETRY EXPONENTIAL (Retries: 20, InitialInterval: 1, MaximumInterval: 60, IntervalType: Minutes);\n\n-- A blank argument: the mapping is kept, its expression left empty, as\n-- Studio Pro stores an argument field left blank. Not the same as `empty`\n-- (a Mendix value); CALL MICROFLOW, CALL NANOFLOW and CALL EXTERNAL ACTION\n-- take it. On a flow call Mendix reports it as CE0127 \"Missing argument\":\n-- it exists so DESCRIBE round-trips. On an external action it is how Studio\n-- Pro 11.16 stores an optional OData parameter left blank.\nCALL MICROFLOW MyModule.ACT_Log (Message = 'x', TraceId = nothing);\nCALL EXTERNAL ACTION MyModule.OrderService.CreateBuffer (command = $Cmd, additional = nothing);\n\n-- A text-template parameter, and a document parameter.\nCALL JAVA ACTION BusinessEvents.PublishBusinessEvent_V2 (\n Subject = 'Order {1} shipped' WITH ({1} = $Order/OrderNumber),\n BusinessEvent = $Event\n);\n$Response = CALL JAVA ACTION AgentEditorCommons.Agent_CallDocument_WithoutHistory (\n Agent = MyModule.PaymentReminderAgent, OptionalContextObject = $Invoice\n);", SeeAlso: []string{"java-action", "microflow.create", "queue", "microflow.workflow-actions"}, }) Register(SyntaxFeature{ - Path: "microflow.workflow-actions", - Summary: "Start and manage workflows from a microflow: call workflow, get workflow data, workflow operation, lock/unlock", + Path: "microflow.workflow-actions", + Features: []string{"workflows/workflow_microflow_actions", "workflows/workflow_operations", "workflows/workflow_operation_pause_continue", "workflows/workflow_operation_retry_unpause", "workflows/get_workflow_data", "workflows/get_workflows", "workflows/get_workflow_activity_records", "workflows/workflow_definition_selection"}, + Summary: "Start and manage workflows from a microflow: call workflow, get workflow data, workflow operation, lock/unlock", Keywords: []string{ "call workflow", "start workflow", "trigger workflow", "workflow from microflow", "get workflow data", "get workflows for", "get workflow activity records", diff --git a/cmd/mxcli/syntax/features_misc.go b/cmd/mxcli/syntax/features_misc.go index 302caf0c5e..46feb526c2 100644 --- a/cmd/mxcli/syntax/features_misc.go +++ b/cmd/mxcli/syntax/features_misc.go @@ -110,6 +110,118 @@ func init() { SeeAlso: []string{"microflow", "domain-model.entity", "page", "document-folder", "create-if-not-exists"}, }) + // `show features` had no topic, so a reader who checked the index concluded + // there was no way to ask which Mendix version a statement needs + // (ako/mxcli#1367). The same registry answers it, the topics' "Requires" + // lines, and the exec-time version gate. + Register(SyntaxFeature{ + Path: "features", + Summary: "LIST FEATURES — which MDL features a Mendix version supports, and since when", + Keywords: []string{ + "show features", "list features", "features", "version", "mendix version", + "supported", "support", "requires", "since", "added since", "min version", "compatibility", + }, + Syntax: "LIST FEATURES; -- the connected project's version\n" + + "LIST FEATURES IN ; -- one area, e.g. workflows, microflows, pages\n" + + "LIST FEATURES FOR VERSION 10.24; -- any version, no project needed\n" + + "LIST FEATURES FOR VERSION 10.24.3;\n" + + "LIST FEATURES ADDED SINCE 11.0; -- what is new from a version on\n\n" + + "-- Areas: agent_documents, domain_model, integration, microflows, modules,\n" + + "-- mpr_format, navigation, oql_functions, pages, scheduled_events,\n" + + "-- security, workflows. SHOW FEATURES is the older spelling (MDL-DEPR002).\n" + + "--\n" + + "-- The table is sdk/versions/*.yaml, the registry `exec` and `check -p`\n" + + "-- gate on: a statement whose feature the project's version lacks is\n" + + "-- refused before anything is written. Each `mxcli syntax ` page\n" + + "-- prints its own \"Requires: Mendix X+\" from the same registry.", + Example: "LIST FEATURES FOR VERSION 10.24;\nLIST FEATURES ADDED SINCE 11.0;", + SeeAlso: []string{"language-header", "status"}, + }) + + // Every DROP statement, in one place. They used to be spread over the topic of + // each document type, and several (DROP ENTITY, DROP MICROFLOW, DROP NANOFLOW, + // DROP SNIPPET) were in none: `mxcli syntax drop` found three unrelated + // topics, and DROP ASSOCIATION could only be found by trial (#1716). + // drop_statements_drift_test.go holds this list to the grammar's + // dropStatement rule. + Register(SyntaxFeature{ + Path: "drop", + Summary: "DROP — remove a document (entity, association, microflow, page, …)", + Keywords: []string{ + "drop", "delete", "remove", "destroy", "drop entity", "drop association", + "drop microflow", "drop nanoflow", "drop page", "drop snippet", "drop module", + "drop enumeration", "drop constant", "drop index", "drop annotation", + }, + Syntax: "-- Domain model\n" + + "DROP ENTITY Module.Name;\n" + + "DROP EXTERNAL ENTITY Module.Name;\n" + + "DROP ASSOCIATION Module.Name;\n" + + "DROP ENUMERATION Module.Name;\n" + + "DROP CONSTANT Module.Name;\n" + + "DROP VALIDATION RULE FOR Module.Entity.Attribute [REGEX | RANGE];\n" + + "DROP REGULAR EXPRESSION Module.Name;\n" + + "DROP INDEX Module.IndexName ON Module.Entity;\n" + + "DROP ANNOTATION 'first line of the caption' IN Module;\n" + + "DROP ANNOTATION AT (x, y) IN Module;\n\n" + + "-- Logic\n" + + "DROP MICROFLOW Module.Name;\n" + + "DROP NANOFLOW Module.Name;\n" + + "DROP RULE Module.Name;\n" + + "DROP JAVA ACTION Module.Name;\n" + + "DROP JAVASCRIPT ACTION Module.Name;\n" + + "DROP WORKFLOW Module.Name;\n" + + "DROP TASK QUEUE Module.Name;\n" + + "DROP SCHEDULED EVENT Module.Name;\n\n" + + "-- Pages and resources\n" + + "DROP PAGE Module.Name;\n" + + "DROP SNIPPET Module.Name;\n" + + "DROP LAYOUT Module.Name;\n" + + "DROP MENU Module.Name;\n" + + "DROP IMAGE COLLECTION Module.Name;\n\n" + + "-- Integration\n" + + "DROP CONSUMED ODATA SERVICE Module.Name;\n" + + "DROP PUBLISHED ODATA SERVICE Module.Name;\n" + + "DROP CONSUMED REST SERVICE Module.Name;\n" + + "DROP PUBLISHED REST SERVICE Module.Name;\n" + + "DROP BUSINESS EVENT SERVICE Module.Name;\n" + + "DROP DATABASE CONNECTION Module.Name;\n" + + "DROP JSON STRUCTURE Module.Name;\n" + + "DROP MESSAGE DEFINITION COLLECTION Module.Name;\n" + + "DROP MESSAGE DEFINITION Module.Name;\n" + + "DROP IMPORT MAPPING Module.Name;\n" + + "DROP EXPORT MAPPING Module.Name;\n" + + "DROP DATA TRANSFORMER Module.Name;\n\n" + + "-- Agents\n" + + "DROP AI MODEL Module.Name;\n" + + "DROP CONSUMED MCP SERVICE Module.Name;\n" + + "DROP KNOWLEDGE BASE Module.Name;\n" + + "DROP AGENT Module.Name;\n\n" + + "-- Project structure\n" + + "DROP MODULE ModuleName;\n" + + "DROP FOLDER 'path' IN Module;\n" + + "DROP CONFIGURATION 'Name';\n\n" + + "-- Every form but INDEX and ANNOTATION takes IF EXISTS after the type\n" + + "-- (DROP ENTITY IF EXISTS Module.Name;) -- see drop-if-exists. Without it\n" + + "-- a missing target is an error, as in SQL.\n" + + "--\n" + + "-- Members of a document are removed with ALTER, not DROP:\n" + + "-- ALTER ENTITY … DROP ATTRIBUTE | DROP INDEX | DROP EVENT HANDLER\n" + + "-- ALTER ENUMERATION … DROP VALUE\n" + + "-- ALTER MESSAGE DEFINITION … DROP MEMBER\n" + + "-- ALTER PAGE / ALTER SNIPPET … DROP WIDGET\n" + + "-- ALTER WORKFLOW … DROP ACTIVITY | DROP OUTCOME | DROP PATH …\n" + + "-- Security has its own: DROP USER ROLE, DROP MODULE ROLE, DROP DEMO USER\n" + + "-- (see security).\n" + + "--\n" + + "-- DROP ASSOCIATION reconciles the entity access rules that named it, and is\n" + + "-- refused while a message definition still exposes it.", + Example: "mdl 1;\n" + + "DROP ASSOCIATION Shop.Order_Customer;\n" + + "DROP ENTITY Shop.OrderLine;\n" + + "DROP MICROFLOW IF EXISTS Shop.ACT_Old;", + SeeAlso: []string{"drop-if-exists", "domain-model.entity.alter", "security"}, + }) + // IF EXISTS sits on every document-level alternative of dropStatement, so it // is documented once here for the same reason OR MODIFY is (#531). Register(SyntaxFeature{ @@ -314,8 +426,9 @@ DISCONNECT;`, // ── Navigation ────────────────────────────────────────────────────── Register(SyntaxFeature{ - Path: "navigation", - Summary: "Navigation profiles — home pages, menus, login pages per device type", + Path: "navigation", + Features: []string{"navigation/profiles", "navigation/home_pages", "navigation/menu_items", "navigation/menu_item_icons", "navigation/native_home_page", "navigation/not_found_page", "navigation/progressive_web_app"}, + Summary: "Navigation profiles — home pages, menus, login pages per device type", Keywords: []string{ "navigation", "nav", "profile", "responsive", "phone", "tablet", "home page", "menu", "login page", @@ -850,8 +963,19 @@ CALL JAVA ACTION (see: mxcli syntax microflow.call): CALL MICROFLOW Ops.ACT_Process(Order = $Order) IN QUEUE Ops.OrderProcessing; A rewrite that does NOT restate a stored binding is refused, because it would -drop it silently. A retry policy on a queued call has no MDL spelling and is -also refused rather than reset — change those in Studio Pro.`, +drop it silently. + +A queued call's retry policy (Mendix 9.10+) follows the queue: + + CALL MICROFLOW Ops.POST_Message(Body = $Body) IN QUEUE Ops.SendQueue + RETRY EXPONENTIAL (Retries: 20, InitialInterval: 1, MaximumInterval: 60, IntervalType: Minutes); + CALL JAVA ACTION Ops.Refresh(Url = $Url) IN QUEUE Ops.SendQueue + RETRY FIXED (Retries: 5, Interval: 10, IntervalType: Minutes); + +Every property is required (IntervalType: Seconds | Minutes | Hours); Retries +must be at least 1 (CE7044). DESCRIBE prints a stored retry, so it round-trips. +A rewrite that still calls into the queue but leaves the retry out is refused +rather than reset: restate it, or state a new one.`, Example: `mdl 1; CREATE TASK QUEUE Ops.OrderProcessing ( Parallelism: 3, @@ -1003,8 +1127,9 @@ CREATE VALIDATION RULE FOR Shop.Product.Price // ── Scheduled events ──────────────────────────────────────────────── Register(SyntaxFeature{ - Path: "scheduled-event", - Summary: "Scheduled events — Mendix's cron: run a microflow on a repeating schedule", + Path: "scheduled-event", + Features: []string{"scheduled_events/schedules"}, + Summary: "Scheduled events — Mendix's cron: run a microflow on a repeating schedule", Keywords: []string{ "scheduled event", "scheduled events", "schedule", "cron", "recurring", "create scheduled event", "drop scheduled event", "describe scheduled event", @@ -1481,8 +1606,9 @@ mxcli check script.mdl -p app.mpr --references -- With reference validation`, }) Register(SyntaxFeature{ - Path: "module.settings", - Summary: "A module's version and module type (Studio Pro's Module settings, General tab)", + Path: "module.settings", + Features: []string{"modules/module_type"}, + Summary: "A module's version and module type (Studio Pro's Module settings, General tab)", Keywords: []string{ "module settings", "module version", "module type", "add-on module", "addon", "app module", "semver", "export module", @@ -1505,8 +1631,9 @@ alter module Inventory set (Version: '2.4.0'); }) Register(SyntaxFeature{ - Path: "module.jar-dependencies", - Summary: "Manage Maven/JAR dependencies in a module's settings", + Path: "module.jar-dependencies", + Features: []string{"modules/jar_dependencies", "modules/jar_dependency_exclusions"}, + Summary: "Manage Maven/JAR dependencies in a module's settings", Keywords: []string{ "jar dependency", "maven", "jar dep", "module settings", "group", "artifact", "classpath", "exclusion", diff --git a/cmd/mxcli/syntax/features_page.go b/cmd/mxcli/syntax/features_page.go index adc6fce5eb..3fd99c7c94 100644 --- a/cmd/mxcli/syntax/features_page.go +++ b/cmd/mxcli/syntax/features_page.go @@ -6,8 +6,9 @@ func init() { // ── Page ────────────────────────────────────────────────────────────── Register(SyntaxFeature{ - Path: "page", - Summary: "Create and manage pages with widgets, datasources, and actions", + Path: "page", + Features: []string{"pages/basic"}, + Summary: "Create and manage pages with widgets, datasources, and actions", Keywords: []string{ "page", "pages", "form", "UI", "user interface", "widget", "layout", "screen", @@ -18,8 +19,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "page.create", - Summary: "CREATE PAGE with parameters, variables, layout, and URL", + Path: "page.create", + Features: []string{"pages/page_parameters", "pages/page_parameter_optional", "pages/page_variables"}, + Summary: "CREATE PAGE with parameters, variables, layout, and URL", Keywords: []string{ "create page", "new page", "page parameters", "page variables", "layout", "url", "folder", @@ -113,8 +115,9 @@ LIST IMPACT OF htmlelement; }) Register(SyntaxFeature{ - Path: "page.widgets", - Summary: "Widget types: containers, data widgets, inputs, actions, display", + Path: "page.widgets", + Features: []string{"pages/pluggable_widgets", "pages/conditional_visibility", "pages/conditional_editability", "pages/responsive_column_widths"}, + Summary: "Widget types: containers, data widgets, inputs, actions, display", Keywords: []string{ "widget", "layoutgrid", "container", "customcontainer", "dataview", "datagrid", "gallery", "listview", "textbox", "textarea", "datepicker", @@ -292,8 +295,9 @@ LIST IMPACT OF htmlelement; }) Register(SyntaxFeature{ - Path: "page.action", - Summary: "Widget actions: save, cancel, close, delete, show page, microflow, nanoflow, nothing", + Path: "page.action", + Features: []string{"pages/complete_task_action", "pages/workflow_page_actions", "pages/flow_call_output_mappings", "pages/flow_call_output_mappings_to_attribute"}, + Summary: "Widget actions: save, cancel, close, delete, show page, microflow, nanoflow, nothing", Keywords: []string{ "action", "save", "cancel", "close", "delete", "show page", "navigate", "microflow", "create object", @@ -320,8 +324,9 @@ LIST IMPACT OF htmlelement; }) Register(SyntaxFeature{ - Path: "page.alter", - Summary: "ALTER PAGE — modify widget properties, insert, drop, replace widgets in-place", + Path: "page.alter", + Features: []string{"pages/alter_page"}, + Summary: "ALTER PAGE — modify widget properties, insert, drop, replace widgets in-place", Keywords: []string{ "alter page", "modify page", "update page", "set property", "insert widget", "drop widget", "replace widget", @@ -335,8 +340,9 @@ LIST IMPACT OF htmlelement; }) Register(SyntaxFeature{ - Path: "page.styling", - Summary: "CSS classes, inline styles, dynamic classes and Atlas design properties — written inline, or ALTERed in place", + Path: "page.styling", + Features: []string{"pages/design_properties_v3"}, + Summary: "CSS classes, inline styles, dynamic classes and Atlas design properties — written inline, or ALTERed in place", Keywords: []string{ "class", "style", "css", "design properties", "atlas", "spacing", "full width", "dynamic classes", "dynamicclasses", @@ -448,8 +454,9 @@ LIST IMPACT OF htmlelement; }) Register(SyntaxFeature{ - Path: "snippet.create", - Summary: "CREATE SNIPPET with parameters, variables, and folder", + Path: "snippet.create", + Features: []string{"pages/snippet_parameters"}, + Summary: "CREATE SNIPPET with parameters, variables, and folder", Keywords: []string{ "create snippet", "new snippet", "snippet parameters", "snippet variables", diff --git a/cmd/mxcli/syntax/features_security.go b/cmd/mxcli/syntax/features_security.go index 698f5a7d4a..4312416a2e 100644 --- a/cmd/mxcli/syntax/features_security.go +++ b/cmd/mxcli/syntax/features_security.go @@ -16,8 +16,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "security.module-role", - Summary: "Create and manage module-level security roles", + Path: "security.module-role", + Features: []string{"security/module_roles"}, + Summary: "Create and manage module-level security roles", Keywords: []string{ "module role", "create role", "drop role", }, @@ -27,8 +28,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "security.entity-access", - Summary: "Grant or revoke entity-level access (CRUD, attribute-level, XPath rules)", + Path: "security.entity-access", + Features: []string{"security/entity_access"}, + Summary: "Grant or revoke entity-level access (CRUD, attribute-level, XPath rules)", Keywords: []string{ "entity access", "grant", "revoke", "read", "write", "create", "delete", "xpath", "row-level security", @@ -129,8 +131,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "security.user-role", - Summary: "Create and manage application-level user roles that bundle module roles", + Path: "security.user-role", + Features: []string{"security/user_roles"}, + Summary: "Create and manage application-level user roles that bundle module roles", Keywords: []string{ "user role", "application role", "manage roles", "add module roles", "remove module roles", @@ -198,8 +201,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "security.demo-user", - Summary: "Create and manage demo users for testing", + Path: "security.demo-user", + Features: []string{"security/demo_users"}, + Summary: "Create and manage demo users for testing", Keywords: []string{ "demo user", "test user", "demo account", "password", "login", diff --git a/cmd/mxcli/syntax/features_versions_test.go b/cmd/mxcli/syntax/features_versions_test.go new file mode 100644 index 0000000000..cc6ee2f8ed --- /dev/null +++ b/cmd/mxcli/syntax/features_versions_test.go @@ -0,0 +1,85 @@ +// SPDX-License-Identifier: Apache-2.0 + +package syntax + +import ( + "sort" + "testing" + + "github.com/mendixlabs/mxcli/sdk/versions" +) + +// ako/mxcli#1367: `mxcli syntax` and sdk/versions/*.yaml kept separate version +// tables and had drifted apart both ways — workflow.boundary-event said 10.6.0 +// where the registry says 10.14.0, view entities 10.18.0 against 10.21.0, and +// data-transformer had no version at all although the registry gates it. A +// topic now names registry features and its versions are read from there; +// Register refuses a hand-written one. These two tests close both directions. + +// A topic may only name a feature the registry has: a typo would silently +// print no requirement. +func TestSyntaxFeaturesResolve(t *testing.T) { + reg, err := versions.Load() + if err != nil { + t.Fatal(err) + } + for _, f := range All() { + for _, key := range f.Features { + area, name, ok := cutFeatureKey(key) + if !ok { + t.Errorf("%s: feature %q is not area/name", f.Path, key) + continue + } + if _, found := reg.Lookup(area, name); !found { + t.Errorf("%s: feature %q is not in sdk/versions/*.yaml", f.Path, key) + } + } + if len(f.Features) > 0 && (f.MinVersion == "" || len(f.Requires) != len(f.Features)) { + t.Errorf("%s: names %v but resolved MinVersion %q, %d requirements", f.Path, f.Features, f.MinVersion, len(f.Requires)) + } + } +} + +// unreferencedFeatures are registry features no syntax topic names. A LEDGER: +// the test fails when a feature is neither referenced nor listed (add it to the +// topic that documents its statement) and when a listed one gets referenced +// (strike it off), so the list only shrinks. +var unreferencedFeatures = map[string]string{ + // Storage-format facts, not statements: no topic documents their syntax. + "mpr_format/association_storage": "storage format, no statement", + "mpr_format/mpr_v2": "storage format, no statement", + "mpr_format/portable_app": "storage format, no statement", +} + +func TestEveryRegistryFeatureHasATopic(t *testing.T) { + reg, err := versions.Load() + if err != nil { + t.Fatal(err) + } + referenced := map[string]bool{} + for _, f := range All() { + for _, k := range f.Features { + referenced[k] = true + } + } + var missing []string + for _, k := range reg.Keys() { + _, listed := unreferencedFeatures[k] + switch { + case referenced[k] && listed: + t.Errorf("%s is referenced by a topic now: strike it off unreferencedFeatures", k) + case !referenced[k] && !listed: + missing = append(missing, k) + } + } + sort.Strings(missing) + for _, k := range missing { + t.Errorf("registry feature %s is named by no syntax topic: add it to Features of the topic that documents it", k) + } + for k := range unreferencedFeatures { + area, name, _ := cutFeatureKey(k) + if _, ok := reg.Lookup(area, name); !ok { + t.Errorf("unreferencedFeatures lists %s, which the registry no longer has", k) + } + } +} diff --git a/cmd/mxcli/syntax/features_workflow.go b/cmd/mxcli/syntax/features_workflow.go index c8013b1ca7..89e8d9568f 100644 --- a/cmd/mxcli/syntax/features_workflow.go +++ b/cmd/mxcli/syntax/features_workflow.go @@ -4,8 +4,9 @@ package syntax func init() { Register(SyntaxFeature{ - Path: "workflow", - Summary: "Multi-step business processes with user tasks, decisions, and parallel paths", + Path: "workflow", + Features: []string{"workflows/basic"}, + Summary: "Multi-step business processes with user tasks, decisions, and parallel paths", Keywords: []string{ "workflow", "business process", "approval", "review", "user task", "decision", "parallel", @@ -26,8 +27,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "workflow.create", - Summary: "Create a new workflow definition with activities and flow", + Path: "workflow.create", + Features: []string{"workflows/annotations", "workflows/overview_page"}, + Summary: "Create a new workflow definition with activities and flow", Keywords: []string{ "create workflow", "new workflow", "define workflow", "parameter", "overview page", "due date", @@ -60,8 +62,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "workflow.event-handlers", - Summary: "Run a microflow when workflow events happen (task started, workflow completed, ...)", + Path: "workflow.event-handlers", + Features: []string{"workflows/event_handlers"}, + Summary: "Run a microflow when workflow events happen (task started, workflow completed, ...)", Keywords: []string{ "event handler", "workflow event", "on workflow events", "any workflow event", "audit", "on created", "on-created microflow", "WorkflowEvent", "OnWorkflowEvent", @@ -114,8 +117,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "workflow.user-task", - Summary: "User task activity — assigns work to users with outcomes", + Path: "workflow.user-task", + Features: []string{"workflows/user_task"}, + Summary: "User task activity — assigns work to users with outcomes", Keywords: []string{ "user task", "human task", "assign", "assignee", "auto assign", "auto-assign", "outcomes", "approve", "reject", "page", @@ -190,13 +194,14 @@ func init() { "-- Microflow targeting: the microflow takes the workflow AND its context object\n" + "CREATE MICROFLOW HR.GetApprovers ($Workflow: System.Workflow, $Request: HR.Request)\nRETURNS List of System.User AS $Approvers\nBEGIN\n RETRIEVE $Approvers FROM System.User;\n RETURN $Approvers;\nEND;\n\n" + "USER TASK Approve 'Approve request'\n TARGETING MICROFLOW HR.GetApprovers\n OUTCOMES 'Done' { };", - MinVersion: "9.0.0", - SeeAlso: []string{"workflow.user-task", "workflow.multi-user-task"}, + Features: []string{"workflows/user_task", "workflows/group_targeting"}, + SeeAlso: []string{"workflow.user-task", "workflow.multi-user-task"}, }) Register(SyntaxFeature{ - Path: "workflow.multi-user-task", - Summary: "Multi-user task — one activity that several users each act on", + Path: "workflow.multi-user-task", + Features: []string{"workflows/multi_user_task", "workflows/multi_user_task_majority_threshold", "workflows/multi_user_task_await_all", "workflows/multi_user_task_microflow_rule"}, + Summary: "Multi-user task — one activity that several users each act on", Keywords: []string{ "multi user task", "multi-user task", "several users", "multiple assignees", "assessors", "voting", "parallel approval", "group approval", @@ -267,8 +272,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "workflow.parallel-split", - Summary: "Parallel split — execute multiple paths concurrently", + Path: "workflow.parallel-split", + Features: []string{"workflows/parallel_split"}, + Summary: "Parallel split — execute multiple paths concurrently", Keywords: []string{ "parallel", "concurrent", "split", "fork", "join", "parallel gateway", "AND", @@ -304,8 +310,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "workflow.ai-agent-task", - Summary: "AI agent task — a workflow step that runs an agent through a microflow (Mendix 11.9+)", + Path: "workflow.ai-agent-task", + Features: []string{"workflows/ai_agent_task"}, + Summary: "AI agent task — a workflow step that runs an agent through a microflow (Mendix 11.9+)", Keywords: []string{ "ai agent task", "agent task", "call agent", "call agent microflow", "agent", "llm", "genai", "AIAgentTaskActivity", @@ -330,8 +337,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "workflow.call-workflow", - Summary: "Call a sub-workflow from within a workflow", + Path: "workflow.call-workflow", + Features: []string{"workflows/call_workflow_activity", "workflows/call_workflow_parameter_mappings", "workflows/workflow_parameter"}, + Summary: "Call a sub-workflow from within a workflow", Keywords: []string{ "call workflow", "sub-workflow", "nested workflow", }, @@ -353,8 +361,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "workflow.jump-to", - Summary: "Jump to another activity — and the activity names it resolves against", + Path: "workflow.jump-to", + Features: []string{"workflows/jump_to", "workflows/wait_for_timer"}, + Summary: "Jump to another activity — and the activity names it resolves against", Keywords: []string{ "jump", "jump to", "goto", "loop back", "activity name", }, @@ -422,8 +431,9 @@ func init() { }) Register(SyntaxFeature{ - Path: "workflow.catalog", - Summary: "Query workflow metadata via catalog tables", + Path: "workflow.catalog", + Features: []string{"workflows/groups"}, + Summary: "Query workflow metadata via catalog tables", Keywords: []string{ "catalog", "query workflows", "workflow metadata", "cross-reference", "callers", "callees", @@ -446,10 +456,10 @@ func init() { // 'Workflows$TimerBoundaryEvent' could not be found"). This entry showed // the bare form, with 'P3D' — not a valid timer expression — as its delay. // ako/view-entity-examples FINDINGS §7. - Syntax: "-- inline, as a clause of a USER TASK or CALL MICROFLOW (after OUTCOMES),\n-- a CALL WORKFLOW or a WAIT FOR NOTIFICATION:\nBOUNDARY EVENT (INTERRUPTING | NON INTERRUPTING) TIMER [REPEAT EVERY AT MOST TIMES] [CAPTION ''] { }\nBOUNDARY EVENT (INTERRUPTING | NON INTERRUPTING) NOTIFICATION [''] { }\n\n-- A notification boundary event (Mendix 11.11+) is triggered by `notify workflow`\n-- targeting its name, so the name is unique in the workflow. ALTER cannot insert\n-- one yet; restate the workflow instead.\n\n-- or add a timer to an existing task:\nALTER WORKFLOW { INSERT INTO { BOUNDARY EVENT (INTERRUPTING | NON INTERRUPTING) TIMER [CAPTION ''] { } } };\n\n-- REPEAT EVERY repeats a NON INTERRUPTING timer (a reminder): unit minute(s),\n-- hour(s), day(s), week(s) or month(s); the count includes the first firing\n-- and is at least 2. MDL-WF22 refuses what Studio Pro refuses (mx check does\n-- not): an interrupting timer, another unit, fewer than 2.\n\n-- Name the kind: a bare TIMER is refused on Mendix 11 (MDL-WF07) — it writes a\n-- type the runtime cannot load. The delay is an expression that yields a\n-- DateTime, e.g. addDays([%CurrentDateTime%], 3). mxcli ends every boundary\n-- path with Mendix's end-of-path marker, so a path may end in a call; use\n-- JUMP TO to return to the task instead.", - Example: "user task ReviewTask 'Review'\n page Module.WF_Review\n outcomes 'Done' { }\n boundary event interrupting timer addDays([%CurrentDateTime%], 3) caption 'Escalate' {\n call microflow Module.WF_Escalate;\n };", - MinVersion: "10.6.0", - SeeAlso: []string{"workflow.user-task", "workflow.notification", "workflow.event-subprocess"}, + Syntax: "-- inline, as a clause of a USER TASK or CALL MICROFLOW (after OUTCOMES),\n-- a CALL WORKFLOW or a WAIT FOR NOTIFICATION:\nBOUNDARY EVENT (INTERRUPTING | NON INTERRUPTING) TIMER [REPEAT EVERY AT MOST TIMES] [CAPTION ''] { }\nBOUNDARY EVENT (INTERRUPTING | NON INTERRUPTING) NOTIFICATION [''] { }\n\n-- A notification boundary event (Mendix 11.11+) is triggered by `notify workflow`\n-- targeting its name, so the name is unique in the workflow. ALTER cannot insert\n-- one yet; restate the workflow instead.\n\n-- or add a timer to an existing task:\nALTER WORKFLOW { INSERT INTO { BOUNDARY EVENT (INTERRUPTING | NON INTERRUPTING) TIMER [CAPTION ''] { } } };\n\n-- REPEAT EVERY repeats a NON INTERRUPTING timer (a reminder): unit minute(s),\n-- hour(s), day(s), week(s) or month(s); the count includes the first firing\n-- and is at least 2. MDL-WF22 refuses what Studio Pro refuses (mx check does\n-- not): an interrupting timer, another unit, fewer than 2.\n\n-- Name the kind: a bare TIMER is refused on Mendix 11 (MDL-WF07) — it writes a\n-- type the runtime cannot load. The delay is an expression that yields a\n-- DateTime, e.g. addDays([%CurrentDateTime%], 3). mxcli ends every boundary\n-- path with Mendix's end-of-path marker, so a path may end in a call; use\n-- JUMP TO to return to the task instead.", + Example: "user task ReviewTask 'Review'\n page Module.WF_Review\n outcomes 'Done' { }\n boundary event interrupting timer addDays([%CurrentDateTime%], 3) caption 'Escalate' {\n call microflow Module.WF_Escalate;\n };", + Features: []string{"workflows/timer_boundary_event", "workflows/interrupting_timer_boundary_event"}, + SeeAlso: []string{"workflow.user-task", "workflow.notification", "workflow.event-subprocess"}, }) Register(SyntaxFeature{ @@ -483,8 +493,8 @@ func init() { " call microflow HR.ACT_Remind;\n" + " };\n" + "end workflow;", - MinVersion: "11.8.0", - SeeAlso: []string{"workflow.notification", "workflow.boundary-event", "workflow.create"}, + Features: []string{"workflows/event_subprocesses", "workflows/timer_event_subprocesses", "workflows/recurring_timer_event_subprocesses"}, + SeeAlso: []string{"workflow.notification", "workflow.boundary-event", "workflow.create"}, }) Register(SyntaxFeature{ @@ -497,9 +507,9 @@ func init() { "-- The point a `notify workflow` action reaches (Workflows$NotificationActivity,\n" + "-- Mendix 11.11+). `wait for notification` is the older activity for the same\n" + "-- purpose and works on every version.", - Example: "notification DocumentsReceived caption 'Documents received';", - MinVersion: "11.11.0", - SeeAlso: []string{"workflow.event-subprocess", "workflow.boundary-event", "workflow.notify"}, + Example: "notification DocumentsReceived caption 'Documents received';", + Features: []string{"workflows/notification_events", "workflows/wait_for_notification", "workflows/interrupting_notification_event_subprocess"}, + SeeAlso: []string{"workflow.event-subprocess", "workflow.boundary-event", "workflow.notify"}, }) Register(SyntaxFeature{ @@ -515,9 +525,9 @@ func init() { "-- a notification cannot reach (a timer start, a user task). The target is\n" + "-- required: without one the build fails CE0166 (MDL-WF16). $Notified is a\n" + "-- Boolean. Before Mendix 11.7 only a wait for notification can be named.", - Example: "$Cancelled = notify workflow $Workflow target HR.Leave.espCancelStart;", - MinVersion: "11.7.0", - SeeAlso: []string{"workflow.event-subprocess", "workflow.notification", "workflow.boundary-event"}, + Example: "$Cancelled = notify workflow $Workflow target HR.Leave.espCancelStart;", + Features: []string{"workflows/notify_workflow", "workflows/notify_target"}, + SeeAlso: []string{"workflow.event-subprocess", "workflow.notification", "workflow.boundary-event"}, }) Register(SyntaxFeature{ diff --git a/cmd/mxcli/syntax/format.go b/cmd/mxcli/syntax/format.go index 13f1d903c1..6385d40578 100644 --- a/cmd/mxcli/syntax/format.go +++ b/cmd/mxcli/syntax/format.go @@ -50,8 +50,15 @@ func writeFeatureDetail(w io.Writer, f SyntaxFeature) { fmt.Fprintf(w, "%s\n\n", strings.Repeat("═", len(f.Path))) fmt.Fprintf(w, "%s\n\n", f.Summary) - if f.MinVersion != "" { - fmt.Fprintf(w, "Requires: Mendix %s+\n\n", f.MinVersion) + switch { + case len(f.Requires) == 1: + fmt.Fprintf(w, "Requires: Mendix %s+\n\n", f.Requires[0].Since) + case len(f.Requires) > 1: + fmt.Fprintln(w, "Requires:") + for _, r := range f.Requires { + fmt.Fprintf(w, " Mendix %-9s %s\n", r.Since+"+", r.Feature) + } + fmt.Fprintln(w) } if len(f.Keywords) > 0 { diff --git a/cmd/mxcli/syntax/registry.go b/cmd/mxcli/syntax/registry.go index 5ac05453a5..29d8dfc3f6 100644 --- a/cmd/mxcli/syntax/registry.go +++ b/cmd/mxcli/syntax/registry.go @@ -9,13 +9,36 @@ import ( // SyntaxFeature describes a discoverable MDL syntax feature. type SyntaxFeature struct { - Path string `json:"path"` - Summary string `json:"summary"` - Keywords []string `json:"keywords"` - Syntax string `json:"syntax"` - Example string `json:"example"` - MinVersion string `json:"min_version,omitempty"` - SeeAlso []string `json:"see_also,omitempty"` + Path string `json:"path"` + Summary string `json:"summary"` + Keywords []string `json:"keywords"` + Syntax string `json:"syntax"` + Example string `json:"example"` + + // Features names the version-registry features (sdk/versions/*.yaml, as + // "area/name") the topic's statements need. It is the ONLY place a topic + // says which Mendix version it needs: MinVersion and Requires are filled + // from the registry, never written by hand, so `mxcli syntax`, `show + // features` and the exec-time gate cannot disagree (ako/mxcli#1367 — they + // had drifted apart in both directions). + Features []string `json:"-"` + + // MinVersion is the version of the FIRST of Features, the topic's primary + // statement (data-transformer's document is 11.9 although the transform + // that uses it is 11.4). Computed; Register refuses one set by hand. + MinVersion string `json:"min_version,omitempty"` + // Requires lists each feature with its own version, since one topic can + // span several (notify workflow is 10.0, its event sub-process target 11.7). + Requires []Requirement `json:"requires,omitempty"` + + SeeAlso []string `json:"see_also,omitempty"` +} + +// Requirement is one registry feature a topic needs. +type Requirement struct { + Feature string `json:"feature"` // "area/name" + Since string `json:"since"` + MDL string `json:"mdl,omitempty"` } var registry []SyntaxFeature @@ -111,6 +134,9 @@ func Register(f SyntaxFeature) { if registeredPaths[f.Path] { panic("syntax: duplicate feature path: " + f.Path) } + if f.MinVersion != "" || len(f.Requires) > 0 { + panic("syntax: " + f.Path + " sets a version by hand; name its sdk/versions feature in Features instead (#1367)") + } registeredPaths[f.Path] = true registry = append(registry, f) } @@ -126,6 +152,7 @@ func ResolveAlias(path string) string { // All returns every registered feature, sorted by path. func All() []SyntaxFeature { + resolveVersions() out := make([]SyntaxFeature, len(registry)) copy(out, registry) sort.Slice(out, func(i, j int) bool { return out[i].Path < out[j].Path }) @@ -134,6 +161,7 @@ func All() []SyntaxFeature { // ByPrefix returns features whose path equals or starts with prefix+".". func ByPrefix(prefix string) []SyntaxFeature { + resolveVersions() var out []SyntaxFeature for _, f := range registry { if f.Path == prefix || strings.HasPrefix(f.Path, prefix+".") { @@ -146,6 +174,7 @@ func ByPrefix(prefix string) []SyntaxFeature { // ByPath returns the feature with the exact path, or nil. func ByPath(path string) *SyntaxFeature { + resolveVersions() for i := range registry { if registry[i].Path == path { return ®istry[i] @@ -180,6 +209,7 @@ func HasPrefix(prefix string) bool { // Substring rather than equality because a segment is often a compound of the // word the reader has in mind: `task` should find `workflow.user-task`. func BySegmentMatch(q string) []SyntaxFeature { + resolveVersions() if q == "" { return nil } @@ -205,6 +235,7 @@ func BySegmentMatch(q string) []SyntaxFeature { // Whole words, not substrings: keywords are prose, and a substring match on // "ant" would answer with every topic that mentions "grant". func ByKeyword(q string) []SyntaxFeature { + resolveVersions() q = strings.ToLower(q) if q == "" { return nil diff --git a/cmd/mxcli/syntax/registry_test.go b/cmd/mxcli/syntax/registry_test.go index 7d2139aaa9..8306b91a9f 100644 --- a/cmd/mxcli/syntax/registry_test.go +++ b/cmd/mxcli/syntax/registry_test.go @@ -140,8 +140,10 @@ func TestWriteText_SingleFeature(t *testing.T) { if !strings.Contains(out, "Example:") { t.Error("output missing Example section") } - if !strings.Contains(out, "Mendix 9.0.0+") { - t.Error("output missing MinVersion") + // From the version registry (#1367): a user task is 9.19.0, group + // targeting 11.2.0. The hand-written 9.0.0 this used to assert was wrong. + if !strings.Contains(out, "Mendix 9.19.0+") || !strings.Contains(out, "Mendix 11.2.0+") { + t.Errorf("output missing the registry versions:\n%s", out) } } diff --git a/cmd/mxcli/syntax/versions.go b/cmd/mxcli/syntax/versions.go new file mode 100644 index 0000000000..22f513a12d --- /dev/null +++ b/cmd/mxcli/syntax/versions.go @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: Apache-2.0 + +package syntax + +import ( + "sync" + + "github.com/mendixlabs/mxcli/sdk/versions" +) + +var resolveOnce sync.Once + +// resolveVersions fills each topic's MinVersion and Requires from the version +// registry, once, before any lookup returns a topic. A feature the registry +// does not have is left out here and reported by TestSyntaxFeaturesResolve. +func resolveVersions() { + resolveOnce.Do(func() { + reg, err := versions.Load() + if err != nil { + return + } + for i := range registry { + f := ®istry[i] + var primary *versions.SemVer + for _, key := range f.Features { + area, name, ok := cutFeatureKey(key) + if !ok { + continue + } + e, found := reg.Lookup(area, name) + if !found { + continue + } + f.Requires = append(f.Requires, Requirement{Feature: key, Since: e.MinVersion.String(), MDL: e.MDL}) + if primary == nil { + v := e.MinVersion + primary = &v + } + } + if primary != nil { + f.MinVersion = primary.String() + } + } + }) +} + +func cutFeatureKey(key string) (area, name string, ok bool) { + for i := 0; i < len(key); i++ { + if key[i] == '/' { + return key[:i], key[i+1:], i > 0 && i < len(key)-1 + } + } + return "", "", false +} diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 9f8d93d2cd..96a8e7a2e7 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -64,7 +64,7 @@ The tables below show exactly which features are available on each Mendix versio | Loop in branches | `LOOP inside IF/ELSE` | Yes | Yes | Yes | Yes | | Loops over a list, while loops | `LOOP $Item IN $List ...`, `WHILE ...` | 9.0.4+ | Yes | Yes | Yes | | Show page with arguments | `SHOW PAGE Module.Page (Param = $Value)` | 9.7+ | Yes | Yes | Yes | -| Calls in a task queue | `CALL MICROFLOW ... IN QUEUE ...`, `CALL JAVA ACTION ... IN QUEUE ...` | 9.10+ | Yes | Yes | Yes | +| Calls in a task queue | `CALL MICROFLOW ... IN QUEUE ... [RETRY FIXED (...)], [RETRY EXPONENTIAL (...)]`, `CALL JAVA ACTION ... IN QUEUE ... [RETRY ...]` | 9.10+ | Yes | Yes | Yes | | Find and filter by expression | `FIND $List WHERE ...`, `FILTER $List WHERE ...` | 9.0.3+ | Yes | Yes | Yes | | Call external action | `CALL EXTERNAL ACTION Module.Service.Action (...)` | 9.19+ (with arguments: --) | Yes (with arguments: 10.2+) | Yes | Yes | | Range of a list | `RANGE $List OFFSET ... LIMIT ...` | 9.24+ | Yes | Yes | Yes | diff --git a/docs-site/src/language/page-patterns.md b/docs-site/src/language/page-patterns.md index 2e1b000022..f648e2ab9b 100644 --- a/docs-site/src/language/page-patterns.md +++ b/docs-site/src/language/page-patterns.md @@ -13,7 +13,7 @@ CREATE PAGE MyModule.Customer_Overview FOLDER 'Customers' Layout: Atlas_Core.Atlas_Default ) { - DATAGRID dgCustomers (DataSource: DATABASE MyModule.Customer, PageSize: 20) { + DATAGRID dgCustomers (DataSource: DATABASE MyModule.Customer, PageSize: 20, itemSelection: Single) { COLUMN (Attribute: Name, Caption: 'Name') COLUMN (Attribute: Email, Caption: 'Email') COLUMN (Attribute: Status, Caption: 'Status') @@ -25,7 +25,7 @@ CREATE PAGE MyModule.Customer_Overview FOLDER 'Customers' ButtonStyle: Primary ) ACTIONBUTTON btnEdit (Caption: 'Edit', Action: SHOW PAGE MyModule.Customer_Edit) - ACTIONBUTTON btnDelete (Caption: 'Delete', Action: DELETE, ButtonStyle: Danger) + ACTIONBUTTON btnDelete (Caption: 'Delete', Action: DELETE $dgCustomers, ButtonStyle: Danger) } } }; @@ -223,7 +223,7 @@ CREATE PAGE HR.Employee_Overview FOLDER 'Employees' Layout: Atlas_Core.Atlas_Default ) { - DATAGRID dgEmployees (DataSource: DATABASE HR.Employee, PageSize: 20) { + DATAGRID dgEmployees (DataSource: DATABASE HR.Employee, PageSize: 20, itemSelection: Single) { COLUMN (Attribute: FullName, Caption: 'Name') COLUMN (Attribute: Department, Caption: 'Department') COLUMN (Attribute: HireDate, Caption: 'Hire Date') @@ -234,7 +234,7 @@ CREATE PAGE HR.Employee_Overview FOLDER 'Employees' ButtonStyle: Primary ) ACTIONBUTTON btnEdit (Caption: 'Edit', Action: SHOW PAGE HR.Employee_Edit) - ACTIONBUTTON btnDelete (Caption: 'Delete', Action: DELETE, ButtonStyle: Danger) + ACTIONBUTTON btnDelete (Caption: 'Delete', Action: DELETE $dgEmployees, ButtonStyle: Danger) } } }; diff --git a/docs-site/src/tools/project-brain.md b/docs-site/src/tools/project-brain.md index 08ce585713..6f2b42323f 100644 --- a/docs-site/src/tools/project-brain.md +++ b/docs-site/src/tools/project-brain.md @@ -199,6 +199,54 @@ implement it. Misfiling is not checked for slices: a slice spans modules by design. +### Before there is a project + +The brain does not need a model to exist: capture requirements from a +specification first, create the app afterwards. Name the brain's directory with +`-C`; no store command opens, or needs, the project. + +```bash +mxcli brain init -C ./App +mxcli brain capture "A planner can create a visit" -s 01-scheduling -a @Scheduling.Visit -C ./App +mxcli brain plan -C ./App +``` + +`-p app.mpr` still works and means the directory that `.mpr` is — or will be — +in. The two are alternatives; passing both is refused. + +`plan` and `check` then report what is true: nothing is built. + +``` +No model at /ws/App.mpr yet: 1 anchor(s) not checked, so nothing counts as built. + +SLICE BUILT PLANNED UNANCHORED +01-scheduling 0 1 + +0 of 1 requirements built, across 1 slice(s). +``` + +Every anchor is **unverifiable** — neither *not found* (that would report the +whole store as stale) nor built. `check` exits 0, and `--json` carries +`"model_absent": true` and an `unverifiable` count. Only a *missing* `.mpr` +means this: one that exists but will not open is still an error. + +### Then create the project in the same directory + +```bash +mxcli new App --version 11.12.1 +``` + +`mxcli new` refuses a non-empty directory with one exception: a directory +holding **only** a brain — `docs/brain/` written by mxcli and its queue in +`.mxcli/brain/` — is adopted. The project is created around it, the store and +the queue are left exactly as they were, and from then on `-C ./App` finds the +one `.mpr` in the directory and counts the same requirements against it. Any +other file in the directory, or a `docs/brain/` mxcli did not write, is still +refused. A file the project would overwrite is reported before anything moves. + +So the brain lives where the project will live, not in a workspace above it: +the directory is the project root from the first requirement onwards. + ## Open questions An open question is a decision that has not been made yet. @@ -233,6 +281,71 @@ Both `brain check` and `mxcli lint` report unanswered questions until one is resolved. A question nobody answers is the one kind of entry that gets more expensive the longer it sits. +## Where an entry came from + +```bash +mxcli brain capture "A planner can create a visit" -s 01-scheduling -a @Scheduling.Visit \ + --source docs/sample-prd.md#L18-21 -p app.mpr +``` + +`--source` records the document an entry was taken from, and optionally where +in it after a `#` — a line range, a heading, whatever locates it. It is rendered +on its own line under the anchors and is in the JSON (`staged --json`): + +```markdown +## A planner can create a visit + +Anchors: `@Scheduling.Visit` · id `72dfdd` · 2026-11-04 +Source: `docs/sample-prd.md#L18-21` +``` + +The path is stored **relative to the project directory** — type it relative to +where you are, or absolute — so it means the same on every checkout. The +document has to be **in the project**: a source outside it, or one that does not +exist, is refused, because provenance pointing at a file nobody else has is not +provenance. If the document later disappears, `brain check` fails with +`NO SOURCE`, as it does for an anchor that names nothing. + +`brain edit --source …` sets or changes it; an edit of the wording keeps it. +The source is provenance, not content: the same requirement found in two places +is still one entry. + +Older mxcli builds read a shard with sources without complaint — they see the +`Source:` line as the first line of the entry's text and keep it when they +rewrite the file. The queue is less forgiving: an older build that rewrites +`.mxcli/brain/staged.jsonl` (on a capture, drop or promote) drops the source of +the entries still queued there. + +## Editing an entry + +```bash +mxcli brain edit 66dcf4 "A planner can create a visit for one named engineer" -p app.mpr +mxcli brain edit a1b2c3 -a @Finance.ACT_Post -p app.mpr # re-anchor, keep the text +mxcli brain move 66dcf4 --slice 02-dispatch -p app.mpr +``` + +An entry's id is a handle — `promote`, `resolve` and prose cite it — so +changing the wording keeps the id, the entry's position in its file, and its +date. The text replaces the title and body; `-a`, if given, replaces the +anchors and is checked as `capture` checks it — with `-a` the text can be left +out. A staged entry is edited in the +queue, a committed one in its shard. + +Two things move an entry to another file, and only these: + +- `move --slice` on a requirement. A decision has no slice: giving it one would + make it a requirement, whose anchors stop failing when they go stale. +- Re-anchoring a decision into another module, so `check` does not report it + misfiled. A decision put in `project.md` on purpose (`promote --to project`) + stays there. + +An id is derived from the content once, at capture, and never again — so after +an edit it no longer matches the text. Duplicates are therefore found by what +an entry *says*, not by its id: capturing the edited wording again is refused +("Already committed as 66dcf4"), and capturing the *original* wording — which +hashes to the edited entry's id — is a new fact and gets an id of its own. An +edit that would make one entry say the same as another is refused. + ## When to capture a decision Capture needs a trigger, not good intentions. The reliable one: @@ -303,6 +416,9 @@ progress from 65/65 to 63/65 and the number was the only symptom. | `brain staged [--since ] [--slice ] [--fail-if-empty]` | Lists the queue with the shard each entry would land in | | `brain promote [--to ]` | Writes it into its shard | | `brain drop ` | Removes it from the queue or from its shard | +| `brain capture "" --source [#]` | Records which document, and where in it, the entry came from | +| `brain edit [""] [-a @Anchor]… [--source ]` | Rewrites it in place, staged or committed; the id does not change | +| `brain move --slice ` | Moves a requirement to another slice; the id does not change | | `brain capture "" --slice [-a @Anchor]…` | Queues a **requirement** of that slice | | `brain capture "" --open [-a @Anchor]…` | Queues an **open question**; anchors not checked | | `brain resolve ""` | Answers it, turning it into a decision in place | diff --git a/docs-wiki/bug-patterns/check-mxbuild-drift.md b/docs-wiki/bug-patterns/check-mxbuild-drift.md index a2ae1bbda0..df38c855e7 100644 --- a/docs-wiki/bug-patterns/check-mxbuild-drift.md +++ b/docs-wiki/bug-patterns/check-mxbuild-drift.md @@ -154,6 +154,17 @@ same expression, not two readings of the same intent — otherwise they drift on the first edit to either. This is the [[duplicate-resolver-drift]] class pointed at mxbuild. +**A builder side effect the definition does not state is invisible to check.** +The pluggable-widget check reads only the `.def.json`; when a builder setter also +writes a *second* property — filling a filter's `attributes` list flips +`attrChoice` to "linked" — the check evaluates the widget's visibility rules +against a configuration nobody stores, and calls the authored list hidden. The +fix is to state the side effect as a mode constant, so both read one source; and +because the drop-down filter got that fix while its three siblings kept the +constant, grep every definition for the same key when fixing one. These only +show with a project (the rules are lifted from the installed `.mpk`), so a +no-project test passes against the bug. + **Better still, run the real operation against a throwaway copy.** The vocabulary of an `ALTER … SET` is partly a switch in the mutator and partly the *stored* widget's own PropertyTypes, which belong to whatever package the project diff --git a/docs-wiki/bug-patterns/misleading-diagnostics.md b/docs-wiki/bug-patterns/misleading-diagnostics.md index d5d1261b12..9646c8e907 100644 --- a/docs-wiki/bug-patterns/misleading-diagnostics.md +++ b/docs-wiki/bug-patterns/misleading-diagnostics.md @@ -92,6 +92,14 @@ refuses it, and the cascade never starts — one error, at the mistake, quoting the author's own header back corrected. Dump the rule-context stack at each error first; a stack of just the program rule is the sign. +**A lint rule with several causes must not name only one.** The same failure +shows outside the parser: MPR008 told every overlapping pair that "each +statement needs its own @position", while an overlap involving a loop is a +spacing problem — the loop's box grows with its body — so authors added +annotations that were already there. The rule already measured each box by its +own stored size; asking *which kind* of box overlapped was enough to name the +loop, its size and the gap it needs. + **The reachable-form question.** Several of these are not really about wording: the reader's next question is "then how *do* I write it?", and a hint that names the working spelling answers it. Where there is no working spelling, saying so is diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 9be42e1d8f..2d7c45f063 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -573,7 +573,7 @@ it is for pages. | Call microflow on a queue | `call microflow Module.Name (Param = $value) in queue Module.Queue;` | Background execution; the queue must exist (CE1613), and the called microflow must return nothing, else CE7033 (**MDL088**) | | Call Java action on a queue | `call java action Module.Name (Param = $value) in queue Module.Queue;` | The Java action must `returns void`, else CE7038 | | Call nanoflow | `$Result = call nanoflow Module.Name (Param = $value);` | | -| Blank call argument | `call microflow Module.Name (A = $a, B = nothing);` | Keeps the parameter mapping with an **empty** expression — what Studio Pro stores for an argument field left blank. Not `B = empty` (the Mendix value `empty`, stored as that text) and not omitting `B` (no mapping at all). Call microflow / call nanoflow only. It exists so `describe` can round-trip such a call: Mendix reports a blank argument as **CE0127** "Missing argument" | +| Blank call argument | `call microflow Module.Name (A = $a, B = nothing);` | Keeps the parameter mapping with an **empty** expression — what Studio Pro stores for an argument field left blank. Not `B = empty` (the Mendix value `empty`, stored as that text) and not omitting `B` (no mapping at all). Call microflow / call nanoflow / call external action only. It exists so `describe` can round-trip such a call: Mendix reports a blank flow-call argument as **CE0127** "Missing argument"; on `call external action` it is how Studio Pro 11.16 stores an optional (nullable) OData parameter left blank, where 11.14 stored `empty` (ako/mxcli#1915) | | Call JS action | `$Result = call javascript action Module.Name (Param = $value);` | JavaScript action (nanoflow/microflow) | | Call Java action | `$Result = call java action Module.Name (Param = $value);` | Java action (microflow only) | | Call web service | `$Result = call web service Module.Service operation OperationName;` | Legacy SOAP; quoted refs are fallback for dangling raw IDs | @@ -619,7 +619,7 @@ it is for pages. | Shared annotation | `@annotation(id: n1, text: 'note')` then `@annotation(id: n1)` | ONE note wired to several activities, which is how Mendix stores it. Without the `id:` the two lines are two separate notes, even with identical text. The id is scoped to the flow being authored and is not stored (#1077) | | Stored annotation | `@annotation(existing: 'note')` | **`alter` fragments only.** Attaches the note of the stored flow whose text is `'note'` — it must be the only one — instead of creating a note. Never printed by `describe`, which declares every note of a whole flow (ako/mxcli#1695) | | Annotation geometry | `@annotation(text: 'note', position: (x, y), size: (w, h))` | The note's own place and box on the canvas. Both are omitted whenever they match what a rewrite re-derives — 100px above the activity, stacked 60px per extra note, at 200×50 — so an ordinary note stays on the short form | -| Free annotation | `@annotation 'text'` before `@position(...)` | Free-floating visual note preserved by order. A free note has no activity to be placed relative to, so DESCRIBE always emits its `position:` | +| Free annotation | `@annotation 'text'` before `@position(...)` | Free-floating visual note preserved by order. A free note has no activity to be placed relative to, so DESCRIBE always emits its `position:`. A note linked only to the start event or to another note is described this way too, after a `-- WARNING` line: MDL has no form for that link, so a rewrite drops it (ako/mxcli#1933) | | IF | `if condition then ... [else ...] end if;` | | | Enum split | `case $Var when Value then ... end case;` | Enumeration decision branches. Bare enum values (never quoted or qualified), one branch per value **including `(empty)`** (MDL056), no `else` (MDL008), no `AS` alias | | Type split | `split type $Var when Module.Entity then ... when (empty) then ... end split;` | Runtime specialization branches. Same `when ... then` shape as the enum split. Needs a branch per subtype **and** the base entity (CE0090); `when (empty) then` is the **null-object** flow, not a default, and cannot be omitted (CE0089). Legacy `case Module.Entity` / `else` still parse (MDL065 warns) | diff --git a/docs/11-proposals/PROPOSAL_external_action_include_associations.md b/docs/11-proposals/PROPOSAL_external_action_include_associations.md new file mode 100644 index 0000000000..bf8a32c5ce --- /dev/null +++ b/docs/11-proposals/PROPOSAL_external_action_include_associations.md @@ -0,0 +1,181 @@ +--- +title: Associations to Include on a Call External Action Argument +status: draft +--- + +# Proposal: Associations to Include on a Call External Action Argument + +**Status:** Draft +**Date:** 2026-11-03 +**Issue:** ako/mxcli#1914 (mendixlabs/mxcli#1721) + +## Problem Statement + +Studio Pro's *Call external action* activity has, per object argument, an +**Associations to include** tree: the associations whose objects are serialised +into the OData request together with the argument. A command DTO with a sub-DTO +linked by `CHANGE $Dto (Module.Quantity = $Qty)` reaches the service without the +sub-object unless `Quantity` is ticked. Nothing fails at build time; the field is +simply missing at runtime. + +#1922 made `create or modify` **carry** a stored tree onto the rewritten call +(`mdl/executor/external_action_carry.go`). What is still missing: + +- `describe microflow` does not print the tree, so a reader cannot see it and a + describe → edit → exec loop only keeps it by the carry, never by what it says. +- A **new** call, a call to a parameter that now binds another entity, or a + `drop` + `create` of the flow cannot author a tree at all. The carry has + nothing to carry. + +## BSON Structure + +Measured on Studio Pro 11.16.0, `ako/TestApp` `Odata.IncludeAssociationsMF` +(committed as `mdl/backend/modelsdk/testdata/TestApp.IncludeAssociationsMF.11_16.bson`), +the `CreateBuffer` call on `Odata.Bug1914`, with `Quantity` ticked and `Unit` +ticked under it: + +``` +Microflows$CallExternalAction + ParameterMappings: [3, + Microflows$ExternalActionParameterMapping + ParameterName: "command" + Argument: "$NewCreateBufferCommand" + ParameterType: DataTypes$ObjectType { Entity: "Odata.CreateBufferCommand" } + AdditionalAttributes: [2] -- 11.5.0, out of scope + IncludedAssociations: [2, + Microflows$IncludedAssociation + Association: "Odata.Quantity" -- BY_NAME, required + IncludedAssociations: [2, + Microflows$IncludedAssociation + Association: "Odata.Unit" + IncludedAssociations: [2] -- written empty, not omitted + IsParent: true ] + IsParent: true ] + Microflows$ExternalActionParameterMapping + ParameterName: "additional" … IncludedAssociations: [2] ] +``` + +- It is a **tree**, not a flag (the upstream issue calls it a boolean; it is not). + Each node names an association; the associations ticked under it are its + children, reached from the entity at that association's far end. +- `IsParent` is true when the object on *this* side of the tree is the + association's owner — its `ParentPointer`, i.e. the MDL `FROM` entity (see + CLAUDE.md, "Association Parent/Child Pointer Semantics"). +- Lists are marker-2 at both levels; a leaf writes its empty nested list. + The reader and writer already exist and match this shape + (`includedAssociationsFromGen` / `includedAssociationsToGen`); only the MDL + surface is missing. +- The control fixture `…IncludeAssociationsMF_Off.11_16.bson` (nothing ticked) + stores `IncludedAssociations: [2]` on every mapping; the writer omits the empty + list, which builds clean (#1073). + +## Proposed MDL Syntax + +A clause on the **argument**, because the tree belongs to one argument and +differs between arguments of the same call: + +```sql +$Response = call external action Odata.Bug1914.CreateBuffer( + command = $NewCreateBufferCommand + include associations (Odata.Quantity (Odata.Unit)), + additional = empty +); +``` + +- `include associations ( A [ ( B, C (D) ) ], … )` — a parenthesised list of + qualified association names; a name followed by its own `( … )` lists the + associations included under it. Nesting mirrors the stored tree one to one. +- `IsParent` is **derived**, not written: for each node, the entity on this side + is the argument's `ParameterType` entity at the root, and the far-end entity of + the parent node below it. `IsParent` is whether that entity is the + association's `FROM` end. +- A **self-association** (both ends the same entity) is **refused**: the + direction cannot be derived, and guessing it would serialise the wrong side. + The refusal names the association and says to tick it in Studio Pro; a stored + one still round-trips through the #1922 carry when the clause is omitted, and + `describe` prints a comment for it instead of a clause it could not re-execute. +- `describe microflow` prints the clause whenever the stored tree is non-empty, + so the round trip is exact and the #1922 carry becomes the fallback for a + script that omits the clause, rather than the only route. +- An **omitted** clause on an argument whose stored tree is non-empty still + **carries** that tree on `create or modify` (#1922), so a script written + before the clause existed never silently drops Studio Pro's settings — but + `check -p` and `exec` **warn**, naming the argument and the carried + associations, and say to state them or write `include associations ()`. The + warning is what tells an author who deleted the clause to remove the + associations that the deletion did nothing. +- `include associations ()` states "none" explicitly and clears the tree. + +### Removing associations + +The clause is the whole tree, so removing is restating it with less: + +```sql +-- stored: Quantity, with Unit under it — keep Quantity, drop Unit +command = $Cmd include associations (Odata.Quantity) + +-- drop them all +command = $Cmd include associations () +``` + +The workflow is describe → delete the entries no longer wanted → `create or +modify`. Deleting the clause itself removes nothing (it is carried, with the +warning above). + +One limitation follows from refusing self-associations: a tree holding a +self-association cannot be restated, so it cannot be **partly** removed from +MDL. `include associations ()` still clears all of it, self-association +included; removing only some entries of such a tree is a Studio Pro edit. + +## Implementation Plan + +| File | Change | +|------|--------| +| `mdl/grammar/domains/MDLMicroflow.g4` | `includeAssociationsClause` after the argument expression in the external-action argument rule; recursive `includedAssociation : qualifiedName (LPAREN includedAssociation (COMMA includedAssociation)* RPAREN)?` | +| `mdl/ast/ast_microflow.go` | `IncludeAssociations []*IncludedAssociationDef` (+ an "explicitly empty" flag) on the external-action argument | +| `mdl/visitor/…` | build the tree | +| `mdl/executor/cmd_microflows_builder_calls.go` | resolve each node: association exists, touches the entity on this side (else refuse, naming the entity), derive `IsParent`; skip the #1922 carry for an argument that states the clause | +| `mdl/executor/cmd_microflows_format_action.go` | print the clause from `ParameterMapping.IncludedAssociations` | +| `mdl/executor/validate_*.go` | the same resolution for `check -p --references` | +| `cmd/mxcli/syntax/features_integration.go` | document the clause | + +No reader/writer change: both already handle the tree. + +## Version Compatibility + +`Microflows$IncludedAssociation` and `ExternalActionParameterMapping.IncludedAssociations` +were introduced in **10.11.0** (`modelsdk/canon/mendixmodelsdk_type_versions.json`, +`modelsdk/version/metamodel_versions_gen.go`). A non-empty clause needs a version +gate at 10.11.0; the explicit empty clause writes nothing and needs none. + +## Test Plan + +- Parser: the nested clause, the empty clause, and a refusal for a name that is + not qualified. +- Builder: the fixture's tree rebuilt from MDL equals the stored one (association + names, nesting, `IsParent`), including a case where `IsParent` is false + (the argument entity is the association's `TO` end). +- Describe: the fixture describes with the clause; describe → exec → describe is + "Unchanged" against the fixture project. +- Refusal: an association that does not touch the entity on its side, and a + self-association. +- Warning: an omitted clause over a stored non-empty tree warns and carries it; + `include associations ()` clears it with no warning; a stored empty tree and + an omitted clause give neither. +- `mx check` on a fresh 10.11+ and 11.x app with the clause: 0 errors; the same + call without it at runtime omits the sub-object (the reporter's symptom). +- `mdl-examples/bug-tests/1914-external-action-include-associations.mdl`. + +## Decisions + +- **Spelling:** the per-argument `include associations ( … )` clause, not a + property list. ADR-0013's property list holds an *activity's* settings; this + setting belongs to one argument, and the clause keeps it next to that argument. +- **Self-associations:** refused (see Proposed MDL Syntax), not defaulted. +- **Omitted clause:** carried, with a warning (see Proposed MDL Syntax), not + carried silently. + +## Open Questions + +1. `AdditionalAttributes` (11.5.0, also per argument, also unauthored) is out of + scope here; it likely needs the same treatment and the same clause position. diff --git a/docs/11-proposals/README.md b/docs/11-proposals/README.md index 5bcacf7d94..0619593ba1 100644 --- a/docs/11-proposals/README.md +++ b/docs/11-proposals/README.md @@ -27,10 +27,10 @@ for display in this README): -## Active Proposals (118) +## Active Proposals (122) -### In Progress (partial) (16) +### In Progress (partial) (17) | Proposal | Status | Summary | |----------|--------|---------| @@ -41,6 +41,7 @@ for display in this README): | [Eval Framework for mxcli + Claude Code](proposal-eval-framework.md) | Partial | An evaluation framework that systematically tests how well Claude Code + mxcli handles real-world Mendix app generation tasks. | | [First-class expressions for expression-typed MDL properties](PROPOSAL_first_class_expressions.md) | Partial | A property that holds a Mendix expression but is declared as a quoted string | | [Implement `mxcli structure` Command](mxcli-structure-proposal.md) | Partial | mxcli is a Go CLI tool for working with Mendix projects. | +| [Module settings — describe and set the module version and module type](PROPOSAL_module_settings.md) | Partial | The General tab of Studio Pro's Module settings dialog has four fields: | | [Multi-Version Support: Consolidated Architecture & Status](MULTI_VERSION_SUPPORT.md) | Partial | Mendix projects vary along three versioning axes, and mxcli must handle all of them correctly: | | [mxcli Linter - Implementation Proposal (Revised)](PROPOSAL_mxcli_linter.md) | Partial | Add an extensible linting framework to mxcli that leverages the existing SQLite-based catalog system. | | [mxcli marketplace — Download & Manage Marketplace Modules](PROPOSAL_marketplace_modules.md) | Partial | Unblocking path #1 below has happened: the content API now returns a | @@ -60,7 +61,7 @@ for display in this README): | [MDL Language Critique and Beta Syntax Freeze](PROPOSAL_mdl_beta_syntax_freeze.md) | Accepted | Related: ADR-0003; resulting decisions ADR-0010, ADR-0011, ADR-0012; design-mdl-syntax skill, PROPOSAL_workflow_microflow_syntax_alignment.m | | [Size containers from their contents, not from a statement count](PROPOSAL_container_autosize.md) | Accepted | Closes the last open item of upstream #884 — | -### Proposed (40) +### Proposed (39) | Proposal | Status | Summary | |----------|--------|---------| @@ -90,7 +91,6 @@ for display in this README): | [SHOW/DESCRIBE JSON Structures](show-describe-json-structures.md) | Proposed | Document type: JsonStructures$JsonStructure | | [SHOW/DESCRIBE Menu Documents](show-describe-menu-documents.md) | Proposed | Document type: Menus$MenuDocument | | [SHOW/DESCRIBE Message Definitions](show-describe-message-definitions.md) | Proposed | Document type: MessageDefinitions$MessageDefinitionCollection | -| [SHOW/DESCRIBE Module Settings](show-describe-module-settings.md) | Proposed | Document type: Projects$ModuleSettings | | [SHOW/DESCRIBE Page Templates](show-describe-page-templates.md) | Proposed | Document type: Forms$PageTemplate | | [SHOW/DESCRIBE Published REST Services](show-describe-published-rest-services.md) | Proposed | Document type: rest$PublishedRestService | | [SHOW/DESCRIBE Queues (Task Queues)](show-describe-queues.md) | Proposed | Document type: Queues$Queue | @@ -105,7 +105,7 @@ for display in this README): | [VS Code Search — Quick Pick + Workspace Symbol](PROPOSAL_vscode_search.md) | Proposed | Full-text search exists in mxcli (mxcli search) but is only accessible via the terminal. | | [Workflow Improvements: ALTER WORKFLOW + Cross-References](PROPOSAL_workflow_improvements.md) | Proposed | Workflow support in mxcli has full CREATE/DESCRIBE/DROP/SHOW coverage with 13 activity types and BSON round-trip fidelity. | -### Draft (49) +### Draft (52) | Proposal | Status | Summary | |----------|--------|---------| @@ -113,10 +113,12 @@ for display in this README): | [Agent loop efficiency — making an mxcli session cost what the work costs](PROPOSAL_agent_loop_efficiency.md) | Draft | measured with diag loop-report, overturning the check-dominant assumption and | | [Architecture Graph Visualization (communities, layers, god-nodes)](PROPOSAL_architecture_graph_visualization.md) | Draft | Issue: TBD (file before implementation) | | [Association Mapping in IMPORT](PROPOSAL_import_associations.md) | Draft | Parent: PROPOSAL_mxcli_sql.md (Phase 3 extension) | +| [Associations to Include on a Call External Action Argument](PROPOSAL_external_action_include_associations.md) | Draft | Issue: ako/mxcli#1914 (mendixlabs/mxcli#1721) | | [Authorable layouts, page templates and building blocks — CREATE, not COPY](PROPOSAL_authorable_layouts.md) | Draft | An app built entirely through MDL cannot put anything in its own topbar. | | [Bulk Change Custom Widget Properties](PROPOSAL_bulk_widget_property_updates.md) | Draft | Custom widgets (pluggable widgets) in Mendix have complex nested property structures. | | [Bulk External Action Support from OData Contracts](PROPOSAL_external_actions_bulk_create.md) | Draft | Issue #143 requests importing all entities and actions from a consumed OData service. | | [Claude Code Prompt: Sprotty-based Domain Model Visualization PoC](proposal_sprotty_visualization.md) | Draft | I'm building a VS Code extension that visualizes Mendix Domain Models using Sprotty. | +| [Configurable Microflow and Nanoflow Layout](PROPOSAL_configurable_flow_layout.md) | Draft | mxcli lays out every flow it creates (create microflow / create nanoflow without | | [Deterministic AI-Capability Dataset & Report Generator](PROPOSAL_ai_capability_dataset.md) | Draft | We publish an AI Capabilities report (ai-capabilities-report-.html) that | | [Entity Positioning in ALTER ENTITY](PROPOSAL_entity_position.md) | Draft | Entity positions in the domain model canvas can only be set during create entity via the @position annotation. | | [Expression Type Checking for mxcli check](PROPOSAL_expression_type_checking.md) | Draft | mxcli check is currently a syntactic validator only. | @@ -148,6 +150,7 @@ for display in this README): | [Offline synchronization configuration — the one thing an offline profile needs that MDL cannot say](PROPOSAL_offline_sync_configuration.md) | Draft | mxcli can author an offline navigation profile — PhoneOffline, | | [OpenAPI import — generate response entities and wire operations to them](PROPOSAL_openapi_response_entities.md) | Draft | Issue: ako/mxcli#1234 (mendixlabs/mxcli#280) | | [Owning the modelsdk/gen codegen — why the vendored generator cannot be adopted as-is](PROPOSAL_codegen_ownership.md) | Draft | Constraint set by the maintainer (2026-08-13): the reflection data is an | +| [Playwright Check Through Studio Pro's MCP Browser](PROPOSAL_playwright_check_via_mcp.md) | Draft | mxcli playwright check gives a one-line text verdict per page of a running app. | | [Playwright Session Reuse and Lifecycle Control](PROPOSAL_playwright_session_reuse.md) | Draft | Builds on proposal-playwright-cli.md, which | | [Project brain — an opt-in store, in a user's Mendix project, for what mxcli cannot compute](PROPOSAL_project_brain.md) | Draft | Audience: users of mxcli building Mendix projects — not mxcli's own | | [RENAME with Reference Refactoring](PROPOSAL_rename_refactoring.md) | Draft | Renaming entities, microflows, pages, and modules is one of the most common refactoring operations. | diff --git a/mdl-examples/bug-tests/1003-dollar-dollar-in-describe-literal.mdl b/mdl-examples/bug-tests/1003-dollar-dollar-in-describe-literal.mdl new file mode 100644 index 0000000000..6a4c0368b3 --- /dev/null +++ b/mdl-examples/bug-tests/1003-dollar-dollar-in-describe-literal.mdl @@ -0,0 +1,31 @@ +mdl 1; +-- Bug #1003: `describe json structure` wrapped the sample in `$$ … $$`. A +-- snippet holding `$$` — a Teamcenter UID such as +-- "SR::N::BOMLine..10.7$$Otklm9GclKA" — ended the literal early, and +-- `mxcli check` on the describe output reported +-- token recognition error at: '$$ +-- +-- The lexer's DOLLAR_STRING has no escape: it ends at the first `$$`. Describe +-- now writes such a text as a single-quoted string (mdlDollarQuote / +-- mdlSourceBlock), which every position describe emits `$$` at accepts — +-- a Java/JavaScript action body and EXTRA CODE accept one since this fix. +-- +-- Below: the forms describe now writes for each position. They must parse and, +-- executed, store the text with its `$$`. + +create module BugTest1003; + +create or modify json structure BugTest1003.BomLine + sample '{ + "uid": "SR::N::BOMLine..10.7$$Otklm9GclKA", + "name": "it''s" +}'; + +create or modify java action BugTest1003.JA_Uid(N: String not null) returns String +as 'return "SR::N::10.7$$Otk" + N;' +extra code $$ +private static final String PREFIX = "x"; +$$; + +create or modify javascript action BugTest1003.JS_Uid() returns String +as 'return Promise.resolve("10.7$$Otk");'; diff --git a/mdl-examples/bug-tests/1005-describe-warning-cites-upstream-923.mdl b/mdl-examples/bug-tests/1005-describe-warning-cites-upstream-923.mdl new file mode 100644 index 0000000000..7839b71e20 --- /dev/null +++ b/mdl-examples/bug-tests/1005-describe-warning-cites-upstream-923.mdl @@ -0,0 +1,35 @@ +mdl 1; +-- ako/mxcli#1005: describe's warnings for a decision whose branches do not nest +-- (and its NOTE for a NORMALIZED rendering, and the unrepresented-merge warning) +-- cited "mxcli #923". In the ako/mxcli fork #923 is an unrelated MCP-backend +-- issue; the one meant is mendixlabs/mxcli#923 ("roundtrip for @merge does not +-- work"), and the text now names it with its owner. +-- +-- The graph below is mendixlabs/mxcli#923's own: +-- split1: true -> split2 false -> merge1 +-- split2: true -> merge1 false -> merge2 +-- merge1 -> log -> merge2 -> end +-- +-- To see a citation: +-- mxcli exec 1005-describe-warning-cites-upstream-923.mdl -p app.mpr +-- mxcli -p app.mpr -c "describe microflow BugTest1005.Crossing normalized" +-- The NOTE line ends "(mendixlabs/mxcli#923, Mode 3)", never "(mxcli #923, ...)". + +create module BugTest1005; + +create microflow BugTest1005.Crossing ($A: Boolean, $B: Boolean) +begin + if $A then + if $B then + join shared_log; + else + join after_log; + end if; + else + join shared_log; + end if; + merge shared_log; + log info node 'BugTest1005' 'shared'; + join after_log; + merge after_log; +end; diff --git a/mdl-examples/bug-tests/1206-parameter-supplies-flow-argument-before-10-1.mdl b/mdl-examples/bug-tests/1206-parameter-supplies-flow-argument-before-10-1.mdl new file mode 100644 index 0000000000..4056633f4b --- /dev/null +++ b/mdl-examples/bug-tests/1206-parameter-supplies-flow-argument-before-10-1.mdl @@ -0,0 +1,16 @@ +-- ako/mxcli#1206: before Mendix 10.1, a page's or snippet's own parameter +-- supplies a flow argument the way an enclosing data container does. Bisected +-- with this script: 9.24.30 and 10.0.0 build at 0 errors; 10.1.0, 10.2.0, +-- 10.3.1, 10.6.25, 10.12.19, 10.24.24 and 11.12.6 report CE1571 three times. +-- check -p now follows the project's version (0 reports below 10.1, 3 from it). +mdl 1; +create module BugTest1206; +create persistent entity BugTest1206.OrderX (Code: String(10)); +create nanoflow BugTest1206.DS_Order ($Order: BugTest1206.OrderX) returns BugTest1206.OrderX +begin + return $Order; +end; +create snippet BugTest1206.SN_Order ( Params: ( $Order: BugTest1206.OrderX ) ) +{ + dataview dvSnippet (datasource: nanoflow BugTest1206.DS_Order(Order = $Order)) { } +}; diff --git a/mdl-examples/bug-tests/1208-keep-9x-database-source-on-rewrite.mdl b/mdl-examples/bug-tests/1208-keep-9x-database-source-on-rewrite.mdl new file mode 100644 index 0000000000..1b523443b3 --- /dev/null +++ b/mdl-examples/bug-tests/1208-keep-9x-database-source-on-rewrite.mdl @@ -0,0 +1,15 @@ +-- ako/mxcli#1208: on a Mendix 9 project, a pluggable widget whose stored source +-- is Studio Pro 9's "Database" source (CustomWidgetDatabaseSource) was written +-- back as an XPath source when the widget was edited — describe prints both as +-- `database from E where …`. A rewrite now keeps the stored type when the WHERE +-- fits its constraint rows. Run on a 9.24 project holding such a grid; editing +-- only the sort keeps the DatabaseSource and its rows. +mdl 1; +create module BugTest1208; +create persistent entity BugTest1208.Cust (Name: String(100)); +create page BugTest1208.P_Cust ( Title: 'Customers', Layout: Atlas_Core.Atlas_Default ) +{ + datagrid dg (DataSource: database from BugTest1208.Cust where [Name = 'x'] sort by Name desc) { + column (Attribute: Name, Caption: 'Name') + } +}; diff --git a/mdl-examples/bug-tests/1211-forward-flow-call.mdl b/mdl-examples/bug-tests/1211-forward-flow-call.mdl new file mode 100644 index 0000000000..6954a4e26d --- /dev/null +++ b/mdl-examples/bug-tests/1211-forward-flow-call.mdl @@ -0,0 +1,14 @@ +-- ako/mxcli#1211: the check-syntax skill said every forward reference fails. +-- A forward CALL MICROFLOW is resolved by a plain `mxcli exec` (which rolls back +-- on failure) — this script runs as written. Only --continue-on-error, the REPL +-- and -c, which cannot roll back, refuse it (mendixlabs/mxcli#1405). +mdl 1; +create module BugTest1211; +create microflow BugTest1211.A () +begin + call microflow BugTest1211.B(); +end; +create microflow BugTest1211.B () +begin + return; +end; diff --git a/mdl-examples/bug-tests/1367-list-features-versions-and-areas.mdl b/mdl-examples/bug-tests/1367-list-features-versions-and-areas.mdl new file mode 100644 index 0000000000..b7fc4aa2d0 --- /dev/null +++ b/mdl-examples/bug-tests/1367-list-features-versions-and-areas.mdl @@ -0,0 +1,9 @@ +-- ako/mxcli#1367: `list features` had no `mxcli syntax` topic, a three-part +-- version was a parse error, and `in ` failed for the areas that are +-- lexer keywords (workflows, microflows, pages, …). All of these parse now; +-- `mxcli syntax features` documents them, and each topic's "Requires" lines +-- come from the same registry (sdk/versions/*.yaml). +mdl 1; +list features for version 10.20.0; +list features for version 10.24; +list features added since 11.0.0; diff --git a/mdl-examples/bug-tests/1397-tostring-decimal-short-attribute.mdl b/mdl-examples/bug-tests/1397-tostring-decimal-short-attribute.mdl new file mode 100644 index 0000000000..83fbff0e94 --- /dev/null +++ b/mdl-examples/bug-tests/1397-tostring-decimal-short-attribute.mdl @@ -0,0 +1,15 @@ +-- ako/mxcli#1397 (mendixlabs/mxcli#1515): check passed expressions the build +-- rejects or that misbehave at runtime. Status of each item on this branch: +-- formatDecimal($d, 2) -> E005 (CE0117) [already on main] +-- log '… {number} …' -> MDL-TEMPLATE02 (CE0712) [already on main] +-- rest body template '{}' -> MDL-TEMPLATE02 (CE0712) [already on main] +-- regex ending in a lone '\' -> MDL-REGEX01 [already on main] +-- toString() into a String(n) attribute -> warning from +-- `check -p … --references` (this fix); round it first, as below. +mdl 1; +create module BugTest1397; +create persistent entity BugTest1397.Item (Lbl: String(10)); +create microflow BugTest1397.SetLabel ($a: DateTime, $b: DateTime, $i: BugTest1397.Item) +begin + change $i (Lbl = toString(round(minutesBetween($a, $b)))); +end; diff --git a/mdl-examples/bug-tests/1684-filter-attributes-not-hidden.mdl b/mdl-examples/bug-tests/1684-filter-attributes-not-hidden.mdl new file mode 100644 index 0000000000..979928109c --- /dev/null +++ b/mdl-examples/bug-tests/1684-filter-attributes-not-hidden.mdl @@ -0,0 +1,37 @@ +mdl 1; +-- ako/mxcli#1684: `mxcli check -p … --references` warned MDL-WIDGET10 +-- property `attributes` is hidden when `attrChoice` is "auto" — the value will be ignored +-- on every text/number/date filter that names attributes. False: exec's +-- SetAttributeObjects stores attrChoice = "linked" whenever it fills the list. +-- The definitions declared attrChoice as the constant "auto"; they now carry an +-- `attributes` mode (attrChoice "linked") like the drop-down filter's +-- `attribute` mode from mendixlabs/mxcli#1519, so check reads what exec writes. +-- +-- The warning needs a project whose DataWidgets .mpk hides `attributes` under +-- "auto" — the rule is lifted from the installed package. +-- +-- mxcli check mdl-examples/bug-tests/1684-filter-attributes-not-hidden.mdl -p app.mpr --references +-- -- no MDL-WIDGET10 +-- mxcli exec … ; describe page BugFilter1684.Sale_Overview +-- -- the filters keep their Attributes; attrChoice is "linked" in the BSON + +create or modify module BugFilter1684; + +create persistent entity BugFilter1684.Sale ( + Name: String(100), + Amount: Decimal, + SoldOn: DateTime +); + +create page BugFilter1684.Sale_Overview (Title: 'Sales', Layout: Atlas_Core.Atlas_Default) { + datagrid dgSales (DataSource: database from BugFilter1684.Sale) { + controlbar { + textfilter tfName (Attributes: [Name]) + numberfilter nfAmount (Attributes: [Amount]) + datefilter dfSoldOn (Attributes: [SoldOn]) + } + column (Attribute: Name, Caption: 'Name') + -- Control: a column filter with no attributes stays attrChoice "auto". + column (Attribute: Amount, Caption: 'Amount') { numberfilter nfCol } + } +}; diff --git a/mdl-examples/bug-tests/1784-workflow-annotation-storage-name.mdl b/mdl-examples/bug-tests/1784-workflow-annotation-storage-name.mdl new file mode 100644 index 0000000000..5399b7f274 --- /dev/null +++ b/mdl-examples/bug-tests/1784-workflow-annotation-storage-name.mdl @@ -0,0 +1,19 @@ +-- ako/mxcli#1784: codegen named a workflow annotation Workflows$AnnotationActivity; +-- the writer wrote Workflows$Annotation. Measured on mxbuild 9.24.30 and 11.12.6: +-- this script (a start annotation and an activity annotation) is 0 errors with +-- Workflows$Annotation, and TypeCacheUnknownTypeException with AnnotationActivity. +mdl 1; +create module BugTest1784; +create persistent entity BugTest1784.Request ( Title: String(100) ); +create microflow BugTest1784.ACT_Validate ($WorkflowContext: BugTest1784.Request) +begin + log info node 'Wf' 'validate'; +end; +create workflow BugTest1784.WF_Request + parameter $WorkflowContext: BugTest1784.Request + display 'Request' + annotation 'Started from the request form' +begin + @annotation 'Checks the request' + call microflow BugTest1784.ACT_Validate; +end workflow; diff --git a/mdl-examples/bug-tests/1810-claim-task-assignee-by-version.mdl b/mdl-examples/bug-tests/1810-claim-task-assignee-by-version.mdl new file mode 100644 index 0000000000..7ecf39166e --- /dev/null +++ b/mdl-examples/bug-tests/1810-claim-task-assignee-by-version.mdl @@ -0,0 +1,13 @@ +-- ako/mxcli#1810: MDL-WORKFLOW10 suggested System.WorkflowUserTask_Assignees on +-- every version, and check --references accepted it on 9.24, which then failed +-- the build (CE1613 "no longer exists"). The association is one association, +-- renamed in 10.0: on 9.x it is WorkflowUserTask_Assignee (a Reference), from +-- 10.0 WorkflowUserTask_Assignees (a ReferenceSet). Measured on mxbuild 9.24.30: +-- the 9.x form below builds at 0 errors. This script is the 10.0+ form. +mdl 1; +create module BugTest1810; +create microflow BugTest1810.ACT_Claim ($Task: System.WorkflowUserTask) +begin + change $Task (System.WorkflowUserTask_Assignees = [%CurrentUser%]); + commit $Task; +end; diff --git a/mdl-examples/bug-tests/1833-mcp-trace-and-dial.mdl b/mdl-examples/bug-tests/1833-mcp-trace-and-dial.mdl index c43960cc48..7189ac889c 100644 --- a/mdl-examples/bug-tests/1833-mcp-trace-and-dial.mdl +++ b/mdl-examples/bug-tests/1833-mcp-trace-and-dial.mdl @@ -8,12 +8,25 @@ -- host.docker.internal whenever that name resolved. Docker Desktop for -- Windows adds it to the hosts file (the LAN address), so on the machine -- Studio Pro runs on the connection timed out: `dial tcp :7782`. +-- 3. A `localhost` URL was dialled through the resolver, which on the reporter's +-- machine answered with a LAN address first; Studio Pro on Windows listens +-- on 127.0.0.1 only. localhost now dials 127.0.0.1 first and keeps the +-- `Host: localhost` header Studio Pro requires. +-- +-- The headline claim — exec / -c with --mcp write the .mpr and mprcontents/ +-- and skip the .mpr.lock refusal — does not reproduce: the CLI with --mcp +-- calls ped_create_document and leaves every model file byte-identical, with +-- the lock present; a statement with no PED route (e.g. `create image +-- collection`) is refused, not written. cmd/mxcli/mcp_exec_no_file_write_test.go +-- pins both, with the fault forced back in as its control. -- -- To verify, with Studio Pro open on a throwaway app and its MCP server on 7782: -- mxcli exec 1833-mcp-trace-and-dial.mdl -p App.mpr --mcp http://localhost:7782/mcp --mcp-trace -- The trace lists `→ ped_create_document` and `→ ped_check_errors` under the --- statement, and the connection needs no --mcp-dial (on Windows with Docker --- Desktop installed too). The constant appears in Studio Pro's App Explorer. +-- statement, and the connection needs no --mcp-dial (on Windows, with or +-- without Docker Desktop). The constant appears in Studio Pro's App Explorer, +-- and a hash of App.mpr + mprcontents/ taken before and after is unchanged +-- until Studio Pro itself saves. mdl 1; diff --git a/mdl-examples/bug-tests/1865-dynamictext-content-not-hidden.mdl b/mdl-examples/bug-tests/1865-dynamictext-content-not-hidden.mdl new file mode 100644 index 0000000000..4062a02b72 --- /dev/null +++ b/mdl-examples/bug-tests/1865-dynamictext-content-not-hidden.mdl @@ -0,0 +1,28 @@ +mdl 1; +-- ako/mxcli#1865 (mendixlabs/mxcli#1707): `mxcli check -p` on 0.28.0 warned +-- MDL-WIDGET10 "property `content` is hidden … — the value will be ignored" on +-- a dynamictext's Content, top-level and in a Data grid 2 column, while the page +-- rendered it and mx check accepted it. No warning without a project. +-- +-- `dynamictext` is built by the page builder's own switch; on 0.28.0 a +-- `.mxcli/widgets/dynamictext.def.json` generated for a package widget could +-- take the keyword over in the registry, and check validated the built-in +-- against that widget's editorConfig rules. mendixlabs/mxcli#1463 refuses a +-- builder-owned keyword in the registry, so current main is clean; +-- TestCheck1865_DynamicTextContentIsNotHidden pins it. +-- +-- mxcli check mdl-examples/bug-tests/1865-dynamictext-content-not-hidden.mdl -p app.mpr --references +-- -- no MDL-WIDGET10 + +create or modify module BugDyn1865; + +create persistent entity BugDyn1865.Item (Name: String(100)); + +create page BugDyn1865.Item_Overview (Title: 'Items', Layout: Atlas_Core.Atlas_Default) { + dynamictext txtIntro (Content: 'All items') + datagrid dgItems (DataSource: database from BugDyn1865.Item) { + column (Attribute: Name, Caption: 'Name') { + dynamictext txtName (Content: 'Item {1}', ContentParams: ({1} = Name)) + } + } +}; diff --git a/mdl-examples/bug-tests/1866-expression-date-and-object-functions.mdl b/mdl-examples/bug-tests/1866-expression-date-and-object-functions.mdl new file mode 100644 index 0000000000..45588eb478 --- /dev/null +++ b/mdl-examples/bug-tests/1866-expression-date-and-object-functions.mdl @@ -0,0 +1,13 @@ +-- ako/mxcli#1866 (mendixlabs/mxcli#1708): isNew, addMonths and the trimTo +-- functions were missing from `mxcli syntax expression`. This is the example +-- the new `mxcli syntax microflow expressions functions` topic shows. +mdl 1; +create module BugTest1866; +create persistent entity BugTest1866.Order (OrderDate: DateTime); +create microflow BugTest1866.DueDate ($Order: BugTest1866.Order) returns DateTime +begin + if isNew($Order) then + return addMonths(trimToDays([%CurrentDateTime%]), 1); + end if; + return endOfMonth($Order/OrderDate); +end; diff --git a/mdl-examples/bug-tests/1911-drop-statements-documented.mdl b/mdl-examples/bug-tests/1911-drop-statements-documented.mdl new file mode 100644 index 0000000000..e6976bf7f3 --- /dev/null +++ b/mdl-examples/bug-tests/1911-drop-statements-documented.mdl @@ -0,0 +1,9 @@ +-- ako/mxcli#1911 (mendixlabs/mxcli#1716): DROP ASSOCIATION parsed and executed +-- but `mxcli syntax` documented it nowhere a reader would look. The statements +-- below are spelled exactly as the new `mxcli syntax drop` topic lists them. +mdl 1; +DROP ASSOCIATION IF EXISTS BugTest.Order_Customer; +DROP ENTITY IF EXISTS BugTest.OrderLine; +DROP MICROFLOW IF EXISTS BugTest.ACT_Old; +DROP NANOFLOW IF EXISTS BugTest.NF_Old; +DROP SNIPPET IF EXISTS BugTest.SN_Old; diff --git a/mdl-examples/bug-tests/1915-call-external-action-blank-argument.mdl b/mdl-examples/bug-tests/1915-call-external-action-blank-argument.mdl new file mode 100644 index 0000000000..8cf38636c1 --- /dev/null +++ b/mdl-examples/bug-tests/1915-call-external-action-blank-argument.mdl @@ -0,0 +1,25 @@ +mdl 1; +-- ako/mxcli#1915: describe microflow printed an unbound `call external action` +-- argument (Studio Pro 11.16 stores an optional OData parameter left blank as +-- `Argument: ""`) as `additional = )`, which does not parse, so every `mdl 1;` +-- create or modify of such a microflow was refused (MDL-V1-REBUILD "the +-- description does not parse"). +-- +-- The blank is now described, and accepted, as `= nothing` — the spelling +-- call microflow / call nanoflow already use for the same stored state. +-- `= empty` is a different state (11.14's, the Mendix value `empty`) and still +-- round-trips as itself. +-- +-- Needs a consumed OData service Odata.Bug1914 whose action CreateBuffer has a +-- required `command` and a nullable `additional`; `mxcli check` without -p +-- checks the syntax only. + +create microflow Odata.IncludeAssociationsBlank ($Cmd: String) +begin + call external action Odata.Bug1914.CreateBuffer(command = $Cmd, additional = nothing); +end; + +create microflow Odata.IncludeAssociationsEmpty ($Cmd: String) +begin + call external action Odata.Bug1914.CreateBuffer(command = $Cmd, additional = empty); +end; diff --git a/mdl-examples/bug-tests/1929-ended-branch-decision-one-merge.mdl b/mdl-examples/bug-tests/1929-ended-branch-decision-one-merge.mdl new file mode 100644 index 0000000000..745579e40a --- /dev/null +++ b/mdl-examples/bug-tests/1929-ended-branch-decision-one-merge.mdl @@ -0,0 +1,50 @@ +mdl 1; +-- ako/mxcli#1929 (mendixlabs/mxcli#1727): a decision one of whose branches +-- ends the flow is described with the decision's own merge as a label: +-- +-- end case; +-- join shared1; +-- merge shared1; +-- +-- Executing that description as a fresh create (drop + create) built the +-- decision's merge AND a second merge for the label, wired merge -> merge, and +-- each further round trip added `join junctionN; merge junctionN;`. +-- +-- Now the label's merge is the decision's merge when the join directly +-- follows the decision and nothing else reaches the label: one ExclusiveMerge, +-- stable across round trips. +-- +-- Verify: exec this, describe Probe.RT_Repro, drop it, exec the description, +-- describe again — the two descriptions are identical and hold no `junction`. + +create module Probe; +create enumeration Probe.Col (R 'R', G 'G', B 'B'); +create persistent entity Probe.Shape (C: Enumeration(Probe.Col)); + +create microflow Probe.RT_Repro ($S: Probe.Shape) returns Integer as $N +begin + declare $N Integer = 0; + case $S/C + when R then return 1; + when G then set $N = 2; + when (empty) then set $N = 3; + end case; + return $N; +end; + +-- The described form, as `describe` prints it: builds the same single merge. +create microflow Probe.RT_Described ($S: Probe.Shape) returns Integer as $N +begin + declare $N Integer = 0; + case $S/C + when R then + return 1; + when G then + set $N = 2; + when (empty) then + set $N = 3; + end case; + join shared1; + merge shared1; + return $N; +end; diff --git a/mdl-examples/bug-tests/1931-queued-call-retry-policy.mdl b/mdl-examples/bug-tests/1931-queued-call-retry-policy.mdl new file mode 100644 index 0000000000..e4a570fa89 --- /dev/null +++ b/mdl-examples/bug-tests/1931-queued-call-retry-policy.mdl @@ -0,0 +1,17 @@ +-- ako/mxcli#1931 (mendixlabs/mxcli#1729): a queued call's retry policy had no +-- MDL spelling, so a flow with one was locked for MDL. Measured on mxbuild +-- 11.12.6: both forms below build at 0 errors; Retries: 0 is CE7044. +mdl 1; +create module BugTest1931; +create task queue BugTest1931.SendQueue ( Parallelism: 1, ClusterWide: true ); +create microflow BugTest1931.Send ($Body: String) +begin + log info node 'Send' '{1}' with ({1} = $Body); +end; +create microflow BugTest1931.ACT_Enqueue () +begin + call microflow BugTest1931.Send(Body = 'x') in queue BugTest1931.SendQueue + retry exponential (Retries: 20, InitialInterval: 1, MaximumInterval: 60, IntervalType: Minutes); + call microflow BugTest1931.Send(Body = 'y') in queue BugTest1931.SendQueue + retry fixed (Retries: 5, Interval: 10, IntervalType: Seconds); +end; diff --git a/mdl-examples/bug-tests/1932-delete-grid-selection.mdl b/mdl-examples/bug-tests/1932-delete-grid-selection.mdl new file mode 100644 index 0000000000..23c7715886 --- /dev/null +++ b/mdl-examples/bug-tests/1932-delete-grid-selection.mdl @@ -0,0 +1,45 @@ +mdl 1; +-- ============================================================================ +-- ako/mxcli#1932 (mendixlabs/mxcli#1730) — Delete button in a grid control bar +-- ============================================================================ +-- +-- A Delete button in a Data grid 2 control bar deletes the grid's selection. +-- Studio Pro stores Forms$DeleteClientAction.SourceVariable = +-- Forms$PageVariable{Widget: "grid1"}. A bare `action: delete` there wrote no +-- source, which mx check reports as CE7005 "No value selection has been made", +-- and describe printed a stored source as plain `delete`, so a describe → exec +-- round trip dropped it. +-- +-- `delete $grid1` names the selection, as `$grid1` names it in a flow argument; +-- describe prints it back the same way. A bare `delete` inside a data view +-- still deletes the context object. The bare form in a control bar is refused +-- (MDL-PAGEDEL01) — see 1932-delete-in-control-bar-without-source.fail.mdl. +-- +-- Measured on mx 11.12.1: this script → 0 errors; the same script with the +-- source not written → CE7005 on both control-bar buttons. The grid needs a +-- selection: with itemSelection left at None, mx reports CE7001 "Widget should +-- be placed inside Data grid 2 'grid1' to use it as a source widget". +-- ============================================================================ + +create module Probe1932; + +create persistent entity Probe1932.Order ( + Name: String(200) +); + +create page Probe1932.Orders ( + params: ( $Order: Probe1932.Order ), + title: 'Orders', + layout: Atlas_Core.Atlas_Default +) { + datagrid grid1 (datasource: database from Probe1932.Order, itemSelection: Single) { + controlbar { + actionbutton btnDelete (caption: 'Delete', action: delete $grid1) + actionbutton btnDeleteClose (caption: 'Delete and close', action: delete $grid1 close page) + } + column (attribute: Name, caption: 'Name') + } + dataview dv (datasource: $Order) { + actionbutton btnDvDelete (caption: 'Delete', action: delete) + } +}; diff --git a/mdl-examples/bug-tests/1932-delete-in-control-bar-without-source.fail.mdl b/mdl-examples/bug-tests/1932-delete-in-control-bar-without-source.fail.mdl new file mode 100644 index 0000000000..5869932fb5 --- /dev/null +++ b/mdl-examples/bug-tests/1932-delete-in-control-bar-without-source.fail.mdl @@ -0,0 +1,31 @@ +mdl 1; +-- ============================================================================ +-- ako/mxcli#1932 (mendixlabs/mxcli#1730) — bare `delete` in a control bar +-- ============================================================================ +-- +-- NEGATIVE TEST: this script must FAIL `mxcli check`. +-- +-- A control bar has no context object, so a bare `delete` there has nothing +-- to delete: it was written with SourceVariable null and mx check reported +-- CE7005 "No value selection has been made", while check --references passed. +-- check now reports MDL-PAGEDEL01 and suggests `delete $grid1`; the working +-- form is in 1932-delete-grid-selection.mdl. +-- ============================================================================ + +create module Probe1932F; + +create persistent entity Probe1932F.Order ( + Name: String(200) +); + +create page Probe1932F.Orders ( + title: 'Orders', + layout: Atlas_Core.Atlas_Default +) { + datagrid grid1 (datasource: database from Probe1932F.Order) { + controlbar { + actionbutton btnDelete (caption: 'Delete', action: delete) + } + column (attribute: Name, caption: 'Name') + } +}; diff --git a/mdl-examples/bug-tests/1933-note-not-linked-to-an-activity.mdl b/mdl-examples/bug-tests/1933-note-not-linked-to-an-activity.mdl new file mode 100644 index 0000000000..91d4ce05e0 --- /dev/null +++ b/mdl-examples/bug-tests/1933-note-not-linked-to-an-activity.mdl @@ -0,0 +1,32 @@ +mdl 1; +-- ako/mxcli#1933 (mendixlabs/mxcli#1731): annotations not linked to an +-- activity were missing from `describe microflow`. +-- +-- A note linked to nothing was already described, as a free note (the first +-- one below). What vanished was a note whose only link ends where the +-- describer never prints an @annotation: the START EVENT, or ANOTHER NOTE. +-- Such a note is now described as a free note, preceded by a +-- `-- WARNING: the note '…' is linked to the start event, which MDL cannot +-- state …` line, because the link itself has no MDL form and a rewrite drops it. +-- +-- Free note syntax: an @annotation with text and position, BEFORE the first +-- statement's own @position. Linked notes keep their syntax: `@annotation 'x'`, +-- and `id:` to share one note between activities. +-- +-- Verify: exec this, then link Note1 to the start event in Studio Pro and +-- describe ACT_notes — Note1 is printed, with the warning. + +create module ExamplesModule; + +create microflow ExamplesModule.ACT_notes () +begin + @annotation(text: 'Note1', position: (120, 60), size: (200, 50)) + @position(314, 200) + @annotation(id: n1, text: 'Note2', position: (383, 89), size: (230, 50)) + declare $Variable Boolean = false; + @position(504, 200) + @annotation(id: n1) + declare $Variable_1 Boolean = false; + @position(700, 200) + return; +end; diff --git a/mdl-examples/bug-tests/1935-message-definition-renamed-attribute.mdl b/mdl-examples/bug-tests/1935-message-definition-renamed-attribute.mdl new file mode 100644 index 0000000000..c59a30a8d8 --- /dev/null +++ b/mdl-examples/bug-tests/1935-message-definition-renamed-attribute.mdl @@ -0,0 +1,27 @@ +-- ako/mxcli#1935 (mendixlabs/mxcli#1733): after RENAME ATTRIBUTE, describe of a +-- message definition printed the member under the old attribute name, and exec +-- of that output failed ("has no attribute"). Describe now names the attribute +-- the member references and keeps the stored exposed name, as below; and +-- check --references refuses a member naming an attribute the entity lacks. +mdl 1; +create module BugTest1935; +create persistent entity BugTest1935.Booking ( + Reference: String(200), + Code: String(200) +); +create message definition collection BugTest1935.MD_Booking +{ + definition BookingMessage for BugTest1935.Booking { + Reference, + Code + } +}; +alter entity BugTest1935.Booking rename attribute Reference to CustomerReference; +-- what describe prints now; re-executable, exposed name unchanged +create or modify message definition collection BugTest1935.MD_Booking +{ + definition BookingMessage for BugTest1935.Booking { + CustomerReference as 'Reference', + Code + } +}; diff --git a/mdl-examples/bug-tests/1939-send-delete-external-object.mdl b/mdl-examples/bug-tests/1939-send-delete-external-object.mdl new file mode 100644 index 0000000000..8d3cc94bc8 --- /dev/null +++ b/mdl-examples/bug-tests/1939-send-delete-external-object.mdl @@ -0,0 +1,38 @@ +mdl 1; +-- ako/mxcli#1939 (mendixlabs/mxcli#1735): Studio Pro's "Send external object" +-- and "Delete external object" activities had no MDL. A flow that saves an +-- external (OData) object could not be written — a plain commit on one is a +-- Studio Pro error — and describe printed `-- Unsupported action: +-- Microflows$SendExternalObject`, so the drop + create rebuild that +-- MDL-V1-REBUILD recommends deleted the activity silently. +-- +-- Run: mxcli exec 1939-send-delete-external-object.mdl -p app.mpr +-- Then: describe microflow OdataSend1939.ACT_Save prints the send; drop it and +-- create it from that description and it is still there; mx check: 0 errors +-- (measured on 11.12.6). `on error continue` on either statement is MDL076 +-- (CE6035). +-- +-- The contract's Definition set is updatable but not deletable, so a +-- `delete external object` on it is CE9435 — not exercised here. + +create or modify module OdataSend1939; + +create or modify constant OdataSend1939.SvcUrl + ( Type: string, DefaultValue: 'https://example.com/odata/v4/App/' ); + +create or modify consumed odata service OdataSend1939.App ( + ODataVersion: 'OData4', + MetadataUrl: './contracts/key-updatable-metadata.xml', + ServiceUrl: '@OdataSend1939.SvcUrl' +); + +create or modify external entities from OdataSend1939.App into OdataSend1939 entities (Definition); + +create or modify microflow OdataSend1939.ACT_Save ($Definition: OdataSend1939.Definition) +begin + change $Definition (Label = 'renamed'); + send external object $Definition refresh; + return; +end; + +describe microflow OdataSend1939.ACT_Save; diff --git a/mdl-examples/bug-tests/533-datasource-argument-error-site.mdl b/mdl-examples/bug-tests/533-datasource-argument-error-site.mdl new file mode 100644 index 0000000000..f5ba170fdf --- /dev/null +++ b/mdl-examples/bug-tests/533-datasource-argument-error-site.mdl @@ -0,0 +1,9 @@ +-- ako/mxcli#533: a malformed page data-source argument was reported only at +-- the `datasource` keyword ("no viable alternative at input 'datasource'"). +-- The parse error now also names the argument and its own line:column, e.g. +-- microflow M.DS_X(Param = ) -> "at line 3:48: `Param` has no value". +-- The well-formed spelling, which parses: +mdl 1; +create page BugTest533.P (title: 'x', layout: Atlas_Core.Atlas_Default) { + dataview DVMain (datasource: microflow BugTest533.DS_X(Param = $Value)) { } +}; diff --git a/mdl-examples/bug-tests/834-association-hop-off-context.mdl b/mdl-examples/bug-tests/834-association-hop-off-context.mdl new file mode 100644 index 0000000000..7d8073e321 --- /dev/null +++ b/mdl-examples/bug-tests/834-association-hop-off-context.mdl @@ -0,0 +1,39 @@ +-- ============================================================================ +-- Bug test: ako/mxcli#834 — a data source's first association does not touch +-- the context entity +-- ============================================================================ +-- +-- `$currentObject/Bug834b.Flag_Account/Bug834b.Flag` inside a data view over an +-- Order: Flag_Account joins Flag and Account, not Order. The builder checked +-- only that the association exists and that Flag is one of its ends, so exec +-- wrote an EntityRefStep that cannot be followed from an Order. +-- +-- This script parses (make check-mdl). Against a project it must be refused: +-- mxcli check this.mdl -p app.mpr --references +-- -> listview flags data source $currentObject/Bug834b.Flag_Account/…: +-- Bug834b.Flag_Account joins Bug834b.Flag and Bug834b.Account, and does +-- not start at Bug834b.Order … [MDL-ASSOCDS03] +-- mxcli exec this.mdl -p app.mpr +-- -> association "Bug834b.Flag_Account" in datasource … does not start at +-- Bug834b.Order; nothing is written. +-- Replace the path with $currentObject/Bug834b.Order_Account/Bug834b.Account/ +-- Bug834b.Flag_Account/Bug834b.Flag (the control) and both pass. +-- ============================================================================ + +mdl 1; + +create module Bug834b; + +create persistent entity Bug834b.Order (Code: String(20)); +create persistent entity Bug834b.Account (Name: String(50)); +create persistent entity Bug834b.Flag (Label: String(50)); +create association Bug834b.Order_Account from Bug834b.Order to Bug834b.Account; +create association Bug834b.Flag_Account from Bug834b.Flag to Bug834b.Account; + +create page Bug834b.P1 (Title: 'P', Layout: Atlas_Core.Atlas_Default, Params: ($Order: Bug834b.Order)) { + dataview dv (DataSource: $Order) { + listview flags (DataSource: $currentObject/Bug834b.Flag_Account/Bug834b.Flag) { + dynamictext x (Content: '{1}', ContentParams: ({1} = Label)) + } + } +}; diff --git a/mdl-examples/bug-tests/834-database-over-association-multi-hop.mdl b/mdl-examples/bug-tests/834-database-over-association-multi-hop.mdl new file mode 100644 index 0000000000..9a1113c0b9 --- /dev/null +++ b/mdl-examples/bug-tests/834-database-over-association-multi-hop.mdl @@ -0,0 +1,35 @@ +-- ============================================================================ +-- Bug test: ako/mxcli#834 — `database from $ctx/…` over two associations +-- ============================================================================ +-- +-- A List View's database source over associations read its path two segments +-- at a time (association, destination). The association source accepts the +-- short form `A1/A2` (mendixlabs/mxcli#1467); this one refused it: +-- entity "Bug834.Flag_Account" in datasource `database from +-- $currentObject/Bug834.Order_Account/Bug834.Flag_Account` does not exist +-- and wrote `A1/E1` with E1 off the association as stated. +-- +-- To verify: +-- 1. mxcli exec this script, then: describe page Bug834.P1 +-- -> listview flags (DataSource: database from $currentObject/ +-- Bug834.Order_Account/Bug834.Account/Bug834.Flag_Account/Bug834.Flag …) +-- 2. exec that output -> "Unchanged page Bug834.P1". +-- ============================================================================ + +mdl 1; + +create module Bug834; + +create persistent entity Bug834.Order (Code: String(20)); +create persistent entity Bug834.Account (Name: String(50)); +create persistent entity Bug834.Flag (Label: String(50)); +create association Bug834.Order_Account from Bug834.Order to Bug834.Account; +create association Bug834.Flag_Account from Bug834.Flag to Bug834.Account; + +create page Bug834.P1 (Title: 'P', Layout: Atlas_Core.Atlas_Default) { + listview orders (DataSource: database from Bug834.Order) { + listview flags (DataSource: database from $currentObject/Bug834.Order_Account/Bug834.Flag_Account where [Label != empty]) { + dynamictext x (Content: '{1}', ContentParams: ({1} = Label)) + } + } +}; diff --git a/mdl-examples/bug-tests/904-accessibilityhelper-describe-roundtrip.mdl b/mdl-examples/bug-tests/904-accessibilityhelper-describe-roundtrip.mdl new file mode 100644 index 0000000000..ca19b06429 --- /dev/null +++ b/mdl-examples/bug-tests/904-accessibilityhelper-describe-roundtrip.mdl @@ -0,0 +1,32 @@ +mdl 1; +-- ============================================================================ +-- ako/mxcli#904 — describe output of an AccessibilityHelper did not re-execute +-- ============================================================================ +-- +-- Two defects. (1) describe emitted `content { … }` for the widget's child +-- slot, refused by exec as MDL-WIDGET26 "`content` is not a container of +-- `accessibilityhelper` — it declares: attr, template". Already fixed by +-- mendixlabs/mxcli#1400: describe now emits `template { … }`, as below. +-- (2) An attributesList item's `attribute:` property (the HTML attribute's +-- NAME, a string) was read by check --references — which exec runs first — as +-- an attribute binding: "Attribute 'aria-label' is bound but there is no +-- enclosing data container providing entity context". The page below is the +-- form describe prints; it must check, execute, and describe back unchanged. +-- +-- Measured on mx 11.12.1: exec → describe → exec reports "Unchanged page", and +-- mx check reports 0 errors. +-- ============================================================================ + +create module Probe904; + +create page Probe904.P ( + title: 'P', + layout: Atlas_Core.Atlas_Default +) { + accessibilityhelper acc1 (targetSelector: '.mx-name-acc1') { + attr (attribute: 'aria-label', valueSourceType: 'text', valueText: 'Hello', attributeCondition: 'true') + template { + dynamictext accContent (content: 'Accessible content area') + } + } +}; diff --git a/mdl/ast/ast_microflow.go b/mdl/ast/ast_microflow.go index fb04d5a2a1..ee7e7add8c 100644 --- a/mdl/ast/ast_microflow.go +++ b/mdl/ast/ast_microflow.go @@ -627,6 +627,19 @@ type RollbackStmt struct { func (s *RollbackStmt) isMicroflowStatement() {} +// ExternalObjectStmt represents: SEND EXTERNAL OBJECT $Var [REFRESH] or +// DELETE EXTERNAL OBJECT $Var [REFRESH] — Studio Pro's "Send external object" +// and "Delete external object" activities (mendixlabs/mxcli#1735). +type ExternalObjectStmt struct { + Delete bool // DELETE EXTERNAL OBJECT; otherwise SEND + Variable string // The external object + RefreshInClient bool // Whether to refresh in client + Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + ErrorHandling *ErrorHandlingClause // Optional ON ERROR clause +} + +func (s *ExternalObjectStmt) isMicroflowStatement() {} + // RetrieveStmt represents: RETRIEVE $Var FROM Entity [WHERE condition] [SORT BY ...] [FIRST | [LIMIT n] [OFFSET n]] [ON ERROR ...] // or: RETRIEVE $Var FROM $Parent/Module.Association (association retrieve) type RetrieveStmt struct { @@ -752,12 +765,30 @@ type CallArgument struct { TemplateParams []TemplateParam } +// QueueRetry is the retry policy of a queued call (mendixlabs/mxcli#1729): +// +// IN QUEUE Mod.Q RETRY FIXED (Retries: 5, Interval: 10, IntervalType: Minutes) +// IN QUEUE Mod.Q RETRY EXPONENTIAL (Retries: 20, InitialInterval: 1, +// MaximumInterval: 60, IntervalType: Minutes) +// +// Kind is "Fixed" or "Exponential"; IntervalType is Seconds, Minutes or Hours. +// The visitor requires every property of the kind, so nothing is defaulted. +type QueueRetry struct { + Kind string + Retries int + Interval int + InitialInterval int + MaximumInterval int + IntervalType string +} + // CallMicroflowStmt represents: [$Result =] CALL MICROFLOW Name (args) [ON ERROR ...] type CallMicroflowStmt struct { OutputVariable string // Optional output variable MicroflowName QualifiedName // Microflow to call Arguments []CallArgument // Arguments Queue *QualifiedName // Optional IN QUEUE clause (task queue to run the call on) + QueueRetry *QueueRetry // Optional RETRY clause after IN QUEUE ErrorHandling *ErrorHandlingClause // Optional ON ERROR clause Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation } @@ -781,6 +812,7 @@ type CallJavaActionStmt struct { ActionName QualifiedName // Java action name Arguments []CallArgument // Arguments Queue *QualifiedName // Optional IN QUEUE clause (task queue to run the call on) + QueueRetry *QueueRetry // Optional RETRY clause after IN QUEUE ErrorHandling *ErrorHandlingClause // Optional ON ERROR clause Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation } diff --git a/mdl/ast/ast_page_v3.go b/mdl/ast/ast_page_v3.go index e4e6a7b124..2a512cce09 100644 --- a/mdl/ast/ast_page_v3.go +++ b/mdl/ast/ast_page_v3.go @@ -342,6 +342,10 @@ type ActionV3 struct { LinkVariable string LinkAttribute string OutcomeValue string // For COMPLETE_TASK + // For DELETE $source: the object deleted when it is not the context + // object — a widget's selection or a page/snippet parameter, written + // without the "$" (ako/mxcli#1932). Empty is the context object. + SourceVariable string // Settings is the action's `with ( … )` list, nil when none was written. Settings *ActionSettingsV3 // Returns is a flow call's `returns ( … )` list: Studio Pro's "Return value diff --git a/mdl/backend/mcp/client.go b/mdl/backend/mcp/client.go index 9194e79fcb..3856d613e3 100644 --- a/mdl/backend/mcp/client.go +++ b/mdl/backend/mcp/client.go @@ -93,7 +93,8 @@ func NewClient(opts ClientOptions) (*Client, error) { // dialPlan is the decision behind the client's dialer: the TCP addresses to // try, in order. An explicit --mcp-dial is the only one. Otherwise the URL's -// own host comes first, and a localhost/127.0.0.1 endpoint falls back to +// own host comes first (preceded by 127.0.0.1 when that host is localhost), +// and a localhost/127.0.0.1 endpoint falls back to // host.docker.internal: when that name resolves, so the same command // works from a devcontainer, where loopback is the container and Studio Pro is // on the Docker host. @@ -112,8 +113,18 @@ func dialPlan(urlHost, explicit string, dockerResolves bool) []string { if err != nil { // no port h, port = urlHost, "80" } - plan := []string{net.JoinHostPort(h, port)} - if (h == "localhost" || h == "127.0.0.1") && dockerResolves { + var plan []string + localhost := strings.EqualFold(h, "localhost") + if localhost { + // Studio Pro on Windows listens on 127.0.0.1 only, and a resolver may + // answer localhost with a LAN address first, so a plain + // `--mcp http://localhost:/mcp` timed out (mendixlabs/mxcli#1684). + // Only the dial address changes: the request URL, and so the Host + // header Studio Pro insists on, stays "localhost". + plan = append(plan, net.JoinHostPort("127.0.0.1", port)) + } + plan = append(plan, net.JoinHostPort(h, port)) + if (localhost || h == "127.0.0.1") && dockerResolves { plan = append(plan, net.JoinHostPort(dockerHostName, port)) } return plan diff --git a/mdl/backend/mcp/client_dial_test.go b/mdl/backend/mcp/client_dial_test.go index dddc00ad62..7241f9b3fd 100644 --- a/mdl/backend/mcp/client_dial_test.go +++ b/mdl/backend/mcp/client_dial_test.go @@ -6,7 +6,10 @@ import ( "context" "errors" "net" + "net/http" + "net/http/httptest" "strings" + "sync" "testing" "time" ) @@ -19,12 +22,18 @@ func TestDialPlan(t *testing.T) { dockerable bool want []string }{ - {"localhost where the docker name resolves: loopback, then the gateway", "localhost:7784", "", true, []string{"localhost:7784", "host.docker.internal:7784"}}, - {"localhost where it does not (macOS host, Issue 8)", "localhost:7784", "", false, []string{"localhost:7784"}}, + // localhost dials 127.0.0.1 first: Studio Pro on Windows listens on + // 127.0.0.1 only, and a resolver that answers localhost with a LAN + // address first made a plain `--mcp http://localhost:7782/mcp` time out + // (mendixlabs/mxcli#1684). The resolver's own answer stays as the next + // try, for a server bound to ::1 only. + {"localhost where the docker name resolves: 127.0.0.1, localhost, then the gateway", "localhost:7784", "", true, []string{"127.0.0.1:7784", "localhost:7784", "host.docker.internal:7784"}}, + {"localhost where it does not (macOS host, Issue 8)", "localhost:7784", "", false, []string{"127.0.0.1:7784", "localhost:7784"}}, + {"localhost is matched without regard to case", "LocalHost:7784", "", false, []string{"127.0.0.1:7784", "LocalHost:7784"}}, {"127.0.0.1 where the docker name resolves", "127.0.0.1:7782", "", true, []string{"127.0.0.1:7782", "host.docker.internal:7782"}}, {"127.0.0.1 where it does not", "127.0.0.1:7782", "", false, []string{"127.0.0.1:7782"}}, {"another host has no fallback", "myhost:9000", "", true, []string{"myhost:9000"}}, - {"no port is port 80", "localhost", "", true, []string{"localhost:80", "host.docker.internal:80"}}, + {"no port is port 80", "localhost", "", true, []string{"127.0.0.1:80", "localhost:80", "host.docker.internal:80"}}, {"--mcp-dial is the only address", "localhost:7782", "127.0.0.1:7782", true, []string{"127.0.0.1:7782"}}, } for _, c := range cases { @@ -98,3 +107,33 @@ func TestClient_LocalhostReachesStudioProOnTheHostWhenTheDockerNameResolves(t *t t.Fatalf("a Studio Pro listening on loopback was not reached: %v", err) } } + +// The dial goes to 127.0.0.1, but the request still says Host: localhost — +// Studio Pro answers HTTP 400 "Invalid Hostname" to any other Host header +// (mendixlabs/mxcli#1684), so the dial address must never leak into the URL. +func TestClient_LocalhostKeepsTheLocalhostHostHeader(t *testing.T) { + var mu sync.Mutex + var hosts []string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + hosts = append(hosts, r.Host) + mu.Unlock() + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-06-18","serverInfo":{"name":"fake","version":"1"}}}`)) + })) + defer srv.Close() + port := srv.URL[strings.LastIndex(srv.URL, ":")+1:] + c, err := NewClient(ClientOptions{URL: "http://localhost:" + port + "/mcp", Timeout: 10 * time.Second}) + if err != nil { + t.Fatal(err) + } + _, _ = c.Initialize() + mu.Lock() + defer mu.Unlock() + if len(hosts) == 0 { + t.Fatal("the server on 127.0.0.1 received no request") + } + if hosts[0] != "localhost:"+port { + t.Errorf("Host header = %q, want %q", hosts[0], "localhost:"+port) + } +} diff --git a/mdl/backend/modelsdk/domainmodel.go b/mdl/backend/modelsdk/domainmodel.go index 73dca645ee..3d6a05b881 100644 --- a/mdl/backend/modelsdk/domainmodel.go +++ b/mdl/backend/modelsdk/domainmodel.go @@ -40,7 +40,7 @@ func (b *Backend) ListDomainModels() ([]*domainmodel.DomainModel, error) { b.populateViewEntityOql(out) // The System module is virtual (not stored in the project); inject its domain // model so platform entities (System.WorkflowUserTask, User, …) resolve. - out = append(out, buildSystemDomainModel()) + out = append(out, buildSystemDomainModel(b.systemVersionAtLeast())) b.resolveInheritedPersistability(b.persistabilityIndexOf(units), out...) return out, nil } @@ -281,7 +281,7 @@ func (b *Backend) GetDomainModel(moduleID model.ID) (*domainmodel.DomainModel, e // The System module is virtual — its domain model is not stored in the project, // so serve the injected one (matching ListDomainModels) so DESCRIBE System.* and // any lookup of a platform entity's domain model resolves instead of erroring. - if sys := buildSystemDomainModel(); sys.ContainerID == moduleID { + if sys := buildSystemDomainModel(b.systemVersionAtLeast()); sys.ContainerID == moduleID { b.resolveInheritedPersistability(b.lazyPersistabilityIndex("", nil), sys) return sys, nil } diff --git a/mdl/backend/modelsdk/domainmodel_get_cost_test.go b/mdl/backend/modelsdk/domainmodel_get_cost_test.go index b43c69d881..49db6ac554 100644 --- a/mdl/backend/modelsdk/domainmodel_get_cost_test.go +++ b/mdl/backend/modelsdk/domainmodel_get_cost_test.go @@ -31,7 +31,7 @@ func TestGetDomainModel_ReadsOnlyItsModulesDomainModel(t *testing.T) { } stored := 0 for _, dm := range all { - if dm.ContainerID != buildSystemDomainModel().ContainerID { + if dm.ContainerID != buildSystemDomainModel(nil).ContainerID { stored++ } } diff --git a/mdl/backend/modelsdk/microflow_external_object_test.go b/mdl/backend/modelsdk/microflow_external_object_test.go new file mode 100644 index 0000000000..36b27fe127 --- /dev/null +++ b/mdl/backend/modelsdk/microflow_external_object_test.go @@ -0,0 +1,86 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "testing" + + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" + "go.mongodb.org/mongo-driver/v2/bson" +) + +// mendixlabs/mxcli#1735 (ako/mxcli#1939). The reference is Studio Pro 11.16's +// own Odata.ODataSendDeleteMF in ako/TestApp (commit 31a34ca, "odata send +// delete example"): its two activities, verbatim but for $ID. +var ( + studioProSend = bson.D{ + {Key: "$ID", Value: "send-1"}, + {Key: "$Type", Value: "Microflows$SendExternalObject"}, + {Key: "ErrorHandlingType", Value: "Rollback"}, + {Key: "RefreshInClient", Value: false}, + {Key: "VariableNameToBeSent", Value: "NewOrders"}, + } + studioProDelete = bson.D{ + {Key: "$ID", Value: "del-1"}, + {Key: "$Type", Value: "Microflows$DeleteExternalObject"}, + {Key: "DeleteVariableName", Value: "NewOrders"}, + {Key: "ErrorHandlingType", Value: "Rollback"}, + {Key: "RefreshInClient", Value: false}, + } +) + +// The reader takes Studio Pro's activities as the two actions, not as +// unknown ones describe prints as a comment. +func TestExternalObject_ReadsStudioProActivities(t *testing.T) { + if s, ok := decodeAction(t, studioProSend).(*microflows.SendExternalObjectAction); !ok || s.Variable != "NewOrders" || + s.ErrorHandlingType != microflows.ErrorHandlingTypeRollback || s.RefreshInClient { + t.Errorf("send read as %#v", decodeAction(t, studioProSend)) + } + if d, ok := decodeAction(t, studioProDelete).(*microflows.DeleteExternalObjectAction); !ok || d.Variable != "NewOrders" || + d.ErrorHandlingType != microflows.ErrorHandlingTypeRollback || d.RefreshInClient { + t.Errorf("delete read as %#v", decodeAction(t, studioProDelete)) + } +} + +// What mxcli writes for `send external object $NewOrders;` and `delete +// external object $NewOrders;` is Studio Pro's property set with Studio Pro's +// values — the error handling stored as Rollback, not left out as a plain +// delete's is. +func TestExternalObject_WritesStudioProPropertySet(t *testing.T) { + send := µflows.SendExternalObjectAction{Variable: "NewOrders"} + send.ID = model.ID("send-1") + del := µflows.DeleteExternalObjectAction{Variable: "NewOrders"} + del.ID = model.ID("del-1") + for _, tc := range []struct { + name string + got bson.D + want bson.D + }{ + {"send", encodeElement(t, microflowActionToGen(send)), studioProSend}, + {"delete", encodeElement(t, microflowActionToGen(del)), studioProDelete}, + } { + t.Run(tc.name, func(t *testing.T) { + want := map[string]any{} + for _, e := range tc.want { + if e.Key != "$ID" { + want[e.Key] = e.Value + } + } + got := map[string]any{} + for _, e := range tc.got { + if e.Key != "$ID" { + got[e.Key] = e.Value + } + } + if len(got) != len(want) { + t.Fatalf("property set = %v, want Studio Pro's %v", got, want) + } + for k, v := range want { + if got[k] != v { + t.Errorf("%s = %v, want Studio Pro's %v (got %v)", k, got[k], v, got) + } + } + }) + } +} diff --git a/mdl/backend/modelsdk/microflow_read_actions.go b/mdl/backend/modelsdk/microflow_read_actions.go index ebf67374ca..71a24bdbc8 100644 --- a/mdl/backend/modelsdk/microflow_read_actions.go +++ b/mdl/backend/modelsdk/microflow_read_actions.go @@ -109,6 +109,24 @@ func actionFromGen(el element.Element) microflows.MicroflowAction { out.ID = model.ID(a.ID()) return out + case *genMf.SendExternalObject: + out := µflows.SendExternalObjectAction{ + Variable: a.VariableNameToBeSent(), + RefreshInClient: a.RefreshInClient(), + ErrorHandlingType: microflows.ErrorHandlingType(a.ErrorHandlingType()), + } + out.ID = model.ID(a.ID()) + return out + + case *genMf.DeleteExternalObject: + out := µflows.DeleteExternalObjectAction{ + Variable: a.DeleteVariableName(), + RefreshInClient: a.RefreshInClient(), + ErrorHandlingType: microflows.ErrorHandlingType(a.ErrorHandlingType()), + } + out.ID = model.ID(a.ID()) + return out + case *genMf.RollbackAction: out := µflows.RollbackObjectAction{ RollbackVariable: a.RollbackVariableName(), @@ -1614,10 +1632,36 @@ func queueSettingsFromRaw(raw bson.Raw) *microflows.QueueSettings { qs := µflows.QueueSettings{Queue: rawStr(doc, "Queue")} if v, err := doc.LookupErr("Retry"); err == nil && v.Type != bson.TypeNull { qs.Retry = v + if rd, ok := v.DocumentOK(); ok { + qs.RetryPolicy = queueRetryFromRaw(rd) + } } return qs } +// queueRetryFromRaw decodes a stored Queues$QueueRetry, or nil for a type this +// build does not know — the raw value is still carried, so the rewrite guard +// refuses rather than drops it. +func queueRetryFromRaw(doc bson.Raw) *microflows.QueueRetry { + num := func(key string) int { n, _ := rawInteger(doc, key); return n } + r := µflows.QueueRetry{ + Retries: num("Retries"), + IntervalType: rawStr(doc, "IntervalType"), + } + switch rawStr(doc, "$Type") { + case "Queues$QueueFixedRetry": + r.Kind = microflows.QueueRetryFixed + r.Interval = num("Interval") + case "Queues$QueueExponentialRetry": + r.Kind = microflows.QueueRetryExponential + r.InitialInterval = num("InitialInterval") + r.MaximumInterval = num("MaximumInterval") + default: + return nil + } + return r +} + // closePageCount reads how many pages a close-page action closes. Mendix stores // it in NumberOfPagesToClose, a string ("" is the default, one page) — the int // NumberOfPages was deleted in 8.11. The int is read only when the string is diff --git a/mdl/backend/modelsdk/microflow_write.go b/mdl/backend/modelsdk/microflow_write.go index 9096454354..bd2c09b3ce 100644 --- a/mdl/backend/modelsdk/microflow_write.go +++ b/mdl/backend/modelsdk/microflow_write.go @@ -588,6 +588,22 @@ func microflowActionToGen(action microflows.MicroflowAction) element.Element { g.SetDeleteVariableName(a.DeleteVariable) g.SetRefreshInClient(a.RefreshInClient) return g + case *microflows.SendExternalObjectAction: + // Microflows$SendExternalObject (mendixlabs/mxcli#1735). + g := genMf.NewSendExternalObject() + g.SetID(element.ID(a.ID)) + g.SetVariableNameToBeSent(a.Variable) + g.SetRefreshInClient(a.RefreshInClient) + g.SetErrorHandlingType(orDefault(string(a.ErrorHandlingType), "Rollback")) + return g + case *microflows.DeleteExternalObjectAction: + // Microflows$DeleteExternalObject (mendixlabs/mxcli#1735). + g := genMf.NewDeleteExternalObject() + g.SetID(element.ID(a.ID)) + g.SetDeleteVariableName(a.Variable) + g.SetRefreshInClient(a.RefreshInClient) + g.SetErrorHandlingType(orDefault(string(a.ErrorHandlingType), "Rollback")) + return g case *microflows.RollbackObjectAction: g := genMf.NewRollbackAction() g.SetID(element.ID(a.ID)) @@ -1164,9 +1180,38 @@ func queueSettingsToGen(qs *microflows.QueueSettings) element.Element { } g := newElem("Queues$QueueSettings", string(qs.ID)) addStr(g, "Queue", qs.Queue) + if r := queueRetryToGen(qs.RetryPolicy); r != nil { + addPart(g, "Retry", r) + } return g } +// queueRetryToGen builds the Queues$QueueFixedRetry / Queues$QueueExponentialRetry +// a `retry fixed|exponential (…)` clause states (mendixlabs/mxcli#1729), with +// exactly the properties the type has. nil — no retry — leaves Retry to the +// registered null default. +func queueRetryToGen(r *microflows.QueueRetry) element.Element { + if r == nil { + return nil + } + switch r.Kind { + case microflows.QueueRetryFixed: + g := newElem("Queues$QueueFixedRetry", string(r.ID)) + addInt32(g, "Retries", int32(r.Retries)) + addInt32(g, "Interval", int32(r.Interval)) + addStr(g, "IntervalType", r.IntervalType) + return g + case microflows.QueueRetryExponential: + g := newElem("Queues$QueueExponentialRetry", string(r.ID)) + addInt32(g, "Retries", int32(r.Retries)) + addInt32(g, "InitialInterval", int32(r.InitialInterval)) + addInt32(g, "MaximumInterval", int32(r.MaximumInterval)) + addStr(g, "IntervalType", r.IntervalType) + return g + } + return nil +} + func newElem(typeName, id string) *element.Base { b := &element.Base{} b.SetTypeName(typeName) diff --git a/mdl/backend/modelsdk/queue_retry_write_test.go b/mdl/backend/modelsdk/queue_retry_write_test.go new file mode 100644 index 0000000000..e19eb0dc0e --- /dev/null +++ b/mdl/backend/modelsdk/queue_retry_write_test.go @@ -0,0 +1,71 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/modelsdk/codec" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// mendixlabs/mxcli#1729 (ako/mxcli#1931): `in queue … retry fixed|exponential +// (…)` writes a Queues$QueueFixedRetry / Queues$QueueExponentialRetry under the +// call's QueueSettings, with exactly the properties the type has, and the +// reader decodes it back so describe can print it. +func TestQueueRetryWrittenAndReadBack(t *testing.T) { + cases := map[string]struct { + in *microflows.QueueRetry + typ string + present []string + }{ + "fixed": { + µflows.QueueRetry{Kind: microflows.QueueRetryFixed, Retries: 5, Interval: 10, IntervalType: "Minutes"}, + "Queues$QueueFixedRetry", []string{"Retries", "Interval", "IntervalType"}, + }, + "exponential": { + µflows.QueueRetry{Kind: microflows.QueueRetryExponential, Retries: 20, InitialInterval: 1, MaximumInterval: 60, IntervalType: "Seconds"}, + "Queues$QueueExponentialRetry", []string{"Retries", "InitialInterval", "MaximumInterval", "IntervalType"}, + }, + } + for name, c := range cases { + t.Run(name, func(t *testing.T) { + el := queueSettingsToGen(µflows.QueueSettings{Queue: "Ops.SendQueue", RetryPolicy: c.in}) + raw, err := (&codec.Encoder{}).Encode(el) + if err != nil { + t.Fatalf("encode: %v", err) + } + retry, ok := bson.Raw(raw).Lookup("Retry").DocumentOK() + if !ok { + t.Fatalf("QueueSettings has no Retry document: %s", bson.Raw(raw)) + } + if got := rawStr(retry, "$Type"); got != c.typ { + t.Errorf("$Type = %q, want %q", got, c.typ) + } + elems, _ := retry.Elements() + if len(elems) != len(c.present)+2 { // + $ID, $Type + t.Errorf("Retry has %d properties, want exactly $ID, $Type and %v: %s", len(elems), c.present, retry) + } + for _, p := range c.present { + if _, err := retry.LookupErr(p); err != nil { + t.Errorf("Retry lacks %s: %s", p, retry) + } + } + + wrapped, _ := bson.Marshal(bson.D{{Key: "QueueSettings", Value: bson.Raw(raw)}}) + back := queueSettingsFromRaw(wrapped) + if back == nil || back.RetryPolicy == nil || *back.RetryPolicy != *c.in { + t.Errorf("read back %+v, want %+v", back.RetryPolicy, c.in) + } + }) + } + + // control: no retry writes Retry: null, as before + el := queueSettingsToGen(µflows.QueueSettings{Queue: "Ops.SendQueue"}) + raw, _ := (&codec.Encoder{}).Encode(el) + if v := bson.Raw(raw).Lookup("Retry"); v.Type != bson.TypeNull { + t.Errorf("an unretried queued call should store Retry: null, got %s", v.Type) + } +} diff --git a/mdl/backend/modelsdk/system_assignee_version_test.go b/mdl/backend/modelsdk/system_assignee_version_test.go new file mode 100644 index 0000000000..03c17607ab --- /dev/null +++ b/mdl/backend/modelsdk/system_assignee_version_test.go @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import "testing" + +// ako/mxcli#1810: the System module is version-dependent. On Mendix 9 the user +// task's assignee association is WorkflowUserTask_Assignee (a Reference); from +// 10.0 it is WorkflowUserTask_Assignees (a ReferenceSet) — one association, +// renamed with its GUID kept, read from the 9.24.30 / 10.0.0 / 11.12.6 modeler +// assemblies. The virtual System model check --references resolves against was +// the 11.x one for every project, so on 9.24 it accepted Assignees (CE1613 at +// build) and refused Assignee, the name 9.24 has. +func TestSystemModelAssigneeFollowsProjectVersion(t *testing.T) { + atLeast := func(maj, min int) func(int, int) bool { + return func(a, b int) bool { return maj > a || (maj == a && min >= b) } + } + cases := []struct { + name string + atLeast func(int, int) bool + want, not string + wantType string + }{ + {"9.24", atLeast(9, 24), "WorkflowUserTask_Assignee", "WorkflowUserTask_Assignees", "Reference"}, + {"10.0", atLeast(10, 0), "WorkflowUserTask_Assignees", "WorkflowUserTask_Assignee", "ReferenceSet"}, + {"11.12", atLeast(11, 12), "WorkflowUserTask_Assignees", "WorkflowUserTask_Assignee", "ReferenceSet"}, + {"unknown", nil, "WorkflowUserTask_Assignees", "WorkflowUserTask_Assignee", "ReferenceSet"}, + } + for _, c := range cases { + dm := buildSystemDomainModel(c.atLeast) + names := map[string]string{} + for _, a := range dm.Associations { + names[a.Name] = string(a.Type) + } + if typ, ok := names[c.want]; !ok || typ != c.wantType { + t.Errorf("%s: want %s (%s), got %q present=%v", c.name, c.want, c.wantType, typ, ok) + } + if _, ok := names[c.not]; ok { + t.Errorf("%s: %s must not exist in this version's System module", c.name, c.not) + } + } +} diff --git a/mdl/backend/modelsdk/system_domain_model_test.go b/mdl/backend/modelsdk/system_domain_model_test.go index a73b20c00e..b1dca8d628 100644 --- a/mdl/backend/modelsdk/system_domain_model_test.go +++ b/mdl/backend/modelsdk/system_domain_model_test.go @@ -20,7 +20,7 @@ func TestGetDomainModel_VirtualSystemModule(t *testing.T) { } t.Cleanup(func() { _ = b.Disconnect() }) - sys := buildSystemDomainModel() + sys := buildSystemDomainModel(nil) dm, err := b.GetDomainModel(sys.ContainerID) if err != nil { t.Fatalf("GetDomainModel(System) errored: %v", err) @@ -39,7 +39,7 @@ func TestGetDomainModel_VirtualSystemModule(t *testing.T) { // ASSOCIATIONS and DESCRIBE MODULE System silently omitted every System // association on the modelsdk engine. func TestSystemDomainModel_IncludesAssociations(t *testing.T) { - dm := buildSystemDomainModel() + dm := buildSystemDomainModel(nil) if len(dm.Associations) != len(meta.ModelerSystemAssociations()) { t.Fatalf("System associations: got %d, want %d", len(dm.Associations), len(meta.ModelerSystemAssociations())) } @@ -86,7 +86,7 @@ func TestSystemDomainModel_IncludesAssociations(t *testing.T) { // clean rule from the same script — the control that pins this to the list // rather than to the GRANT path reading it. func TestSystemDomainModel_OmitsRuntimeOnlyEntities(t *testing.T) { - dm := buildSystemDomainModel() + dm := buildSystemDomainModel(nil) entities := make(map[string]bool, len(dm.Entities)) for _, e := range dm.Entities { diff --git a/mdl/backend/modelsdk/system_module_read.go b/mdl/backend/modelsdk/system_module_read.go index 3e0973e447..40de4ba3a0 100644 --- a/mdl/backend/modelsdk/system_module_read.go +++ b/mdl/backend/modelsdk/system_module_read.go @@ -20,7 +20,11 @@ import ( // mxbuild with CE1613 "no longer exists" — upstream #972, where every // specialization of System.Image inherited an access-rule member for // System.Thumbnail_Image. See meta.SystemEntityDef.RuntimeOnly. -func buildSystemDomainModel() *domainmodel.DomainModel { +// +// atLeast is the project version's IsAtLeast, nil when it is unknown: the +// System module differs between versions (ako/mxcli#1810), and a check against +// the wrong one accepts a name the project's build then refuses. +func buildSystemDomainModel(atLeast func(major, minor int) bool) *domainmodel.DomainModel { dm := &domainmodel.DomainModel{ContainerID: model.ID(meta.SystemModuleID)} dm.ID = model.ID(meta.SystemDomainModelID) for _, e := range meta.ModelerSystemEntities() { @@ -56,7 +60,7 @@ func buildSystemDomainModel() *domainmodel.DomainModel { // association on the modelsdk engine (legacy already builds them via // BuildSystemDomainModel). IDs are synthetic and match the entity ID scheme // so the Parent/Child columns resolve to qualified names. - for _, a := range meta.ModelerSystemAssociations() { + for _, a := range meta.ModelerSystemAssociationsIn(atLeast) { assoc := &domainmodel.Association{ Name: a.Name, ParentID: model.ID("System." + a.Parent), @@ -95,3 +99,16 @@ func systemAttrType(a meta.SystemAttrDef) domainmodel.AttributeType { return &domainmodel.StringAttributeType{Length: a.Length} } } + +// systemVersionAtLeast is the project version's IsAtLeast, or nil when the +// project version is not known. +func (b *Backend) systemVersionAtLeast() func(major, minor int) bool { + if b == nil { + return nil + } + pv := b.ProjectVersion() + if pv == nil || pv.MajorVersion == 0 { + return nil + } + return pv.IsAtLeast +} diff --git a/mdl/backend/modelsdk/system_module_read_test.go b/mdl/backend/modelsdk/system_module_read_test.go index 8624f30912..a8a7ec2c01 100644 --- a/mdl/backend/modelsdk/system_module_read_test.go +++ b/mdl/backend/modelsdk/system_module_read_test.go @@ -12,7 +12,7 @@ import ( // virtual System module's attributes must each have a unique, non-empty ID. // Empty IDs collide on attributes_data.Id and break every catalog query. func TestSystemDomainModel_UniqueAttrIDs(t *testing.T) { - dm := buildSystemDomainModel() + dm := buildSystemDomainModel(nil) seen := map[string]string{} for _, e := range dm.Entities { for _, a := range e.Attributes { @@ -53,7 +53,7 @@ func TestSystemDomainModel_StringLengthsReachTheModel(t *testing.T) { "Error.Message": 0, } - dm := buildSystemDomainModel() + dm := buildSystemDomainModel(nil) seen := map[string]bool{} for _, e := range dm.Entities { for _, a := range e.Attributes { @@ -86,7 +86,7 @@ func TestSystemDomainModel_StringLengthsReachTheModel(t *testing.T) { // System.FileDocument read as storing no owner and `[System.owner = …]` on any // of its specializations was refused at check time while mxbuild built it. func TestSystemDomainModel_SystemMemberFlagsReachTheModel(t *testing.T) { - dm := buildSystemDomainModel() + dm := buildSystemDomainModel(nil) byName := map[string]*domainmodel.Entity{} for _, e := range dm.Entities { byName[e.Name] = e diff --git a/mdl/backend/modelsdk/widget_write.go b/mdl/backend/modelsdk/widget_write.go index 3444312501..c0351981d0 100644 --- a/mdl/backend/modelsdk/widget_write.go +++ b/mdl/backend/modelsdk/widget_write.go @@ -46,6 +46,13 @@ func init() { // value, leaving the rest at Go's zero value, never marked dirty, and so // omitted. Registering them here covers all three construction sites and // any future one, which setting every field at each site would not. + // A Delete action that deletes its context object stores SourceVariable + // null — 13 of 13 in ako/TestApp — and mxcli omitted the key; one that + // deletes a grid's selection stores a PageVariable there (ako/mxcli#1932). + // The property is 10.4+; below that the null is stripped. + codec.RegisterTypeDefaults("Forms$DeleteClientAction", codec.TypeDefaults{ + NullFields: []string{"SourceVariable"}, + }) codec.RegisterTypeDefaults("Forms$PageVariable", codec.TypeDefaults{ EmptyStringFields: []string{"LocalVariable", "PageParameter", "SnippetParameter", "SubKey", "Widget"}, FalseFields: []string{"UseAllPages"}, @@ -2324,6 +2331,13 @@ func clientActionToGen(a pages.ClientAction) (element.Element, error) { g := genPg.NewDeleteClientAction() assignID(g) g.SetClosePage(x.ClosePage) + // A grid's selection is a PageVariable naming only the widget, as + // Studio Pro stores it for a control-bar Delete button; without it + // mx check reports CE7005 (ako/mxcli#1932). The property is 10.4+: + // canon.StripUndeclaredProperties refuses it on an older project. + if sv := x.SourceVariable; sv != nil { + g.SetSourceVariable(pageVariableToGen(sv.Widget, sv.Variable, sv.Kind)) + } g.SetDisabledDuringExecution(actionDisabledDuringExecution(x.ActionExecution)) return g, nil case *pages.PageClientAction: diff --git a/mdl/backend/modelsdk/widget_write_delete_source_1932_test.go b/mdl/backend/modelsdk/widget_write_delete_source_1932_test.go new file mode 100644 index 0000000000..ed97d3596c --- /dev/null +++ b/mdl/backend/modelsdk/widget_write_delete_source_1932_test.go @@ -0,0 +1,84 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// ako/mxcli#1932 (mendixlabs/mxcli#1730): a Delete button in a Data grid 2 +// control bar deletes the grid's selection. Studio Pro stores it as +// Forms$DeleteClientAction.SourceVariable = Forms$PageVariable{Widget: "grid1"}, +// the other name slots empty — the same PageVariable shape as a flow argument +// passing a grid's selection (#1560). mxcli wrote SourceVariable null, which mx +// check reports as CE7005 "No value selection has been made". +func TestClientActionToGen_DeleteWritesSelectionSource(t *testing.T) { + el, err := clientActionToGen(&pages.DeleteClientAction{ + BaseElement: model.BaseElement{ID: "d"}, + SourceVariable: &pages.WidgetVariable{Widget: "grid1"}, + }) + if err != nil { + t.Fatalf("clientActionToGen: %v", err) + } + d := encodeToD(t, el) + sv := lookupD(d, "SourceVariable") + pv, ok := sv.(bson.D) + if !ok { + t.Fatalf("SourceVariable = %#v, want a Forms$PageVariable document", sv) + } + want := map[string]any{ + "$Type": "Forms$PageVariable", "Widget": "grid1", "PageParameter": "", + "SnippetParameter": "", "LocalVariable": "", "SubKey": "", "UseAllPages": false, + } + for k, v := range want { + if got := lookupD(pv, k); got != v { + t.Errorf("SourceVariable.%s = %#v, want %#v", k, got, v) + } + } +} + +// A page parameter fills the PageParameter slot alone. +func TestClientActionToGen_DeleteWritesParameterSource(t *testing.T) { + el, err := clientActionToGen(&pages.DeleteClientAction{ + BaseElement: model.BaseElement{ID: "d"}, + SourceVariable: &pages.WidgetVariable{Variable: "Order", Kind: "parameter"}, + }) + if err != nil { + t.Fatalf("clientActionToGen: %v", err) + } + pv, _ := lookupD(encodeToD(t, el), "SourceVariable").(bson.D) + if lookupD(pv, "PageParameter") != "Order" || lookupD(pv, "Widget") != "" { + t.Errorf("SourceVariable = %v, want PageParameter Order, Widget \"\"", pv) + } +} + +// Control: a delete in a data view deletes its context object and stores +// SourceVariable null — the key present, as on all 13 Studio Pro-authored Delete +// actions in ako/TestApp (mxcli omitted it before). +func TestClientActionToGen_DeleteWithoutSourceStaysNull(t *testing.T) { + el, err := clientActionToGen(&pages.DeleteClientAction{BaseElement: model.BaseElement{ID: "d"}}) + if err != nil { + t.Fatalf("clientActionToGen: %v", err) + } + d := encodeToD(t, el) + if !hasKey(d, "SourceVariable") { + t.Fatalf("SourceVariable key absent, want it present and null") + } + if v := lookupD(d, "SourceVariable"); v != nil { + t.Errorf("SourceVariable = %#v, want null", v) + } +} + +func lookupD(d bson.D, key string) any { + for _, e := range d { + if e.Key == key { + return e.Value + } + } + return nil +} diff --git a/mdl/executor/association_first_hop_834_test.go b/mdl/executor/association_first_hop_834_test.go new file mode 100644 index 0000000000..34b2c19ad9 --- /dev/null +++ b/mdl/executor/association_first_hop_834_test.go @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// ako/mxcli#834: a hop whose association does not touch the entity it starts +// from was written when the author named its destination. `$Order/Mod.Account_Flag/Mod.Flag` +// — Account_Flag joins Account and Flag, not Order — passed every check the +// builder made (the association exists; Flag is one of its ends) and stored an +// EntityRefStep Mendix cannot follow from an Order. +// +// The control is the same hop from the entity it does start at. +func TestAssociationHop_RefusesAssociationOffTheContextEntity(t *testing.T) { + for _, c := range []struct { + name string + ds *ast.DataSourceV3 + refused bool + }{ + {"association source, first hop off the context", &ast.DataSourceV3{Type: "association", ContextVariable: "Order", Reference: "Mod.Account_Flag/Mod.Flag"}, true}, + {"association source, control", &ast.DataSourceV3{Type: "association", ContextVariable: "Order", Reference: "Mod.Order_Account/Mod.Account"}, false}, + {"association source, second hop off its entity", &ast.DataSourceV3{Type: "association", ContextVariable: "Order", Reference: "Mod.Order_Account/Mod.Account/Mod.Order_Account/Mod.Order"}, false}, + } { + t.Run(c.name, func(t *testing.T) { + _, _, err := assocHopsPB().buildDataSourceV3(c.ds) + assertHopRefusal(t, err, c.refused, c.ds.Reference) + }) + } + for _, c := range []struct { + name string + path string + refused bool + }{ + {"database over association, first hop off the context", "Mod.Account_Flag/Mod.Flag", true}, + {"database over association, control", "Mod.Order_Account/Mod.Account", false}, + } { + t.Run(c.name, func(t *testing.T) { + _, _, err := assocHopsPB().buildDatabaseOverAssociationV3(&ast.DataSourceV3{ + Type: "database", ContextVariable: "Order", AssociationPath: c.path, + }) + assertHopRefusal(t, err, c.refused, c.path) + }) + } +} + +func assertHopRefusal(t *testing.T, err error, refused bool, path string) { + t.Helper() + switch { + case refused && err == nil: + t.Fatalf("%s from an Order succeeded; Mod.Account_Flag does not touch Mod.Order", path) + case refused && !strings.Contains(err.Error(), "Mod.Order"): + t.Errorf("error %q does not name the entity the hop starts from", err) + case !refused && err != nil: + t.Fatalf("%s: %v", path, err) + } +} + +// The same refusal at check time (MDL-ASSOCDS03), so `mxcli check -p` reports +// what exec would refuse. viewAssocCtx's project: Item_Customer joins Item and +// Customer, Stats_Customer joins Stats and Customer. +func TestCheckAssocHopStart(t *testing.T) { + ctx := viewAssocCtx(t) + for _, tc := range []struct { + name, script, param, ds string + reported bool + }{ + {"association source off the context", "", "Item", "$currentObject/C1611.Stats_Customer/C1611.Stats", true}, + {"database over association off the context", "", "Item", "DATABASE FROM $currentObject/C1611.Stats_Customer/C1611.Stats", true}, + {"unqualified, off the context", "", "Item", "$currentObject/Stats_Customer/C1611.Stats", true}, + // controls + {"from an end", "", "Item", "$currentObject/C1611.Item_Customer/C1611.Customer", false}, + {"from the other end", "", "Customer", "DATABASE FROM $currentObject/C1611.Item_Customer/C1611.Item", false}, + {"from a specialization the script declares", "create persistent entity C1611.Special extends C1611.Item (X: String(10));\n", + "Special", "$currentObject/C1611.Item_Customer/C1611.Customer", false}, + {"unknown association is not judged here", "", "Item", "$currentObject/C1611.Nope/C1611.Customer", false}, + } { + t.Run(tc.name, func(t *testing.T) { + var hits []string + for _, e := range assocListErrs(t, ctx, tc.script+viewAssocPage(tc.param, "dataview", tc.ds)) { + if strings.Contains(e, "MDL-ASSOCDS03") { + hits = append(hits, e) + } + } + if got := len(hits) > 0; got != tc.reported { + t.Errorf("MDL-ASSOCDS03 reported = %v, want %v: %q", got, tc.reported, hits) + } + }) + } +} diff --git a/mdl/executor/cmd_agenteditor_agents.go b/mdl/executor/cmd_agenteditor_agents.go index 4703e7942d..5c155f0d5f 100644 --- a/mdl/executor/cmd_agenteditor_agents.go +++ b/mdl/executor/cmd_agenteditor_agents.go @@ -126,10 +126,10 @@ func describeAgentEditorAgent(ctx *ExecContext, name ast.QualifiedName) error { lines = append(lines, fmt.Sprintf(" TopP: %g", *a.TopP)) } if a.SystemPrompt != "" { - lines = append(lines, fmt.Sprintf(" SystemPrompt: %s", formatAgentString(a.SystemPrompt))) + lines = append(lines, fmt.Sprintf(" SystemPrompt: %s", formatAgentString(ctx, a.SystemPrompt))) } if a.UserPrompt != "" { - lines = append(lines, fmt.Sprintf(" UserPrompt: %s", formatAgentString(a.UserPrompt))) + lines = append(lines, fmt.Sprintf(" UserPrompt: %s", formatAgentString(ctx, a.UserPrompt))) } for i, line := range lines { @@ -257,11 +257,10 @@ func findAgentEditorAgent(ctx *ExecContext, moduleName, agentName string) *agent // formatAgentString formats a string value for MDL output. Single-line // values use single quotes; multi-line values use $$...$$ dollar-quoting -// so they round-trip cleanly through the parser (which doesn't support -// newlines inside single-quoted strings). -func formatAgentString(s string) string { +// unless they hold `$$`, which would end the literal early (ako/mxcli#1003). +func formatAgentString(ctx *ExecContext, s string) string { if strings.ContainsAny(s, "\r\n") { - return "$$" + s + "$$" + return mdlDollarQuote(ctx, s) } - return "'" + escapeSQLString(s) + "'" + return mdlQuote(ctx, s) } diff --git a/mdl/executor/cmd_datatransformer.go b/mdl/executor/cmd_datatransformer.go index 44905ec503..6ab79bb2e3 100644 --- a/mdl/executor/cmd_datatransformer.go +++ b/mdl/executor/cmd_datatransformer.go @@ -86,13 +86,13 @@ func describeDataTransformer(ctx *ExecContext, name ast.QualifiedName) error { fmt.Fprintln(w, "{") for _, step := range dt.Steps { - if strings.Contains(step.Expression, "\n") { + if strings.Contains(step.Expression, "\n") && !strings.Contains(step.Expression, "$$") { // Multi-line: use $$ quoting fmt.Fprintf(w, " %s $$\n%s\n $$;\n", step.Technology, step.Expression) } else { - // Single-line: use regular string - expr := strings.ReplaceAll(step.Expression, "'", "''") - fmt.Fprintf(w, " %s '%s';\n", step.Technology, expr) + // Single-line, or holding `$$`, which ends a dollar-quoted + // literal early (ako/mxcli#1003): a regular string + fmt.Fprintf(w, " %s %s;\n", step.Technology, mdlQuote(ctx, step.Expression)) } } diff --git a/mdl/executor/cmd_fragments.go b/mdl/executor/cmd_fragments.go index e7b43dcc8d..d9882bf06e 100644 --- a/mdl/executor/cmd_fragments.go +++ b/mdl/executor/cmd_fragments.go @@ -321,10 +321,14 @@ func formatActionV3(a *ast.ActionV3) string { case "close": return "close page" case "delete": + out := "delete" + if a.SourceVariable != "" { + out += " $" + a.SourceVariable + } if a.ClosePage { - return "delete close page" + out += " close page" } - return "delete" + return out case "param": return "$" + a.Target case "create": diff --git a/mdl/executor/cmd_javaactions.go b/mdl/executor/cmd_javaactions.go index cc7b33fec7..e3d7d98b36 100644 --- a/mdl/executor/cmd_javaactions.go +++ b/mdl/executor/cmd_javaactions.go @@ -129,21 +129,21 @@ func describeJavaAction(ctx *ExecContext, name ast.QualifiedName) error { // longer existed. source := readJavaActionSource(ctx.MprPath, name.Module, name.Name) javaCode := javaActionUserCode(source) - sb.WriteString("\nas $$\n") + var body strings.Builder if javaCode != "" { if imports := javaactions.StatedImports(source, ja.Parameters, ja.ReturnType); len(imports) > 0 { - sb.WriteString(strings.Join(imports, "\n")) - sb.WriteString("\n\n") + body.WriteString(strings.Join(imports, "\n")) + body.WriteString("\n\n") } - sb.WriteString(javaCode) + body.WriteString(javaCode) } else { - sb.WriteString(javaSourceOmittedBody) + body.WriteString(javaSourceOmittedBody) } - sb.WriteString("\n$$") + sb.WriteString("\nas ") + sb.WriteString(mdlSourceBlock(ctx, body.String())) if _, extra := javaactions.RetainedSections(source); javaCode != "" && extra != "" { - sb.WriteString("\nextra code $$\n") - sb.WriteString(extra) - sb.WriteString("\n$$") + sb.WriteString("\nextra code ") + sb.WriteString(mdlSourceBlock(ctx, extra)) } sb.WriteString(";") diff --git a/mdl/executor/cmd_javascript_actions.go b/mdl/executor/cmd_javascript_actions.go index 2216f4c0c3..0519996325 100644 --- a/mdl/executor/cmd_javascript_actions.go +++ b/mdl/executor/cmd_javascript_actions.go @@ -137,13 +137,12 @@ func describeJavaScriptAction(ctx *ExecContext, name ast.QualifiedName) error { // emit one; when the .js source can't be read (add-on modules without source // on disk) emit a placeholder so the output still re-parses (cf. #637). userCode, extraCode := readJavaScriptActionSource(ctx.MprPath, name.Module, name.Name) - sb.WriteString("\nas $$\n") - if userCode != "" { - sb.WriteString(userCode) - } else { - sb.WriteString(jsSourceOmittedBody) + if userCode == "" { + userCode = jsSourceOmittedBody } - sb.WriteString("\n$$;") + sb.WriteString("\nas ") + sb.WriteString(mdlSourceBlock(ctx, userCode)) + sb.WriteString(";") stmt := sb.String() if gap := jsTypeParameterGap(jsa); gap != "" { diff --git a/mdl/executor/cmd_jsonstructures.go b/mdl/executor/cmd_jsonstructures.go index d351fe1c2b..a0b2a7c170 100644 --- a/mdl/executor/cmd_jsonstructures.go +++ b/mdl/executor/cmd_jsonstructures.go @@ -110,7 +110,7 @@ func describeJsonStructure(ctx *ExecContext, name ast.QualifiedName) error { if js.JsonSnippet != "" { snippet := types.PrettyPrintJSON(js.JsonSnippet) if strings.Contains(snippet, "'") || strings.Contains(snippet, "\n") { - fmt.Fprintf(ctx.Output, "\n sample $$%s$$", snippet) + fmt.Fprintf(ctx.Output, "\n sample %s", mdlDollarQuote(ctx, snippet)) } else { fmt.Fprintf(ctx.Output, "\n sample '%s'", snippet) } diff --git a/mdl/executor/cmd_jsonstructures_mock_test.go b/mdl/executor/cmd_jsonstructures_mock_test.go index 3374cf7fbf..02a3860098 100644 --- a/mdl/executor/cmd_jsonstructures_mock_test.go +++ b/mdl/executor/cmd_jsonstructures_mock_test.go @@ -139,3 +139,47 @@ func TestDescribeJsonStructure_NotFound(t *testing.T) { ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) assertError(t, describeJsonStructure(ctx, ast.QualifiedName{Module: "OrderMgmt", Name: "NoSuch"})) } + +// TestDescribeJsonStructure_SnippetContainingDollarDollar (ako/mxcli#1003): a +// snippet holding `$$` — a Teamcenter UID such as +// "SR::N::BOMLine..10.7$$Otklm9GclKA" — ended describe's `$$ … $$` literal +// early, and `mxcli check` on the output reported `token recognition error at: +// '$$`. The controls are the same snippet without the `$$`, which keeps the +// dollar-quoted form, and one ending in a single `$`. +func TestDescribeJsonStructure_SnippetContainingDollarDollar(t *testing.T) { + for _, snippet := range []string{ + `{"uid": "SR::N::BOMLine..10.7$$Otklm9GclKA", "name": "it's"}`, + `{"uid": "SR::N::BOMLine..10.7Otklm9GclKA", "name": "it's"}`, + `{"path": "C:\\temp$$x", "price": "5$"}`, + } { + mod := mkModule("Tc") + js := &types.JsonStructure{ + BaseElement: model.BaseElement{ID: nextID("js")}, + ContainerID: mod.ID, + Name: "BomLine", + JsonSnippet: snippet, + } + h := mkHierarchy(mod) + withContainer(h, js.ContainerID, mod.ID) + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListJsonStructuresFunc: func() ([]*types.JsonStructure, error) { return []*types.JsonStructure{js}, nil }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, describeJsonStructure(ctx, ast.QualifiedName{Module: "Tc", Name: "BomLine"})) + prog, errs := visitor.Build(describedSource(ctx, buf.String())) + if len(errs) > 0 { + t.Fatalf("describe of snippet %q does not parse: %v\n%s", snippet, errs, buf.String()) + } + var got *ast.CreateJsonStructureStmt + for _, s := range prog.Statements { + if cs, ok := s.(*ast.CreateJsonStructureStmt); ok { + got = cs + } + } + want := types.PrettyPrintJSON(snippet) + if got == nil || got.JsonSnippet != want { + t.Errorf("snippet %q did not read back; got %+v from:\n%s", snippet, got, buf.String()) + } + } +} diff --git a/mdl/executor/cmd_mermaid.go b/mdl/executor/cmd_mermaid.go index 4250a501ac..4b0ca72b21 100644 --- a/mdl/executor/cmd_mermaid.go +++ b/mdl/executor/cmd_mermaid.go @@ -539,6 +539,10 @@ func mermaidActionLabel(a *microflows.ActionActivity, entityNames map[model.ID]s return "rollback $" + sanitizeMermaidLabel(act.RollbackVariable) } return "Rollback" + case *microflows.SendExternalObjectAction: + return "send external object $" + sanitizeMermaidLabel(act.Variable) + case *microflows.DeleteExternalObjectAction: + return "delete external object $" + sanitizeMermaidLabel(act.Variable) case *microflows.CreateVariableAction: if act.VariableName != "" { return "declare $" + sanitizeMermaidLabel(act.VariableName) @@ -710,6 +714,12 @@ func mermaidActionDetails(a *microflows.ActionActivity, entityNames map[model.ID lines = append(lines, "Variable: $"+act.RollbackVariable) } + case *microflows.SendExternalObjectAction: + lines = append(lines, "Variable: $"+act.Variable) + + case *microflows.DeleteExternalObjectAction: + lines = append(lines, "Variable: $"+act.Variable) + case *microflows.RetrieveAction: if act.OutputVariable != "" { lines = append(lines, "Output: $"+act.OutputVariable) diff --git a/mdl/executor/cmd_messagedefinitions.go b/mdl/executor/cmd_messagedefinitions.go index 1b2bf79444..c925caf305 100644 --- a/mdl/executor/cmd_messagedefinitions.go +++ b/mdl/executor/cmd_messagedefinitions.go @@ -633,8 +633,9 @@ func describeMessageMembers(ctx *ExecContext, members []*model.MessageDefinition } switch { case m.Kind == "Attribute": - fmt.Fprintf(ctx.Output, "%s%s%s%s%s\n", indent, m.OriginalName, - exposedClause(m, m.OriginalName), exampleClause(m), sep) + name := memberAttributeName(m) + fmt.Fprintf(ctx.Output, "%s%s%s%s%s\n", indent, name, + exposedClause(m, name), exampleClause(m), sep) case len(m.Children) == 0: fmt.Fprintf(ctx.Output, "%s%s/%s%s { }%s\n", indent, m.Association, m.Entity, exposedClause(m, shortEntityName(m.Entity)), sep) @@ -647,6 +648,24 @@ func describeMessageMembers(ctx *ExecContext, members []*model.MessageDefinition } } +// memberAttributeName is the attribute an Attribute member is printed by: the +// last segment of the stored attribute reference, which is what create resolves +// the member against. OriginalName is the element's own stored name and lags a +// rename — `rename attribute` updates the reference and leaves the element name +// for Studio Pro's refresh, which is CE6524 until then (mendixlabs/mxcli#1733). +// Printing OriginalName made describe emit an attribute that no longer exists; +// printing the reference with the stored ExposedName as `as '…'` round-trips +// and keeps the exposed name, i.e. the message contract, unchanged. +func memberAttributeName(m *model.MessageDefinitionElement) string { + if m.Attribute == "" { + return m.OriginalName + } + if i := strings.LastIndex(m.Attribute, "."); i >= 0 { + return m.Attribute[i+1:] + } + return m.Attribute +} + // exampleClause emits `example '...'`, the one authored field besides the name. // Rare — 1 of 4,707 elements — but describe dropping it silently is what makes // describe -> exec lossy, so it is emitted whenever set. @@ -910,10 +929,13 @@ func availableChildren(n *model.MessageDefinitionElement) string { } // findChildByName matches on the member's OWN name — the attribute's or the -// target entity's — which is what `add`/`drop`/`set member X` names. +// target entity's — which is what `add`/`drop`/`set member X` names. For an +// attribute that is the attribute it references, which is how describe prints +// it; the stored element name is accepted too, since it lags a rename until +// Studio Pro refreshes the definition (mendixlabs/mxcli#1733). func findChildByName(n *model.MessageDefinitionElement, name string) *model.MessageDefinitionElement { for _, c := range n.Children { - if c.OriginalName == name { + if c.OriginalName == name || (c.Kind == "Attribute" && memberAttributeName(c) == name) { return c } } diff --git a/mdl/executor/cmd_microflow_elk.go b/mdl/executor/cmd_microflow_elk.go index 3021179f1e..5e7f348f73 100644 --- a/mdl/executor/cmd_microflow_elk.go +++ b/mdl/executor/cmd_microflow_elk.go @@ -308,7 +308,8 @@ func classifyAction(a *microflows.ActionActivity) string { switch a.Action.(type) { case *microflows.CreateObjectAction, *microflows.ChangeObjectAction, *microflows.CommitObjectsAction, *microflows.DeleteObjectAction, - *microflows.RollbackObjectAction: + *microflows.RollbackObjectAction, *microflows.SendExternalObjectAction, + *microflows.DeleteExternalObjectAction: return "object" case *microflows.RetrieveAction: return "retrieve" diff --git a/mdl/executor/cmd_microflows_build.go b/mdl/executor/cmd_microflows_build.go index e79a7989eb..f39b4de7ac 100644 --- a/mdl/executor/cmd_microflows_build.go +++ b/mdl/executor/cmd_microflows_build.go @@ -225,6 +225,11 @@ func buildMicroflowFromStmt(ctx *ExecContext, s *ast.CreateMicroflowStmt, opts b if err := checkNoQueuedCalls(ctx, existingID, qualifiedName, s); err != nil { return nil, err } + // A whole-flow rebuild writes only what the statement says, so an + // activity MDL cannot express is lost (mendixlabs/mxcli#1735). + if types := unsupportedActionTypes(existingObjects); len(types) > 0 { + return nil, unsupportedActionLossError("microflow", qualifiedName, "rebuilding it from this statement", types) + } // Same reasoning for a REST body the writer cannot express: the rebuild // would drop it, DESCRIBE would not show it missing, and the app would // still build. @@ -570,6 +575,14 @@ func buildNanoflowFromStmt(ctx *ExecContext, s *ast.CreateNanoflowStmt, opts bui // A rewrite that carried no doc comment keeps the stored one (#1018). existingDocumentation = existing.Documentation preserveDocumentation = true + // As for a microflow: a whole-flow rebuild loses an activity MDL cannot + // express (mendixlabs/mxcli#1735). + if opts.AllowCreate { + if types := unsupportedActionTypes(existing.ObjectCollection); len(types) > 0 { + return nil, unsupportedActionLossError("nanoflow", s.Name.Module+"."+s.Name.Name, + "rebuilding it from this statement", types) + } + } } // For CREATE OR REPLACE/MODIFY, reuse the existing ID to preserve references diff --git a/mdl/executor/cmd_microflows_builder_actions.go b/mdl/executor/cmd_microflows_builder_actions.go index 6b80f9874d..0c49789ca2 100644 --- a/mdl/executor/cmd_microflows_builder_actions.go +++ b/mdl/executor/cmd_microflows_builder_actions.go @@ -296,6 +296,39 @@ func (fb *flowBuilder) addDeleteAction(s *ast.DeleteObjectStmt) model.ID { return activity.ID } +// addExternalObjectAction creates a SEND or DELETE EXTERNAL OBJECT activity +// (mendixlabs/mxcli#1735). +func (fb *flowBuilder) addExternalObjectAction(s *ast.ExternalObjectStmt) model.ID { + activityX := fb.posX + var action microflows.MicroflowAction + id := model.BaseElement{ID: model.ID(types.GenerateID())} + eh := explicitErrorHandling(fb, s.ErrorHandling) + if s.Delete { + action = µflows.DeleteExternalObjectAction{BaseElement: id, Variable: s.Variable, RefreshInClient: s.RefreshInClient, ErrorHandlingType: eh} + } else { + action = µflows.SendExternalObjectAction{BaseElement: id, Variable: s.Variable, RefreshInClient: s.RefreshInClient, ErrorHandlingType: eh} + } + + activity := µflows.ActionActivity{ + BaseActivity: microflows.BaseActivity{ + BaseMicroflowObject: microflows.BaseMicroflowObject{ + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + Position: model.Point{X: fb.posX, Y: fb.posY}, + Size: model.Size{Width: ActivityWidth, Height: ActivityHeight}, + }, + AutoGenerateCaption: true, + }, + Action: action, + } + + fb.objects = append(fb.objects, activity) + fb.posX += fb.spacing + + fb.finishCustomErrorHandler(activity.ID, activityX, s.ErrorHandling, "") + + return activity.ID +} + // addRollbackAction creates a ROLLBACK statement. func (fb *flowBuilder) addRollbackAction(s *ast.RollbackStmt) model.ID { activityX := fb.posX @@ -1476,7 +1509,10 @@ func (fb *flowBuilder) addRetrieveAction(s *ast.RetrieveStmt) model.ID { BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, RangeType: microflows.RangeTypeFirst, } - } else if s.Limit != "" { + } else if s.Limit != "" || s.Offset != "" { + // An offset alone is a Custom range too; testing the limit only + // wrote `offset 5000` as the "all" range, and the retrieve + // returned the first objects instead (mendixlabs/mxcli#1737). dbSource.Range = µflows.Range{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, RangeType: microflows.RangeTypeCustom, diff --git a/mdl/executor/cmd_microflows_builder_calls.go b/mdl/executor/cmd_microflows_builder_calls.go index 378bbba198..a9651a003f 100644 --- a/mdl/executor/cmd_microflows_builder_calls.go +++ b/mdl/executor/cmd_microflows_builder_calls.go @@ -133,7 +133,7 @@ func (fb *flowBuilder) addLogMessageAction(s *ast.LogStmt) model.ID { // activity rather than the script, which makes a typo here expensive to trace. // // `what` names the statement in the error (e.g. "CALL MICROFLOW"). -func (fb *flowBuilder) buildQueueSettings(q *ast.QualifiedName, what string) *microflows.QueueSettings { +func (fb *flowBuilder) buildQueueSettings(q *ast.QualifiedName, retry *ast.QueueRetry, what string) *microflows.QueueSettings { if q == nil { return nil } @@ -141,10 +141,22 @@ func (fb *flowBuilder) buildQueueSettings(q *ast.QualifiedName, what string) *mi if !fb.queueExists(q.Module, q.Name) { fb.addError("%s ... IN QUEUE '%s': task queue not found in the project (create it with `create task queue %s (Parallelism: 1)`)", what, qn, qn) } - return µflows.QueueSettings{ + qs := µflows.QueueSettings{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, Queue: qn, } + if retry != nil && retry.Kind != "" { + qs.RetryPolicy = µflows.QueueRetry{ + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + Kind: retry.Kind, + Retries: retry.Retries, + Interval: retry.Interval, + InitialInterval: retry.InitialInterval, + MaximumInterval: retry.MaximumInterval, + IntervalType: retry.IntervalType, + } + } + return qs } // queueExists reports whether a Queues$Queue with the given module-qualified @@ -198,7 +210,7 @@ func (fb *flowBuilder) addCallMicroflowAction(s *ast.CallMicroflowStmt) model.ID BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, Microflow: mfQN, ParameterMappings: mappings, - QueueSettings: fb.buildQueueSettings(s.Queue, "CALL MICROFLOW"), + QueueSettings: fb.buildQueueSettings(s.Queue, s.QueueRetry, "CALL MICROFLOW"), } action := µflows.MicroflowCallAction{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, @@ -441,7 +453,7 @@ func (fb *flowBuilder) addCallJavaActionAction(s *ast.CallJavaActionStmt) model. ErrorHandlingType: fb.ehType(s.ErrorHandling), JavaAction: actionQN, ParameterMappings: mappings, - QueueSettings: fb.buildQueueSettings(s.Queue, "CALL JAVA ACTION"), + QueueSettings: fb.buildQueueSettings(s.Queue, s.QueueRetry, "CALL JAVA ACTION"), ResultVariableName: s.OutputVariable, UseReturnVariable: s.OutputVariable != "", } diff --git a/mdl/executor/cmd_microflows_builder_control.go b/mdl/executor/cmd_microflows_builder_control.go index 7d2cfc5f07..c67e7dafde 100644 --- a/mdl/executor/cmd_microflows_builder_control.go +++ b/mdl/executor/cmd_microflows_builder_control.go @@ -918,6 +918,8 @@ func statementErrorHandling(stmt ast.MicroflowStatement) *ast.ErrorHandlingClaus return s.ErrorHandling case *ast.DeleteObjectStmt: return s.ErrorHandling + case *ast.ExternalObjectStmt: + return s.ErrorHandling case *ast.CallMicroflowStmt: return s.ErrorHandling case *ast.CallNanoflowStmt: diff --git a/mdl/executor/cmd_microflows_builder_flows.go b/mdl/executor/cmd_microflows_builder_flows.go index 2b8c23a15b..e91074d2f4 100644 --- a/mdl/executor/cmd_microflows_builder_flows.go +++ b/mdl/executor/cmd_microflows_builder_flows.go @@ -788,6 +788,8 @@ func errorHandlerStatementVarRefs(stmt ast.MicroflowStatement) []string { refs = append(refs, s.Variable) case *ast.DeleteObjectStmt: refs = append(refs, s.Variable) + case *ast.ExternalObjectStmt: + refs = append(refs, s.Variable) case *ast.DownloadFileStmt: refs = append(refs, s.FileDocument) case *ast.SynchronizeStmt: diff --git a/mdl/executor/cmd_microflows_builder_graph.go b/mdl/executor/cmd_microflows_builder_graph.go index 76b6eb8fbb..bd31e816cf 100644 --- a/mdl/executor/cmd_microflows_builder_graph.go +++ b/mdl/executor/cmd_microflows_builder_graph.go @@ -705,6 +705,8 @@ func (fb *flowBuilder) addStatement(stmt ast.MicroflowStatement) model.ID { return fb.addDeleteAction(s) case *ast.RollbackStmt: return fb.addRollbackAction(s) + case *ast.ExternalObjectStmt: + return fb.addExternalObjectAction(s) case *ast.IfStmt: return fb.addIfStatement(s) case *ast.LoopStmt: diff --git a/mdl/executor/cmd_microflows_builder_merge.go b/mdl/executor/cmd_microflows_builder_merge.go index 7dc6194c1b..d48e4ab896 100644 --- a/mdl/executor/cmd_microflows_builder_merge.go +++ b/mdl/executor/cmd_microflows_builder_merge.go @@ -189,6 +189,9 @@ func (fb *flowBuilder) resolveJoins() { if !ok { continue // already reported as undeclared } + if fb.adoptDecisionMerge(reg, e, m) { + continue + } var flow *microflows.SequenceFlow if e.caseValue != "" { flow = newHorizontalFlowWithCase(e.origin, m.ID, e.caseValue) @@ -214,6 +217,67 @@ func (fb *flowBuilder) resolveJoins() { fb.dropUnreachedMerges(reg) } +// adoptDecisionMerge makes a label's merge BE the merge a decision closed on, +// when the script spells it `end case; join L; … merge L;` and nothing else +// reaches L. It reports whether it did; no edge is then drawn for the join. +// +// DESCRIBE writes exactly that when the decision's own merge is the one it has +// to name — typically a decision one of whose branches ends the flow, so the +// branches post-dominate at the exit rather than at the merge (ako/mxcli#1929, +// mendixlabs/mxcli#1727). `end case` and `merge L` then denote one stored node. +// Built as two, wired merge -> merge, every describe -> drop + create added a +// merge, which the next describe named `join junctionN; merge junctionN;`. +// +// Narrow on purpose: +// - the origin is an unlabelled merge (one the builder drew to close a +// decision), not a label's own merge; +// - it has no way out but this join; +// - the join is the label's only way in, and no path falls into `merge L`. +// A label other paths also reach is a real crossing, and the decision's +// merge in front of it is a node of its own; +// - the join leaves a merge, not a split's branch (no case value). +func (fb *flowBuilder) adoptDecisionMerge(reg *labelRegistry, e joinEdge, m *microflows.ExclusiveMerge) bool { + if e.caseValue != "" || e.origin == m.ID { + return false + } + var origin *microflows.ExclusiveMerge + for _, o := range fb.objects { + if o != nil && o.GetID() == e.origin { + origin, _ = o.(*microflows.ExclusiveMerge) + break + } + } + if origin == nil { + return false + } + for _, lm := range reg.merges { + if lm.ID == origin.ID { + return false + } + } + joins := 0 + for _, j := range reg.joins { + if j.label == e.label { + joins++ + } + } + if joins != 1 { + return false + } + for _, f := range fb.flows { + if f != nil && (f.OriginID == origin.ID || f.DestinationID == m.ID) { + return false + } + } + for _, f := range fb.flows { + if f != nil && f.DestinationID == origin.ID { + f.DestinationID = m.ID + } + } + fb.dropObject(origin.ID) + return true +} + // dropUnreachedMerges removes declared merges that no flow reaches. Declaring a // merge and never joining it is a no-op in the author's head and a build error // in Mendix, so it is reported rather than shipped. diff --git a/mdl/executor/cmd_microflows_builder_validate.go b/mdl/executor/cmd_microflows_builder_validate.go index 3da408d806..dfc241401e 100644 --- a/mdl/executor/cmd_microflows_builder_validate.go +++ b/mdl/executor/cmd_microflows_builder_validate.go @@ -310,6 +310,11 @@ func (fb *flowBuilder) validateStatement(stmt ast.MicroflowStatement) { case *ast.RollbackStmt: // No error handling to validate + case *ast.ExternalObjectStmt: + if s.ErrorHandling != nil && len(s.ErrorHandling.Body) > 0 { + fb.validateStatements(s.ErrorHandling.Body) + } + case *ast.RetrieveStmt: fb.validateOutputVariable(s.Variable, "retrieve") // Register retrieved variable diff --git a/mdl/executor/cmd_microflows_create.go b/mdl/executor/cmd_microflows_create.go index 313e6056ec..dd40bd0069 100644 --- a/mdl/executor/cmd_microflows_create.go +++ b/mdl/executor/cmd_microflows_create.go @@ -74,6 +74,13 @@ func execCreateMicroflow(ctx *ExecContext, s *ast.CreateMicroflowStmt) error { if err := validateMicroflowRules(s); err != nil { return err } + // The splice path does not reach the rebuild's queue guard, and a + // call restated without its stored retry is an edit it would make. + if stored := findMicroflowByQualifiedName(ctx, s.Name.String()); stored != nil { + if err := checkNoDroppedQueueRetries(ctx, stored.ID, s.Name.String(), s, nil); err != nil { + return err + } + } if handled, err := modifyFlowInPlace(ctx, microflowDecl(s)); handled || err != nil { return err } diff --git a/mdl/executor/cmd_microflows_describe_ended_branch_merge_test.go b/mdl/executor/cmd_microflows_describe_ended_branch_merge_test.go new file mode 100644 index 0000000000..af09c8805c --- /dev/null +++ b/mdl/executor/cmd_microflows_describe_ended_branch_merge_test.go @@ -0,0 +1,125 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" +) + +// ako/mxcli#1929 (mendixlabs/mxcli#1727): a decision one of whose branches +// ends the flow, while the others merge, is described as +// +// end case; +// join shared1; +// merge shared1; +// +// — the label naming the decision's OWN merge. Building that description made +// the decision's merge and a second one for the label, wired merge -> merge at +// the same point, and every further describe -> drop + create added another +// (`join junctionN; merge junctionN;`). The runtime behaviour was the same; the +// model, its diagram and every diff were not. +// +// assertDescriptionRebuildsGraph cannot see this: it contracts merges away by +// design. So the merge count is compared directly, and the description must be +// a fixed point. + +const endedBranchCaseMDL = `create microflow Probe.RT_Repro ($S: Probe.Shape) returns Integer as $N +begin + declare $N Integer = 0; + case $S/C + when R then return 1; + when G then set $N = 2; + when (empty) then set $N = 3; + end case; + return $N; +end;` + +const endedBranchSplitTypeMDL = `create microflow M.ST ($A: M.Animal) returns Integer as $N +begin + declare $N Integer = 0; + split type $A + when M.Dog then + return 1; + when M.Cat then + set $N = 2; + when M.Animal then + set $N = 3; + end split; + return $N; +end;` + +const endedBranchElsifMDL = `create microflow M.EI ($X: Integer) returns Integer as $N +begin + declare $N Integer = 0; + if $X = 1 then + set $N = 1; + elsif $X = 2 then + return 2; + else + set $N = 3; + end if; + return $N; +end;` + +// roundTripMergeCounts builds src, then rebuilds its description `rounds` +// times, returning the merge count of each generation and the descriptions. +func roundTripMergeCounts(t *testing.T, src string, rounds int) ([]int, []string) { + t.Helper() + header := src[:strings.Index(src, "\nbegin")] + var counts []int + var outs []string + cur := src + for i := 0; i <= rounds; i++ { + out, oc := describeBuilt(t, cur) + counts = append(counts, countMerges(oc)) + outs = append(outs, out) + if i == 0 { + assertDescriptionRebuildsGraph(t, out, oc) + } + cur = header + "\nbegin\n" + out + "\nend;" + } + return counts, outs +} + +func TestDescribeEndedBranch_RoundTripKeepsOneMerge(t *testing.T) { + for name, src := range map[string]string{ + "case": endedBranchCaseMDL, + "split type": endedBranchSplitTypeMDL, + "elsif": endedBranchElsifMDL, + } { + t.Run(name, func(t *testing.T) { + counts, outs := roundTripMergeCounts(t, src, 2) + for i, n := range counts { + if n != counts[0] { + t.Errorf("generation %d has %d merges, the original %d — describe -> create added a merge:\n%s", + i, n, counts[0], outs[i]) + break + } + } + if outs[1] != outs[0] { + t.Errorf("the description is not a fixed point:\nfirst:\n%s\nsecond:\n%s", outs[0], outs[1]) + } + if strings.Contains(outs[len(outs)-1], "junction") { + t.Errorf("a later round trip named an extra merge:\n%s", outs[len(outs)-1]) + } + }) + } +} + +// CONTROL: a real crossing, where a label is reached from more than one place +// and the decision's merge is a separate node, keeps every merge it had. +func TestDescribeEndedBranch_RealCrossingsKeepTheirMerges(t *testing.T) { + for name, src := range map[string]string{ + "crossed inside branch": crossedInsideBranchMDL, + "interleaved if": interleavedIfMDL, + "fully interleaved": fullyInterleavedIfMDL, + } { + t.Run(name, func(t *testing.T) { + counts, outs := roundTripMergeCounts(t, src, 1) + if counts[1] != counts[0] { + t.Errorf("merges %d -> %d across a round trip:\n%s\n---\n%s", counts[0], counts[1], outs[0], outs[1]) + } + }) + } +} diff --git a/mdl/executor/cmd_microflows_external_object_test.go b/mdl/executor/cmd_microflows_external_object_test.go new file mode 100644 index 0000000000..ff92b7b8ab --- /dev/null +++ b/mdl/executor/cmd_microflows_external_object_test.go @@ -0,0 +1,127 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// mendixlabs/mxcli#1735 (ako/mxcli#1939): Studio Pro's "Send external object" +// and "Delete external object" activities had no MDL. A flow that saves an +// external (OData) object could not be written — a plain commit on one is a +// Studio Pro error — and describe printed `-- Unsupported action`, so a drop + +// create of such a flow from its description silently lost the activity. + +// externalObjectActionOf builds the one statement of a microflow body and +// returns the action the builder makes of it. +func externalObjectActionOf(t *testing.T, stmt string) microflows.MicroflowAction { + t.Helper() + src := "create microflow M.F ($District: M.District) begin\n " + stmt + "\nend;" + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("%q does not parse: %v", stmt, errs) + } + mf := prog.Statements[0].(*ast.CreateMicroflowStmt) + fb := &flowBuilder{varTypes: map[string]string{"District": "M.District"}} + fb.addExternalObjectAction(mf.Body[0].(*ast.ExternalObjectStmt)) + if len(fb.errors) > 0 { + t.Fatalf("%q: builder errors: %v", stmt, fb.errors) + } + return fb.objects[0].(*microflows.ActionActivity).Action +} + +func TestExternalObject_SendAndDeleteBuildTheirActions(t *testing.T) { + cases := []struct { + stmt string + describe string + check func(t *testing.T, a microflows.MicroflowAction) + }{ + { + "send external object $District;", "send external object $District;", + func(t *testing.T, a microflows.MicroflowAction) { + s, ok := a.(*microflows.SendExternalObjectAction) + if !ok || s.Variable != "District" || s.RefreshInClient { + t.Fatalf("action = %#v, want a send of $District without refresh", a) + } + }, + }, + { + "send external object $District refresh;", "send external object $District refresh;", + func(t *testing.T, a microflows.MicroflowAction) { + s, ok := a.(*microflows.SendExternalObjectAction) + if !ok || !s.RefreshInClient { + t.Fatalf("action = %#v, want a send with refresh", a) + } + }, + }, + { + "delete external object $District;", "delete external object $District;", + func(t *testing.T, a microflows.MicroflowAction) { + d, ok := a.(*microflows.DeleteExternalObjectAction) + if !ok || d.Variable != "District" || d.RefreshInClient { + t.Fatalf("action = %#v, want a delete of $District without refresh", a) + } + }, + }, + { + // The `on error` clause is printed by the flow walk, not by the + // action's own line (as for every action). + "delete external object $District refresh on error continue;", "delete external object $District refresh;", + func(t *testing.T, a microflows.MicroflowAction) { + d, ok := a.(*microflows.DeleteExternalObjectAction) + if !ok || !d.RefreshInClient || d.ErrorHandlingType != microflows.ErrorHandlingTypeContinue { + t.Fatalf("action = %#v, want a delete with refresh and on error continue", a) + } + }, + }, + } + e := newTestExecutor() + for _, tc := range cases { + t.Run(tc.stmt, func(t *testing.T) { + a := externalObjectActionOf(t, tc.stmt) + tc.check(t, a) + // describe prints what parses back to the same action. + got := e.formatAction(a, nil, nil) + if got != tc.describe { + t.Fatalf("describe = %q, want %q", got, tc.describe) + } + if strings.HasPrefix(got, "--") { + t.Fatalf("describe still prints the action as unsupported: %q", got) + } + }) + } +} + +// A plain `delete $X` is still the Delete object activity: `external object` +// is what selects the external one. +func TestExternalObject_PlainDeleteIsUnchanged(t *testing.T) { + src := "create microflow M.F ($Order: M.Order) begin\n delete $Order;\nend;" + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatal(errs) + } + if _, ok := prog.Statements[0].(*ast.CreateMicroflowStmt).Body[0].(*ast.DeleteObjectStmt); !ok { + t.Fatalf("`delete $Order` parsed as %T", prog.Statements[0].(*ast.CreateMicroflowStmt).Body[0]) + } +} + +// `on error continue` on either activity is CE6035 (measured on 11.12.6, one +// microflow per project; the default and a custom handler, with or without +// rollback, are accepted). check refuses it before mxbuild does. +func TestExternalObject_ContinueIsRefused(t *testing.T) { + for _, verb := range []string{"send", "delete"} { + t.Run(verb, func(t *testing.T) { + stmt := &ast.ExternalObjectStmt{Delete: verb == "delete", Variable: "D", + ErrorHandling: &ast.ErrorHandlingClause{Type: ast.ErrorHandlingContinue}} + keyword, activity := continueUnsupportedStatement(stmt) + if keyword != verb+" external object" || activity == "" { + t.Fatalf("continueUnsupportedStatement = %q, %q; want %q refused", keyword, activity, verb+" external object") + } + }) + } +} diff --git a/mdl/executor/cmd_microflows_format_action.go b/mdl/executor/cmd_microflows_format_action.go index 812497a892..09023f9236 100644 --- a/mdl/executor/cmd_microflows_format_action.go +++ b/mdl/executor/cmd_microflows_format_action.go @@ -133,7 +133,8 @@ func refreshModifier(refresh bool) string { // // Widget and workflow expressions are not microflow expressions: they are // stored as written in both languages, so their describers only trim. -// describeCallArgument renders a call microflow / call nanoflow argument. A +// describeCallArgument renders a call microflow / call nanoflow / call external +// action argument. A // blank stored argument — the mapping is present, its expression empty, which // is what Studio Pro writes for an argument field left empty — is `nothing`. // Printing it through describeExpr gave `Param = ,` / `Param = )`, which does @@ -542,6 +543,12 @@ func formatAction( case *microflows.RollbackObjectAction: return fmt.Sprintf("rollback $%s%s;", a.RollbackVariable, refreshModifier(a.RefreshInClient)) + case *microflows.SendExternalObjectAction: + return fmt.Sprintf("send external object $%s%s;", a.Variable, refreshModifier(a.RefreshInClient)) + + case *microflows.DeleteExternalObjectAction: + return fmt.Sprintf("delete external object $%s%s;", a.Variable, refreshModifier(a.RefreshInClient)) + case *microflows.CreateListAction: // Use EntityQualifiedName (BY_NAME_REFERENCE) or fall back to EntityID lookup entityName := a.EntityQualifiedName @@ -907,7 +914,11 @@ func formatAction( var params []string for _, pm := range a.ParameterMappings { - params = append(params, fmt.Sprintf("%s = %s", pm.ParameterName, describeExpr(ctx, pm.Argument))) + // describeCallArgument, not describeExpr: Studio Pro 11.16 stores an + // optional parameter left blank as `Argument: ""`, which printed as + // `name = )` and did not parse (ako/mxcli#1915). `nothing` is the + // blank; `empty` (11.14's spelling of the same field) stays a value. + params = append(params, fmt.Sprintf("%s = %s", pm.ParameterName, describeCallArgument(ctx, pm.Argument))) } paramStr := "" @@ -2383,7 +2394,25 @@ func queueClauseMDL(qs *microflows.QueueSettings) string { if qs == nil || qs.Queue == "" { return "" } - return " in queue " + qs.Queue + return " in queue " + qs.Queue + queueRetryClauseMDL(qs.RetryPolicy) +} + +// queueRetryClauseMDL renders a queued call's retry policy, so a stored retry +// round-trips (mendixlabs/mxcli#1729). Every property is printed: the clause +// has no defaults. +func queueRetryClauseMDL(r *microflows.QueueRetry) string { + if r == nil { + return "" + } + switch r.Kind { + case microflows.QueueRetryFixed: + return fmt.Sprintf(" retry fixed (Retries: %d, Interval: %d, IntervalType: %s)", + r.Retries, r.Interval, r.IntervalType) + case microflows.QueueRetryExponential: + return fmt.Sprintf(" retry exponential (Retries: %d, InitialInterval: %d, MaximumInterval: %d, IntervalType: %s)", + r.Retries, r.InitialInterval, r.MaximumInterval, r.IntervalType) + } + return "" } // mdlAggregateKeyword maps a Mendix aggregate function to the MDL keyword that diff --git a/mdl/executor/cmd_microflows_issue_citation_test.go b/mdl/executor/cmd_microflows_issue_citation_test.go new file mode 100644 index 0000000000..44f0abddf3 --- /dev/null +++ b/mdl/executor/cmd_microflows_issue_citation_test.go @@ -0,0 +1,85 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +// ako/mxcli#1005: the describe warnings cited "mxcli #923", which in the fork +// these builds ship from is an unrelated MCP-backend issue. The issue they mean +// is mendixlabs/mxcli#923 ("roundtrip for @merge does not work"), so every +// user-facing citation has to carry the owner. + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +func assertCitesUpstream923(t *testing.T, what, text string) { + t.Helper() + if !strings.Contains(text, "mendixlabs/mxcli#923") { + t.Errorf("%s does not cite mendixlabs/mxcli#923: %q", what, text) + } + if strings.Contains(text, "mxcli #923") { + t.Errorf("%s still cites the ambiguous \"mxcli #923\": %q", what, text) + } +} + +func TestIssueCitation_IrreducibleWarningNamesUpstream(t *testing.T) { + got := irreducibleGraphWarnings(reporterCollection(), nil) + if len(got) != 1 { + t.Fatalf("want 1 warning, got %d: %v", len(got), got) + } + assertCitesUpstream923(t, "the non-nesting decision warning", got[0]) +} + +func TestIssueCitation_DroppedMergeWarningNamesUpstream(t *testing.T) { + f := newRejoinFixture() + f.add("start", µflows.StartEvent{BaseMicroflowObject: f.base(0)}) + f.add("act", µflows.ActionActivity{BaseActivity: microflows.BaseActivity{BaseMicroflowObject: f.base(100)}}) + f.add("merge", µflows.ExclusiveMerge{BaseMicroflowObject: f.base(200)}) + f.add("end", µflows.EndEvent{BaseMicroflowObject: f.base(300)}) + f.edge("start", "act", false) + f.edge("act", "merge", false) + f.edge("merge", "end", false) + + got := droppedMergeWarnings(nil, f.col, labelRejoinMerges(f.col)) + if len(got) != 1 { + t.Fatalf("want 1 warning, got %d: %v", len(got), got) + } + assertCitesUpstream923(t, "the unrepresented-merge warning", got[0]) +} + +func TestIssueCitation_NormalizedNoteNamesUpstream(t *testing.T) { + _, notes := normalizeCollection(reporterGraph("$B").col) + if len(notes) != 1 { + t.Fatalf("want 1 note, got %v", notes) + } + assertCitesUpstream923(t, "the NORMALIZED note", notes[0]) +} + +// The bug-test script's flow (mdl-examples/bug-tests/1005-…) is the reporter's +// graph written with join points; NORMALIZED folds it and the note carries +// the citation, as the script's header says. +func TestIssueCitation_BugTestScriptNormalizes(t *testing.T) { + oc := buildMicroflowFromMDL(t, `create microflow BugTest1005.Crossing ($A: Boolean, $B: Boolean) +begin + if $A then + if $B then + join shared_log; + else + join after_log; + end if; + else + join shared_log; + end if; + merge shared_log; + log info node 'BugTest1005' 'shared'; + join after_log; + merge after_log; +end;`) + _, notes := normalizeCollection(oc) + if len(notes) != 1 { + t.Fatalf("want 1 NORMALIZED note, got %v", notes) + } + assertCitesUpstream923(t, "the NORMALIZED note on the bug-test flow", notes[0]) +} diff --git a/mdl/executor/cmd_microflows_normalize.go b/mdl/executor/cmd_microflows_normalize.go index 669ec47328..8c288c90ed 100644 --- a/mdl/executor/cmd_microflows_normalize.go +++ b/mdl/executor/cmd_microflows_normalize.go @@ -172,7 +172,7 @@ func foldSplit(work *microflows.MicroflowObjectCollection, f microflowgraph.Find return fmt.Sprintf( "-- NOTE: the decision at %s was NORMALIZED - its branches were folded into one condition. "+ "This is equivalent to the microflow but is NOT its shape: re-executing this MDL builds "+ - "a graph with fewer nodes and a different layout (mxcli #923, Mode 3)", + "a graph with fewer nodes and a different layout (mendixlabs/mxcli#923, Mode 3)", positionOf(f.Split)), nil } diff --git a/mdl/executor/cmd_microflows_retrieve_first_test.go b/mdl/executor/cmd_microflows_retrieve_first_test.go index d987248345..f5d56eb7c5 100644 --- a/mdl/executor/cmd_microflows_retrieve_first_test.go +++ b/mdl/executor/cmd_microflows_retrieve_first_test.go @@ -146,3 +146,22 @@ end;` }) } } + +// mendixlabs/mxcli#1737 (ako/mxcli#1941): `retrieve … offset 5000` without a +// limit was written as the "all" range: the offset was dropped, so the +// retrieve returned the first objects instead of skipping 5000, with check and +// exec both silent. An offset alone is a Custom range too. +func TestRetrieveRange_OffsetWithoutLimit(t *testing.T) { + const body = "create microflow M.F () begin\n retrieve $O from M.Order offset 5000;\nend;" + for _, header := range []string{"", "mdl 1;\n"} { + t.Run(strings.TrimSpace(header), func(t *testing.T) { + rng, varType := retrieveRangeOf(t, header+body) + if rng == nil || rng.RangeType != microflows.RangeTypeCustom || rng.Offset != "5000" || rng.Limit != "" { + t.Fatalf("range = %+v, want a Custom range with offset 5000 and no limit", rng) + } + if !strings.HasPrefix(varType, "List of ") { + t.Errorf("output typed %q, want a list", varType) + } + }) + } +} diff --git a/mdl/executor/cmd_microflows_show.go b/mdl/executor/cmd_microflows_show.go index 799534a147..2216991e38 100644 --- a/mdl/executor/cmd_microflows_show.go +++ b/mdl/executor/cmd_microflows_show.go @@ -1398,7 +1398,7 @@ func irreducibleGraphWarnings(oc *microflows.MicroflowObjectCollection, declared out = append(out, fmt.Sprintf( "-- WARNING: the decision%s has %d branches that do not nest - %s. "+ "This description is NOT equivalent to the microflow and must not be re-executed over it; "+ - "edit it in Studio Pro instead (mxcli #923, %s)", + "edit it in Studio Pro instead (mendixlabs/mxcli#923, %s)", pos, f.BranchCount, detail, f.Class)) } return out diff --git a/mdl/executor/cmd_microflows_show_helpers.go b/mdl/executor/cmd_microflows_show_helpers.go index c584f9b8a4..27863323db 100644 --- a/mdl/executor/cmd_microflows_show_helpers.go +++ b/mdl/executor/cmd_microflows_show_helpers.go @@ -29,6 +29,10 @@ type describedAnnotation struct { // Only a shared note needs a label, so the everyday `@annotation 'text'` // form is left exactly as it was. Shared bool + + // Unstated names, for a note described as free, each link of it that MDL + // has no form for (to the start event, to another note). + Unstated []string } // annotationEmitter renders the @annotation lines for a describe traversal. @@ -213,25 +217,66 @@ func (e *annotationEmitter) lines(target model.ID, activityPos model.Point, targ return out } -// collectFreeAnnotations returns the annotations no AnnotationFlow points at — -// notes that sit on the canvas documenting the flow as a whole. +// collectFreeAnnotations returns the annotations the describer has no activity +// to print under — notes that sit on the canvas documenting the flow as a +// whole, and notes whose only links end where no `@annotation` is ever printed. +// +// The second kind used to vanish (ako/mxcli#1933, mendixlabs/mxcli#1731): a +// note linked only to the start event, or to another note, counted as linked, +// so it was not free, and its target prints nothing, so it was not attached +// either. Each is now described as a free note, with Unstated naming the link +// MDL cannot say. func collectFreeAnnotations(oc *microflows.MicroflowObjectCollection) []describedAnnotation { if oc == nil { return nil } - referencedAnnotations := make(map[model.ID]bool) + objects := map[model.ID]microflows.MicroflowObject{} + var collect func(c *microflows.MicroflowObjectCollection) + collect = func(c *microflows.MicroflowObjectCollection) { + if c == nil { + return + } + for _, o := range c.Objects { + if o == nil { + continue + } + objects[o.GetID()] = o + if loop, ok := o.(*microflows.LoopedActivity); ok { + collect(loop.ObjectCollection) + } + } + } + collect(oc) + + attached := make(map[model.ID]bool) + unstated := make(map[model.ID][]string) for _, af := range oc.AnnotationFlows { - referencedAnnotations[af.OriginID] = true + if af == nil { + continue + } + if kind := unstatableNoteEnd(objects[af.DestinationID]); kind == "" { + attached[af.OriginID] = true + } else { + unstated[af.OriginID] = append(unstated[af.OriginID], kind) + } + // A link drawn the other way round (from an element to the note) is + // not read as attaching the note by buildAnnotationsByTarget either. + if _, isNote := objects[af.DestinationID].(*microflows.Annotation); isNote { + if _, fromNote := objects[af.OriginID].(*microflows.Annotation); !fromNote { + unstated[af.DestinationID] = append(unstated[af.DestinationID], "an element by a line drawn from that element") + } + } } var result []describedAnnotation for _, obj := range oc.Objects { if annot, ok := obj.(*microflows.Annotation); ok { - if !referencedAnnotations[annot.ID] && annot.Caption != "" { + if !attached[annot.ID] && annot.Caption != "" { result = append(result, describedAnnotation{ ID: annot.ID, Caption: annot.Caption, Position: annot.Position, Size: annot.Size, + Unstated: unstated[annot.ID], }) } } @@ -239,6 +284,20 @@ func collectFreeAnnotations(oc *microflows.MicroflowObjectCollection) []describe return result } +// unstatableNoteEnd names the far end of a note's link when DESCRIBE cannot +// print the note there: the start event (no statement of its own) or another +// note. "" for an element that carries its notes as `@annotation` — and for +// one the collection does not hold, which is left to the traversal as before. +func unstatableNoteEnd(o microflows.MicroflowObject) string { + switch o.(type) { + case *microflows.StartEvent: + return "the start event" + case *microflows.Annotation: + return "another note" + } + return "" +} + func prependFreeAnnotationLines(ctx *ExecContext, oc *microflows.MicroflowObjectCollection, activityLines []string) []string { freeAnnots := collectFreeAnnotations(oc) if len(freeAnnots) == 0 || len(activityLines) == 0 { @@ -247,6 +306,14 @@ func prependFreeAnnotationLines(ctx *ExecContext, oc *microflows.MicroflowObject prefix := make([]string, 0, len(freeAnnots)) for _, note := range freeAnnots { + for _, end := range note.Unstated { + // A link MDL has no form for is dropped by any rewrite; say so + // rather than let the round trip lose it quietly. + prefix = append(prefix, fmt.Sprintf( + "-- WARNING: the note %s is linked to %s, which MDL cannot state. It is described "+ + "unattached, and re-executing this description drops the link", + mdlQuote(ctx, note.Caption), end)) + } // A free note is attached to nothing, so there is no activity position // to derive its default from — the writer places it from wherever the // cursor happens to be. Its geometry is therefore always spelled out. diff --git a/mdl/executor/cmd_microflows_show_merge.go b/mdl/executor/cmd_microflows_show_merge.go index 9a2e19b0f2..d59bc43ead 100644 --- a/mdl/executor/cmd_microflows_show_merge.go +++ b/mdl/executor/cmd_microflows_show_merge.go @@ -458,7 +458,7 @@ func droppedMergeWarningsIn(ctx *ExecContext, oc *microflows.MicroflowObjectColl "it joins no decision and no error handler. `create or modify` from this MDL leaves it in place, "+ "but anything that rebuilds the flow from it (`create or replace`, an mdl 0 fallback) removes it. "+ "The microflow behaves the same either way (a merge with one incoming path is a no-op), "+ - "but the node disappears from the diagram (mxcli #923)", + "but the node disappears from the diagram (mendixlabs/mxcli#923)", p.X, p.Y)) } diff --git a/mdl/executor/cmd_pages_action_delete_source_1932_test.go b/mdl/executor/cmd_pages_action_delete_source_1932_test.go new file mode 100644 index 0000000000..2126f43622 --- /dev/null +++ b/mdl/executor/cmd_pages_action_delete_source_1932_test.go @@ -0,0 +1,169 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "context" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// ako/mxcli#1932 (mendixlabs/mxcli#1730): a Delete button in a Data grid 2 +// control bar. Studio Pro stores Forms$DeleteClientAction with SourceVariable = +// Forms$PageVariable{Widget: "grid1"} — the grid's selection. describe printed +// it as plain `Action: delete` and exec wrote SourceVariable null, which mx check +// reports as CE7005 "No value selection has been made"; check --references +// passed. The spelling is `delete $grid1`, as `$grid1` already names the grid's +// selection in a flow argument (#1560). + +func TestDeleteAction_DescribeReadsSelectionSource(t *testing.T) { + ctx := (&Executor{}).newExecContext(context.Background()) + for _, tc := range []struct { + name string + stored map[string]any + want string + }{ + {"grid selection", map[string]any{ + "$Type": "Forms$DeleteClientAction", "ClosePage": false, "SourceVariable": gridSelection("grid1"), + }, "delete $grid1"}, + {"grid selection, close page", map[string]any{ + "$Type": "Forms$DeleteClientAction", "ClosePage": true, "SourceVariable": gridSelection("grid1"), + }, "delete $grid1 close page"}, + // Control: the context object (a data view's) stores no source. + {"no source (control)", map[string]any{ + "$Type": "Forms$DeleteClientAction", "ClosePage": false, "SourceVariable": nil, + }, "delete"}, + } { + t.Run(tc.name, func(t *testing.T) { + if got := renderClientActionMDL(ctx, tc.stored); got != tc.want { + t.Errorf("described %q, want %q", got, tc.want) + } + }) + } +} + +// buildDeleteFrom parses `Action: ` on a button and builds it with a page +// builder that knows grid1 as a widget of the document and Order as a page +// parameter. +func buildDeleteFrom(t *testing.T, mdl string, argCtx pageArgContext) (pages.ClientAction, error) { + t.Helper() + prog, errs := visitor.Build("create page M.P (Title: 'T', Layout: A.L) {\n" + + " actionbutton b1 (Caption: 'Delete', Action: " + mdl + ")\n}") + if len(errs) > 0 { + t.Fatalf("%q does not parse: %v", mdl, errs) + } + a := prog.Statements[0].(*ast.CreatePageStmtV3).Widgets[0].Properties["Action"].(*ast.ActionV3) + pb := &pageBuilder{ + argCtx: argCtx, + paramScope: map[string]model.ID{"Order": ""}, + isDocumentWidget: func(name string) bool { return name == "grid1" }, + } + return pb.buildClientActionV3(a) +} + +func TestDeleteAction_SelectionSourceRoundTrips(t *testing.T) { + inControlBar := enteringDataWidget(&ast.DataSourceV3{Type: "database", Reference: "M.Order"}, "M.Order").inDataWidget("grid1") + inControlBar.selectionOf = "grid1" + for _, tc := range []struct { + mdl string + closePage bool + want *pages.WidgetVariable + }{ + {"delete $grid1", false, &pages.WidgetVariable{Widget: "grid1"}}, + {"delete $grid1 close page", true, &pages.WidgetVariable{Widget: "grid1"}}, + {"delete $Order", false, &pages.WidgetVariable{Variable: "Order", Kind: "parameter"}}, + } { + t.Run(tc.mdl, func(t *testing.T) { + built, err := buildDeleteFrom(t, tc.mdl, inControlBar) + if err != nil { + t.Fatalf("build: %v", err) + } + d, ok := built.(*pages.DeleteClientAction) + if !ok { + t.Fatalf("built %T, want *pages.DeleteClientAction", built) + } + if d.SourceVariable == nil || *d.SourceVariable != *tc.want { + t.Errorf("SourceVariable = %+v, want %+v", d.SourceVariable, tc.want) + } + if d.ClosePage != tc.closePage { + t.Errorf("ClosePage = %v, want %v", d.ClosePage, tc.closePage) + } + }) + } +} + +// Control: inside a data view the context object is the source, stored as no +// SourceVariable — exactly what was written before. +func TestDeleteAction_DataViewDeleteHasNoSource(t *testing.T) { + built, err := buildDeleteFrom(t, "delete", enteringDataWidget(&ast.DataSourceV3{Type: "parameter", Reference: "$Order"}, "M.Order")) + if err != nil { + t.Fatalf("build: %v", err) + } + if d := built.(*pages.DeleteClientAction); d.SourceVariable != nil { + t.Errorf("SourceVariable = %+v, want nil (the data view's object)", d.SourceVariable) + } +} + +func TestDeleteAction_UnknownSourceRefused(t *testing.T) { + _, err := buildDeleteFrom(t, "delete $nope", atDocumentRoot()) + if err == nil || !strings.Contains(err.Error(), "$nope") { + t.Fatalf("err = %v, want a refusal naming $nope", err) + } +} + +// A bare `delete` with no object to delete is the CE7005 button the issue +// reports. Refused at exec in a control bar and at a document's root; ALTER +// PAGE, which cannot see what encloses the button, stands down. +func TestDeleteAction_BareDeleteWithoutObjectRefused(t *testing.T) { + inControlBar := enteringDataWidget(&ast.DataSourceV3{Type: "database", Reference: "M.Order"}, "M.Order").inDataWidget("grid1") + inControlBar.selectionOf = "grid1" + if _, err := buildDeleteFrom(t, "delete", inControlBar); err == nil || !strings.Contains(err.Error(), "delete $grid1") { + t.Errorf("control bar: err = %v, want a refusal suggesting `delete $grid1`", err) + } + if _, err := buildDeleteFrom(t, "delete", atDocumentRoot()); err == nil || !strings.Contains(err.Error(), "CE7005") { + t.Errorf("document root: err = %v, want a refusal naming CE7005", err) + } + if _, err := buildDeleteFrom(t, "delete", pageArgContext{}); err != nil { + t.Errorf("unknown context (ALTER PAGE): err = %v, want none", err) + } +} + +// The check-time mirror: `mxcli check` reports the CE7005 button without a +// project. +func TestDeleteAction_CheckReportsBareDeleteInControlBar(t *testing.T) { + violations := func(button string) []string { + prog, errs := visitor.Build(`create page M.P (Title: 'T', Layout: A.L) { + datagrid grid1 (DataSource: database from M.Order) { + controlbar { + actionbutton btnDelete (Caption: 'Delete', Action: ` + button + `) + } + column (attribute: Name, caption: 'Name') + } + dataview dv (DataSource: database from M.Order) { + actionbutton btnDvDelete (Caption: 'Delete', Action: delete) + } +}`) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + var out []string + for _, v := range validateWidgetPropertiesOf(prog.Statements[0], LoadWidgetRegistry("")) { + if v.RuleID == "MDL-PAGEDEL01" { + out = append(out, v.Message) + } + } + return out + } + if got := violations("delete"); len(got) != 1 || !strings.Contains(got[0], "btnDelete") { + t.Errorf("bare delete in the control bar: got %v, want one MDL-PAGEDEL01 on btnDelete", got) + } + // Control: the selection spelled out, and the data view's button, are clean. + if got := violations("delete $grid1"); len(got) != 0 { + t.Errorf("delete $grid1: got %v, want none", got) + } +} diff --git a/mdl/executor/cmd_pages_builder_v3.go b/mdl/executor/cmd_pages_builder_v3.go index cde793157b..ebfc599e70 100644 --- a/mdl/executor/cmd_pages_builder_v3.go +++ b/mdl/executor/cmd_pages_builder_v3.go @@ -1197,9 +1197,24 @@ func (pb *pageBuilder) applyDatabaseClausesV3(dbSource *pages.DatabaseSource, ds // exists"), and `A1/E1/A2/E2` as one step whose DestinationEntity was the // literal "E1/A2/E2" — the project would not load. func (pb *pageBuilder) resolveAssociationSourceHops(reference, fromEntity string) (entityPath, destEntity string, err error) { + hops, dest, err := pb.resolveAssociationHops(reference, fromEntity) + if err != nil { + return "", "", err + } + pairs := make([]string, 0, 2*len(hops)) + for _, h := range hops { + pairs = append(pairs, h.Association, h.DestinationEntity) + } + return strings.Join(pairs, "/"), dest, nil +} + +// resolveAssociationHops is resolveAssociationSourceHops' traversal: one step +// per hop, and the entity the last one arrives at. The association source and +// `database from $ctx/…` (buildDatabaseOverAssociationV3) both read a path +// through it, so every spelling one accepts the other does (ako/mxcli#834). +func (pb *pageBuilder) resolveAssociationHops(reference, fromEntity string) (hops []pages.AttributeRefStep, destEntity string, err error) { segs := strings.Split(reference, "/") at := fromEntity - var pairs []string for i := 0; i < len(segs); { // Both halves of an EntityRefStep are BY_NAME references, and Mendix // resolves either one it cannot find to null — so the association half @@ -1224,19 +1239,30 @@ func (pb *pageBuilder) resolveAssociationSourceHops(reference, fromEntity string // association would sail through and be written // qualified-but-nonexistent, which Mendix resolves to null just // the same. Verify it exists. - return "", "", mdlerrors.NewValidationf( + return nil, "", mdlerrors.NewValidationf( "association %q for datasource %q does not exist — "+ "writing it would produce a project Mendix cannot open; "+ "a bare name is qualified with the module of the context entity (%s), "+ "so an association declared elsewhere must be named in full", assoc, reference, at) } + // The hop must start where the path is: an association joining two + // other entities has both ends, so the destination check below + // passes it, and the step it writes cannot be followed from `at` + // (ako/mxcli#834). A derived destination needs no such check — + // resolveAssociationDestination finds none off the association. + if at != "" && !pb.isEndOrSpecialization(at, from, to) { + return nil, "", mdlerrors.NewValidationf( + "association %q in datasource %q does not start at %s — its ends are %s and %s; "+ + "each hop must follow an association of the entity the previous one arrives at", + assoc, reference, at, from, to) + } // Neither the next association nor an end of this one: a misspelled // association read as an entity, or an entity off the path. Either // way the step would name an entity the hop does not arrive at. // Studio Pro may store a specialization of an end, so accept that. - if dest != from && dest != to && !pb.isSpecializationOf(dest, from) && !pb.isSpecializationOf(dest, to) { - return "", "", mdlerrors.NewValidationf( + if !pb.isEndOrSpecialization(dest, from, to) { + return nil, "", mdlerrors.NewValidationf( "%q in datasource %q is neither an association nor an end of association %q (%s, %s) — "+ "each hop is `Module.Assoc` or `Module.Assoc/Module.DestinationEntity`", dest, reference, assoc, from, to) @@ -1252,16 +1278,22 @@ func (pb *pageBuilder) resolveAssociationSourceHops(reference, fromEntity string // instead of writing a structurally invalid unit; the author can name the // destination explicitly as `Assoc/Module.Entity`. (issuetracker #14) if dest == "" { - return "", "", mdlerrors.NewValidationf( + return nil, "", mdlerrors.NewValidationf( "cannot resolve the destination entity of association %q for datasource %q — "+ "writing it unresolved would produce a project Mendix cannot open; "+ "name the destination explicitly, e.g. `%s/Module.Entity`", assoc, reference, assoc) } - pairs = append(pairs, assoc, dest) + hops = append(hops, pages.AttributeRefStep{Association: assoc, DestinationEntity: dest}) at = dest } - return strings.Join(pairs, "/"), at, nil + return hops, at, nil +} + +// isEndOrSpecialization reports whether entity is one of an association's ends +// (from, to) or a specialization of one. +func (pb *pageBuilder) isEndOrSpecialization(entity, from, to string) bool { + return entity == from || entity == to || pb.isSpecializationOf(entity, from) || pb.isSpecializationOf(entity, to) } // isAssociationSegment reports whether a path segment following an association @@ -1293,8 +1325,9 @@ func (pb *pageBuilder) buildListViewDataSourceV3(ds *ast.DataSourceV3) (pages.Da // IndirectEntityRef (ako/mxcli#721 L5). It keeps the XPath, sort and search an // association source (`$ctx/Assoc`) does not have. // -// The path's segments pair up as association/destination. Describe prints every -// destination, because Studio Pro may store a specialization of the +// The path is read hop by hop as an association source's is +// (resolveAssociationHops): a destination after an association is optional. +// Describe prints every destination, because Studio Pro may store a specialization of the // association's own end (Administration.Account over System.User's // WorkflowUserTask_Assignees) and re-deriving it would write a different entity. // A trailing association with no destination is resolved as an association @@ -1314,36 +1347,12 @@ func (pb *pageBuilder) buildDatabaseOverAssociationV3(ds *ast.DataSourceV3) (pag } } - segs := strings.Split(ds.AssociationPath, "/") - var steps []pages.AttributeRefStep - at := fromEntity - for i := 0; i < len(segs); i += 2 { - assoc := pb.resolveAssociationPathIn(segs[i], at) - if _, _, ok := pb.associationEndpoints(assoc); !ok { - return nil, "", mdlerrors.NewValidationf( - "association %q in datasource `database from $%s/%s` does not exist — "+ - "writing it would produce a project Mendix cannot open", - assoc, ds.ContextVariable, ds.AssociationPath) - } - dest := "" - if i+1 < len(segs) { - dest = segs[i+1] - if _, err := pb.resolveEntity(ast.QualifiedName{Module: pb.extractModule(dest), Name: pb.extractName(dest)}); err != nil { - return nil, "", mdlerrors.NewValidationf( - "entity %q in datasource `database from $%s/%s` does not exist", - dest, ds.ContextVariable, ds.AssociationPath) - } - } else { - dest = pb.resolveAssociationDestination(assoc, at) - } - if dest == "" { - return nil, "", mdlerrors.NewValidationf( - "cannot resolve the entity association %q arrives at in datasource `database from $%s/%s` — "+ - "name it after the association, e.g. `%s/Module.Entity`", - assoc, ds.ContextVariable, ds.AssociationPath, assoc) - } - steps = append(steps, pages.AttributeRefStep{Association: assoc, DestinationEntity: dest}) - at = dest + // One traversal for both builders: the association source's, which takes + // the short form `A1/A2`, the full `A1/E1/A2/E2` and mixtures, and refuses a + // destination that is not an end of its association (ako/mxcli#834). + steps, at, err := pb.resolveAssociationHops(ds.AssociationPath, fromEntity) + if err != nil { + return nil, "", err } entityID, err := pb.resolveEntity(ast.QualifiedName{Module: pb.extractModule(at), Name: pb.extractName(at)}) @@ -1816,6 +1825,10 @@ func (pb *pageBuilder) buildClientActionV3Base(action *ast.ActionV3) (pages.Clie }, nil case "delete": + source, err := pb.deleteSource(action.SourceVariable) + if err != nil { + return nil, err + } return &pages.DeleteClientAction{ BaseElement: model.BaseElement{ ID: model.ID(types.GenerateID()), @@ -1824,7 +1837,8 @@ func (pb *pageBuilder) buildClientActionV3Base(action *ast.ActionV3) (pages.Clie // The visitor sets it for `delete close page`; dropping it here // turned a describe → exec round trip into a button that no longer // closes its page (ako/mxcli#950). - ClosePage: action.ClosePage, + ClosePage: action.ClosePage, + SourceVariable: source, }, nil case "create": diff --git a/mdl/executor/cmd_pages_delete_source.go b/mdl/executor/cmd_pages_delete_source.go new file mode 100644 index 0000000000..938380f20c --- /dev/null +++ b/mdl/executor/cmd_pages_delete_source.go @@ -0,0 +1,98 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// A page Delete action deletes the context object, or — Studio Pro's +// Forms$DeleteClientAction.SourceVariable — a widget's selection or a page or +// snippet parameter. A Delete button in a Data grid 2 control bar stores +// SourceVariable = Forms$PageVariable{Widget: "grid1"}; without it mx check +// reports CE7005 "No value selection has been made" (ako/mxcli#1932, +// mendixlabs/mxcli#1730). MDL spells it `delete $grid1`, as a flow argument +// names a grid's selection (#1560). + +// deleteSource resolves the `$source` of `delete $source` to the +// SourceVariable it stores, or refuses a bare `delete` that has no object to +// delete. name is written without the "$"; "" is the context object. +func (pb *pageBuilder) deleteSource(name string) (*pages.WidgetVariable, error) { + if name == "" || strings.EqualFold(name, "currentObject") { + if msg := deleteWithoutObjectProblem(pb.currentWidget, pb.argCtx); msg != "" { + return nil, mdlerrors.NewValidation(msg + " [MDL-PAGEDEL01]") + } + return nil, nil + } + variable, kind := pb.classifyFlowArgValue("$" + name) + if kind == "" && pb.argCtx.namesSelection("$"+name) { + variable, kind = "$"+name, "widget" + } + variable = strings.TrimPrefix(variable, "$") + switch kind { + case "widget": + return &pages.WidgetVariable{Widget: variable}, nil + case "parameter", "snippet": + return &pages.WidgetVariable{Variable: variable, Kind: kind}, nil + } + return nil, mdlerrors.NewValidation(fmt.Sprintf( + "delete $%s: $%s is neither a widget of this document (whose selection is deleted) nor an "+ + "object parameter of this page or snippet — write `delete` to delete the context object [MDL-PAGEDEL01]", + name, name)) +} + +// deleteWithoutObjectProblem is the single wording `mxcli check` and `exec` +// share for a bare `delete` with nothing to delete: in a grid's control bar, +// or outside every data widget. "" when there is an object, or when the pass +// cannot see what encloses the widget (ALTER PAGE), where it stands down. +func deleteWithoutObjectProblem(widgetName string, c pageArgContext) string { + where := "widget `" + widgetName + "`" + if widgetName == "" { + where = "this widget" + } + if c.selectionOf != "" { + return fmt.Sprintf( + "%s: `delete` in the control bar of `%s` has no object to delete — a control bar has no context "+ + "object, and the button would be stored with no source, which mx check reports as CE7005 "+ + "\"No value selection has been made\". Write `delete $%s` to delete the grid's selection", + where, c.selectionOf, c.selectionOf) + } + if c.known && !c.present { + return fmt.Sprintf( + "%s: `delete` is not inside a data view, list view or grid row, so there is no context object to "+ + "delete, which mx check reports as CE7005 \"No value selection has been made\". Write "+ + "`delete $` to delete a widget's selection or `delete $` for an object parameter "+ + "of this page or snippet, or put the button inside a data view", + where) + } + return "" +} + +// validateDeleteSource is the check-time mirror of the executor guard. +func validateDeleteSource(w *ast.WidgetV3, c pageArgContext, locationPrefix string) []linter.Violation { + if w == nil { + return nil + } + action := w.GetAction() + if action == nil || action.Type != "delete" { + return nil + } + if action.SourceVariable != "" && !strings.EqualFold(action.SourceVariable, "currentObject") { + return nil + } + msg := deleteWithoutObjectProblem(w.Name, c) + if msg == "" { + return nil + } + return []linter.Violation{{ + RuleID: "MDL-PAGEDEL01", + Severity: linter.SeverityError, + Message: locationPrefix + ": " + msg, + }} +} diff --git a/mdl/executor/cmd_pages_describe_output.go b/mdl/executor/cmd_pages_describe_output.go index e823664e12..982644d75e 100644 --- a/mdl/executor/cmd_pages_describe_output.go +++ b/mdl/executor/cmd_pages_describe_output.go @@ -1658,6 +1658,11 @@ func renderClientActionBaseMDL(ctx *ExecContext, action map[string]any) string { return "close page" case "Forms$DeleteClientAction", "Pages$DeleteClientAction": result := "delete" + // A grid's selection (a control-bar button) or a page/snippet + // parameter; null is the context object (ako/mxcli#1932). + if src := flowArgVariableValue(action["SourceVariable"]); src != "" { + result += " " + src + } if closePage, ok := action["ClosePage"].(bool); ok && closePage { result += " close page" } diff --git a/mdl/executor/cmd_pages_pluggable_passthrough.go b/mdl/executor/cmd_pages_pluggable_passthrough.go index b7ec9ff6d2..a9854c234e 100644 --- a/mdl/executor/cmd_pages_pluggable_passthrough.go +++ b/mdl/executor/cmd_pages_pluggable_passthrough.go @@ -440,3 +440,39 @@ func bsonIDKey(v any) string { } return "" } + +// storedUsesDatabaseSource reports whether the stored pluggable widget named +// name holds a 9.x database source (CustomWidgets$CustomWidgetDatabaseSource) +// anywhere in its Object (ako/mxcli#1208). +func (pt *pluggablePassthrough) storedUsesDatabaseSource(name string) bool { + if pt == nil || name == "" { + return false + } + sp, ok := pt.byName[name] + if !ok || sp == nil { + return false + } + return bsonHasType(sp.obj, "CustomWidgets$CustomWidgetDatabaseSource") +} + +// bsonHasType reports whether v, or anything below it, is a document of $Type t. +func bsonHasType(v any, t string) bool { + switch x := v.(type) { + case bson.D: + for _, e := range x { + if e.Key == "$Type" && e.Value == t { + return true + } + if bsonHasType(e.Value, t) { + return true + } + } + case bson.A: + for _, el := range x { + if bsonHasType(el, t) { + return true + } + } + } + return false +} diff --git a/mdl/executor/database_over_association_hops_834_test.go b/mdl/executor/database_over_association_hops_834_test.go new file mode 100644 index 0000000000..6c360a6485 --- /dev/null +++ b/mdl/executor/database_over_association_hops_834_test.go @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// ako/mxcli#834: `database from $ctx/Assoc/Entity …` (the List View / pluggable +// database source over associations, ako/mxcli#721 L5) read its path two +// segments at a time, association then destination. The association source had +// been taught every spelling of a multi-hop path (mendixlabs/mxcli#1467), this +// builder had not: the short form `A1/A2` read A2 as an entity and was refused +// ("entity … does not exist"), and `A1/A2/E2` stored A2 as the destination of +// A1 — so the two builders disagreed about the same path. +// +// Both now resolve hops with one function, so every spelling the association +// source accepts builds the same steps here. +func TestDatabaseOverAssociation_MultiHopSpellings(t *testing.T) { + want := []pages.AttributeRefStep{ + {Association: "Mod.Order_Account", DestinationEntity: "Mod.Account"}, + {Association: "Mod.Account_Flag", DestinationEntity: "Mod.Flag"}, + } + for _, path := range []string{ + "Mod.Order_Account/Mod.Account/Mod.Account_Flag/Mod.Flag", // full form (describe's output) — the control + "Mod.Order_Account/Mod.Account_Flag", // short form + "Mod.Order_Account/Mod.Account_Flag/Mod.Flag", // first destination derived + "Mod.Order_Account/Mod.Account/Mod.Account_Flag", // last destination derived + } { + t.Run(path, func(t *testing.T) { + ds, entity, err := assocHopsPB().buildDatabaseOverAssociationV3(&ast.DataSourceV3{ + Type: "database", ContextVariable: "Order", AssociationPath: path, + }) + if err != nil { + t.Fatalf("database from $Order/%s: %v", path, err) + } + src := ds.(*pages.DatabaseSource) + if len(src.EntitySteps) != len(want) { + t.Fatalf("steps = %+v, want %+v", src.EntitySteps, want) + } + for i := range want { + if src.EntitySteps[i].Association != want[i].Association || src.EntitySteps[i].DestinationEntity != want[i].DestinationEntity { + t.Errorf("step %d = %+v, want %+v", i, src.EntitySteps[i], want[i]) + } + } + if entity != "Mod.Flag" { + t.Errorf("entity = %q, want Mod.Flag", entity) + } + }) + } +} + +// A destination that exists but is not an end of its association was written +// as stated — a step naming an entity the hop does not arrive at. +func TestDatabaseOverAssociation_RefusesDestinationOffTheAssociation(t *testing.T) { + const path = "Mod.Order_Account/Mod.Flag" + _, _, err := assocHopsPB().buildDatabaseOverAssociationV3(&ast.DataSourceV3{ + Type: "database", ContextVariable: "Order", AssociationPath: path, + }) + if err == nil { + t.Fatalf("database from $Order/%s succeeded; Mod.Flag is not an end of Mod.Order_Account", path) + } + if !strings.Contains(err.Error(), "Mod.Flag") { + t.Errorf("error %q does not name the destination", err) + } +} diff --git a/mdl/executor/describe_dollar_quote_1003_test.go b/mdl/executor/describe_dollar_quote_1003_test.go new file mode 100644 index 0000000000..370b9f2f10 --- /dev/null +++ b/mdl/executor/describe_dollar_quote_1003_test.go @@ -0,0 +1,108 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/sdk/javaactions" +) + +// ako/mxcli#1003: describe wrapped long text in `$$ … $$`, and the lexer ends a +// DOLLAR_STRING at the first `$$` it meets — it has no escape. A text holding +// `$$` (a Teamcenter UID in a JSON sample, a Java string literal) ended the +// literal early and `mxcli check` on the output reported +// `token recognition error at: '$$`. Every describer that emits `$$` now goes +// through mdlDollarQuote / mdlSourceBlock, which fall back to a string literal. +// Each case has a control without `$$`, which must keep the dollar form. + +func TestDescribeJavaAction_SourceContainingDollarDollar(t *testing.T) { + for _, c := range []struct { + name, body, extra string + dollarForm bool // the body is written as `as $$ … $$` + }{ + {"control", `return "a" + N;`, "private static final String S = \"x\";", true}, + {"body holds $$", `return "SR::N::10.7$$Otk" + N;`, "private static final String S = \"x\";", false}, + {"extra code holds $$", `return "a" + N;`, "private static final String S = \"C:\\\\x$$y\";", true}, + } { + t.Run(c.name, func(t *testing.T) { + ja := &javaactions.JavaAction{ + Name: "JA_Uid", + ReturnType: &javaactions.StringType{}, + Parameters: []*javaactions.JavaActionParameter{ + {Name: "N", IsRequired: true, ParameterType: &javaactions.StringType{}}, + }, + } + ja.ID = nextID("ja") + dir := t.TempDir() + mprPath := filepath.Join(dir, "App.mpr") + source := javaactions.GenerateSource("ModJ", "JA_Uid", c.body, ja.Parameters, ja.ReturnType, + []string{ + "import com.mendix.systemwideinterfaces.core.IContext;", + "import com.mendix.systemwideinterfaces.core.UserAction;", + }, c.extra, "") + javaDir := filepath.Join(dir, "javasource", "modj", "actions") + if err := os.MkdirAll(javaDir, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(javaDir, "JA_Uid.java"), []byte(source), 0o644); err != nil { + t.Fatal(err) + } + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ReadJavaActionByNameFunc: func(string) (*javaactions.JavaAction, error) { return ja, nil }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withMprPath(mprPath)) + assertNoError(t, describeJavaAction(ctx, ast.QualifiedName{Module: "ModJ", Name: "JA_Uid"})) + out := buf.String() + if got := strings.Contains(out, "\nas $$\n"); got != c.dollarForm { + t.Errorf("dollar-quoted body = %v, want %v:\n%s", got, c.dollarForm, out) + } + s := findStmt[*ast.CreateJavaActionStmt](t, reparse(t, describedSource(ctx, out)), out) + if s.JavaCode != c.body { + t.Errorf("body read back as %q, want %q\n%s", s.JavaCode, c.body, out) + } + if s.ExtraCode != c.extra { + t.Errorf("extra code read back as %q, want %q\n%s", s.ExtraCode, c.extra, out) + } + }) + } +} + +// The other positions describe writes `$$` at: an agent's prompt, a data +// transformer's JSLT step, a JavaScript action's body. Each is checked by +// parsing the helper's output where describe puts it. +func TestDollarQuoteHelpers_ReparseAtEveryPosition(t *testing.T) { + ctx, _ := newMockCtx(t) + for _, text := range []string{ + "line one\nline two", // control + "uid 10.7$$Otklm9\nit's", // holds $$ + "costs 5$", // ends in $ + "C:\\temp$$x\nsecond line", // backslash: the string fallback must keep it + } { + header := describedSource(ctx, "") + + prog, errs := visitor.Build(header + "create agent M.A (UsageType: Task, SystemPrompt: " + mdlDollarQuote(ctx, text) + ");") + if len(errs) > 0 { + t.Fatalf("agent prompt %q does not parse: %v", text, errs) + } + if a := prog.Statements[len(prog.Statements)-1].(*ast.CreateAgentStmt); a.SystemPrompt != text { + t.Errorf("agent prompt read back as %q, want %q", a.SystemPrompt, text) + } + + js := header + "create javascript action M.J() returns Boolean as " + mdlSourceBlock(ctx, text) + ";" + prog, errs = visitor.Build(js) + if len(errs) > 0 { + t.Fatalf("javascript body %q does not parse: %v\n%s", text, errs, js) + } + if j := prog.Statements[len(prog.Statements)-1].(*ast.CreateJavaScriptActionStmt); j.JavaScriptCode != text { + t.Errorf("javascript body read back as %q, want %q", j.JavaScriptCode, text) + } + } +} diff --git a/mdl/executor/describe_message_definition_renamed_attr_test.go b/mdl/executor/describe_message_definition_renamed_attr_test.go new file mode 100644 index 0000000000..275d56ad0c --- /dev/null +++ b/mdl/executor/describe_message_definition_renamed_attr_test.go @@ -0,0 +1,63 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" +) + +// mendixlabs/mxcli#1733 (ako/mxcli#1935): after `alter entity … rename attribute +// Reference to CustomerReference`, the stored member points at +// MyFirstModule.Booking.CustomerReference but keeps OriginalName/ExposedName +// "Reference" until Studio Pro refreshes it. Describe printed `Reference,` — +// the stored element name as if it were the attribute — and exec of that +// output failed: `MyFirstModule.Booking has no attribute "Reference"`. +// +// The member must be printed by the attribute it references, with the stored +// exposed name kept: `CustomerReference as 'Reference'`. +func TestDescribeMessageDefinitionNamesTheReferencedAttribute(t *testing.T) { + mod := mkModule("MyFirstModule") + c := &model.MessageDefinitionCollection{ + BaseElement: model.BaseElement{ID: nextID("mdc")}, + ContainerID: mod.ID, + Name: "MD_Booking", + Definitions: []*model.MessageDefinition{ + {Name: "BookingMessage", Root: &model.MessageDefinitionElement{Kind: "Entity", Entity: "MyFirstModule.Booking", ExposedName: "Booking", + Children: []*model.MessageDefinitionElement{ + // as stored after the rename + {Kind: "Attribute", Attribute: "MyFirstModule.Booking.CustomerReference", OriginalName: "Reference", ExposedName: "Reference"}, + // control: an unrenamed member still prints bare + {Kind: "Attribute", Attribute: "MyFirstModule.Booking.Code", OriginalName: "Code", ExposedName: "Code"}, + }}}, + }, + } + h := mkHierarchy(mod) + withContainer(h, c.ContainerID, mod.ID) + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListMessageDefinitionCollectionsFunc: func() ([]*model.MessageDefinitionCollection, error) { + return []*model.MessageDefinitionCollection{c}, nil + }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execDescribeMessageDefinitionCollection(ctx, ast.QualifiedName{Module: "MyFirstModule", Name: "MD_Booking"})) + out := buf.String() + + if !strings.Contains(out, "CustomerReference as 'Reference',") { + t.Errorf("describe should name the referenced attribute and keep the exposed name:\n%s", out) + } + if !strings.Contains(out, " Code\n") { + t.Errorf("an unrenamed member should print bare:\n%s", out) + } + + s := findStmt[*ast.CreateMessageDefinitionCollectionStmt](t, reparse(t, out), out) + m := s.Definitions[0].Members[0] + if m.Attribute != "CustomerReference" || m.ExposedName != "Reference" { + t.Errorf("reparsed member = attribute %q exposed %q, want CustomerReference / Reference", m.Attribute, m.ExposedName) + } +} diff --git a/mdl/executor/document_names_emitted_755_test.go b/mdl/executor/document_names_emitted_755_test.go index 191404dd9d..6037f5c475 100644 --- a/mdl/executor/document_names_emitted_755_test.go +++ b/mdl/executor/document_names_emitted_755_test.go @@ -48,7 +48,7 @@ func TestContractExternalEntityUsesStudioProName(t *testing.T) { func TestMissingQueueAdviceUsesStudioProName(t *testing.T) { mb := &mock.MockBackend{ListQueuesFunc: func() ([]*types.Queue, error) { return nil, nil }} fb := &flowBuilder{backend: mb} - fb.buildQueueSettings(&ast.QualifiedName{Module: "Ops", Name: "Jobs"}, "call microflow") + fb.buildQueueSettings(&ast.QualifiedName{Module: "Ops", Name: "Jobs"}, nil, "call microflow") if len(fb.errors) != 1 { t.Fatalf("errors = %v, want one", fb.errors) } diff --git a/mdl/executor/external_action_blank_argument_test.go b/mdl/executor/external_action_blank_argument_test.go new file mode 100644 index 0000000000..78fce5761c --- /dev/null +++ b/mdl/executor/external_action_blank_argument_test.go @@ -0,0 +1,153 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// ako/mxcli#1915: Studio Pro 11.16 stores an optional OData action parameter +// left blank as `Argument: ""` (11.14 stored the expression `empty`, measured +// for #1073). DESCRIBE printed it as `additional = )`, which does not parse, so +// every `mdl 1;` rewrite of the microflow was refused with MDL-V1-REBUILD +// "the description does not parse". +// +// The two stored states are distinct and both real, so each has to come back +// as itself. MDL already has the spelling for the first: `Param = nothing`, the +// blank argument of call microflow / call nanoflow, which this extends to call +// external action. `empty` stays `= empty`, and omitting a parameter still +// writes no mapping at all. + +// storedBlankArgumentCall is the reference shape from Odata.IncludeAssociationsMF +// (ako/TestApp, 11.16.0): `additional` blank, beside a required `command` and, +// as the control, a nullable `note` holding the expression `empty`. +func storedBlankArgumentCall() *microflows.CallExternalAction { + return µflows.CallExternalAction{ + ConsumedODataService: "Odata.Bug1073", + Name: "RunCommand", + ParameterMappings: []*microflows.ExternalActionParameterMapping{ + {ParameterName: "command", Argument: "$Cmd", ParameterDataType: "String", CanBeEmpty: false}, + {ParameterName: "additional", Argument: "", ParameterDataType: "String", CanBeEmpty: true}, + {ParameterName: "note", Argument: "empty", ParameterDataType: "String", CanBeEmpty: true}, + }, + } +} + +// buildExternalCallFrom parses one `call external action` statement and runs +// the real builder on it against canBeEmptyMetadata's contract. +func buildExternalCallFrom(t *testing.T, stmt string) (*ast.CallExternalActionStmt, *microflows.CallExternalAction, []string) { + t.Helper() + svcID, modID := model.ID("svc-1"), model.ID("mod-1") + mb := &mock.MockBackend{ + ListConsumedODataServicesFunc: func() ([]*model.ConsumedODataService, error) { + return []*model.ConsumedODataService{{ + BaseElement: model.BaseElement{ID: svcID}, + ContainerID: modID, + Name: "Bug1073", + Metadata: canBeEmptyMetadata, + }}, nil + }, + } + fb := &flowBuilder{ + posX: 100, posY: 100, spacing: HorizontalSpacing, backend: mb, + hierarchy: &ContainerHierarchy{moduleNames: map[model.ID]string{modID: "Odata"}}, + varTypes: map[string]string{"Cmd": "String"}, + declaredVars: map[string]string{"Cmd": "String"}, + } + prog, errs := visitor.Build("create microflow Odata.ACT_Probe ($Cmd: String)\nbegin\n " + stmt + "\nend;") + if len(errs) > 0 { + t.Fatalf("the statement does not parse — the reported symptom: %v\n%s", errs, stmt) + } + call := prog.Statements[0].(*ast.CreateMicroflowStmt).Body[0].(*ast.CallExternalActionStmt) + fb.addCallExternalActionAction(call) + for _, obj := range fb.objects { + if act, ok := obj.(*microflows.ActionActivity); ok { + if built, ok := act.Action.(*microflows.CallExternalAction); ok { + return call, built, fb.GetErrors() + } + } + } + t.Fatal("the builder produced no CallExternalAction") + return nil, nil, nil +} + +func runCommandContract(t *testing.T) *types.EdmAction { + t.Helper() + doc, err := types.ParseEdmx(canBeEmptyMetadata) + if err != nil || len(doc.Actions) != 1 { + t.Fatalf("parsing the contract: %v", err) + } + return doc.Actions[0] +} + +func externalMappingShape(pms []*microflows.ExternalActionParameterMapping) string { + var parts []string + for _, pm := range pms { + parts = append(parts, pm.ParameterName+"="+`"`+pm.Argument+`"`) + if pm.CanBeEmpty { + parts[len(parts)-1] += "?" + } + } + return strings.Join(parts, ", ") +} + +func TestCallExternalAction_BlankArgumentRoundTrips(t *testing.T) { + stored := storedBlankArgumentCall() + described := newTestExecutor().formatAction(stored, nil, nil) + + if strings.Contains(described, "= )") || strings.Contains(described, "= ,") { + t.Fatalf("a blank argument is described with no value, which does not parse: %s", described) + } + if !strings.Contains(described, "additional = nothing") { + t.Errorf("the blank argument is not described as `= nothing`: %s", described) + } + // Control: the `empty` expression is a value and is still spelled. + if !strings.Contains(described, "note = empty") { + t.Errorf("the `empty` mapping lost its spelling: %s", described) + } + + stmt, rebuilt, errs := buildExternalCallFrom(t, described) + if len(errs) > 0 { + t.Fatalf("rebuilding the description reported errors: %v", errs) + } + if got, want := externalMappingShape(rebuilt.ParameterMappings), externalMappingShape(stored.ParameterMappings); got != want { + t.Errorf("describe -> build changed the mappings\n got: %s\nwant: %s\ndescription: %s", got, want, described) + } + + // The described statement names every parameter, so the contract check + // that `check --references` and exec run accepts it. + action := runCommandContract(t) + if err := checkExternalActionParameters(externalCall{stmt: stmt, flow: "Odata.ACT_Probe"}, action); err != nil { + t.Errorf("the described call is refused against its own contract: %v", err) + } +} + +// Control: the 11.14 form, every unfilled argument the expression `empty`, +// describes and rebuilds byte-identically, and omitting a parameter is still +// not the blank: a required one left out is refused as before. +func TestCallExternalAction_EmptyArgumentAndRequiredStillHold(t *testing.T) { + stored := storedBlankArgumentCall() + stored.ParameterMappings[1].Argument = "empty" + described := newTestExecutor().formatAction(stored, nil, nil) + if !strings.Contains(described, "additional = empty") { + t.Fatalf("the `empty` argument is not described as `= empty`: %s", described) + } + _, rebuilt, _ := buildExternalCallFrom(t, described) + if got, want := externalMappingShape(rebuilt.ParameterMappings), externalMappingShape(stored.ParameterMappings); got != want { + t.Errorf("describe -> build changed the mappings\n got: %s\nwant: %s", got, want) + } + + stmt, _, _ := buildExternalCallFrom(t, "call external action Odata.Bug1073.RunCommand(additional = empty, note = empty);") + action := runCommandContract(t) + if err := checkExternalActionParameters(externalCall{stmt: stmt, flow: "Odata.ACT_Probe"}, action); err == nil { + t.Error("leaving the required `command` out was accepted") + } +} diff --git a/mdl/executor/flow_unsupported_guard.go b/mdl/executor/flow_unsupported_guard.go new file mode 100644 index 0000000000..4a4dccf703 --- /dev/null +++ b/mdl/executor/flow_unsupported_guard.go @@ -0,0 +1,32 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// FlowUnsupportedRule is check's and exec's preflight refusal of a `create or +// modify` that exec would carry out as a whole-flow rebuild (mdl 0, the +// MDL-V1-REBUILD warning) of a flow holding activities MDL cannot express. +// +// The rebuild itself is refused by the guard in buildMicroflowFromStmt +// (validate_unsupported_actions.go), and a drop + create by +// validateDropRecreateKeepsActivities. But the rebuild guard only fires when +// exec reaches the statement, after every statement before it has been +// written — a half-applied script. Predicting it in the verdict pass refuses +// the script before anything runs, as the drop + create guard already does. +const FlowUnsupportedRule = "MDL-FLOW-UNSUPPORTED" + +// unsupportedRebuildViolation is the verdict pass's report of that refusal, +// worded as exec's (unsupportedActionLossError). +func unsupportedRebuildViolation(kind string, qn ast.QualifiedName, types []string) linter.Violation { + return linter.Violation{ + RuleID: FlowUnsupportedRule, + Severity: linter.SeverityError, + Message: "exec would refuse this statement: " + unsupportedActionLossError(kind, qn.String(), + "the whole-flow rebuild exec runs for this statement", types).Error(), + Location: linter.Location{Module: qn.Module, DocumentType: kind, DocumentName: qn.Name}, + } +} diff --git a/mdl/executor/flow_unsupported_guard_test.go b/mdl/executor/flow_unsupported_guard_test.go new file mode 100644 index 0000000000..121398d202 --- /dev/null +++ b/mdl/executor/flow_unsupported_guard_test.go @@ -0,0 +1,185 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "context" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// The unsupported-activity guards (validate_unsupported_actions.go) on Studio +// Pro's own flows: WorkflowCommons in ako/TestApp holds +// Microflows$GenerateJumpToOptionsAction and ApplyJumpToOptionAction, which +// mxcli has no MDL for. A drop + create from the description and the mdl 0 +// whole-flow rebuild must be refused before anything is written — the rebuild +// in the verdict pass too (FlowUnsupportedRule), so a script is not left +// half-applied; the in-place splice under mdl 1 keeps the activity and stays +// allowed (mendixlabs/mxcli#1735, #1738). + +const unsupportedFlow = "WorkflowCommons.ACT_Workflow_JumpTo" + +// describeQualified prints describe output for a flow named Module.Name. +func describeQualified(t *testing.T, exec *Executor, kind, name string) string { + t.Helper() + var buf bytes.Buffer + saved := exec.output + exec.output = &buf + defer func() { exec.output = saved }() + if err := afRun(t, exec, "describe "+kind+" "+name+";"); err != nil { + t.Fatal(err) + } + return buf.String() +} + +func verdictRules(t *testing.T, exec *Executor, src string) map[string]string { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse: %v\n%s", errs[0], src) + } + got := map[string]string{} + for _, v := range exec.CheckFlowVerdicts(prog) { + got[v.RuleID] += v.Message + "\n" + } + return got +} + +// withoutHeader strips a leading `mdl n;` line from describe output. +func withoutHeader(s string) string { + if i := strings.Index(s, "\n"); i >= 0 && strings.HasPrefix(strings.TrimSpace(s), "mdl ") { + return s[i+1:] + } + return s +} + +func dropRecreateRefusals(t *testing.T, exec *Executor, src string) []error { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse: %v\n%s", errs[0], src) + } + return validateDropRecreateKeepsActivities(exec.newExecContext(context.Background()), prog) +} + +func TestUnsupportedActionGuard_DropAndCreateIsRefused(t *testing.T) { + exec, _ := openTestAppCopy(t) + described := describeQualified(t, exec, "microflow", unsupportedFlow) + if !strings.Contains(described, "Unsupported action: Microflows$GenerateJumpToOptionsAction") { + t.Fatalf("control: %s no longer holds the unsupported action:\n%s", unsupportedFlow, described) + } + src := "mdl 1;\ndrop microflow " + unsupportedFlow + ";\n" + withoutHeader(described) + errs := dropRecreateRefusals(t, exec, src) + if len(errs) != 1 || !strings.Contains(errs[0].Error(), "Microflows$GenerateJumpToOptionsAction") { + t.Fatalf("drop + create of %s: want one refusal naming the action, got %v", unsupportedFlow, errs) + } +} + +// Control: a drop + create of a flow with nothing unsupported is a deliberate +// rebuild and stays allowed. +func TestUnsupportedActionGuard_PlainRebuildIsAllowed(t *testing.T) { + exec, _ := openTestAppCopy(t) + const plain = "WorkflowCommons.SUB_WorkflowTaskDetail_Delete" + described := describeQualified(t, exec, "microflow", plain) + if strings.Contains(described, "Unsupported action") { + t.Fatalf("control subject %s holds an unsupported action:\n%s", plain, described) + } + src := "mdl 1;\ndrop microflow " + plain + ";\n" + withoutHeader(described) + if errs := dropRecreateRefusals(t, exec, src); len(errs) != 0 { + t.Fatalf("a flow without unsupported actions was refused: %v", errs) + } +} + +// Under mdl 0 a `create or modify` the splice cannot make is rebuilt as a +// whole (MDL-V1-REBUILD warning) — the same loss, refused the same way. The +// insert at the start does not fit between the stored start event and the +// decision, which the script draws where they are, so the splice refuses it +// and mdl 0 falls back to the rebuild. +func TestUnsupportedActionGuard_Mdl0RebuildIsRefused(t *testing.T) { + exec, _ := openTestAppCopy(t) + // ACT_Workflow_JumpTo cannot be built from its description at all — the + // unsupported action declares a variable the rest reads — so its verdict + // is "unpredictable", not "rebuilt". This one builds. + const applyFlow = "WorkflowCommons.ACT_WorkflowJumpToDetails_Apply" + described := withoutHeader(describeQualified(t, exec, "microflow", applyFlow)) + // Taking the then-branch's return away changes where the decision's paths + // end, which the splice cannot make: mdl 1 refuses it, mdl 0 rebuilds the + // flow (MDL-V1-REBUILD warning). + i := strings.Index(described, "show page WorkflowCommons.Workflow_ActionConfirmation") + if i < 0 { + t.Fatalf("control: the then-branch's show page is gone:\n%s", described) + } + start := i + strings.Index(described[i:], "\n") + 1 + end := start + strings.Index(described[start:], "return;\n") + len("return;\n") + src := "mdl 0;\n" + described[:start] + described[end:] + if _, errs := visitor.Build(src); len(errs) > 0 { + t.Fatalf("the edited script does not parse: %v\n%s", errs, src) + } + rules := verdictRules(t, exec, src) + if _, rebuilt := rules[FlowRebuildRule]; !rebuilt { + t.Fatalf("control: this edit no longer reaches the mdl 0 rebuild; verdicts: %v", rules) + } + if msg, ok := rules[FlowUnsupportedRule]; !ok || !strings.Contains(msg, "Microflows$ApplyJumpToOptionAction") { + t.Fatalf("the mdl 0 rebuild of %s was not refused naming the action; verdicts: %v", applyFlow, rules) + } + + // exec without the verdict pass (--no-check, or the executor driven + // directly) refuses the rewrite too, and the activity stays. + if err := agreeExec(t, exec, src); err == nil || !strings.Contains(err.Error(), "cannot express") { + t.Fatalf("exec of the rebuild = %v, want the refusal", err) + } + if got := describeQualified(t, exec, "microflow", applyFlow); !strings.Contains(got, "Unsupported action: Microflows$ApplyJumpToOptionAction") { + t.Fatalf("the refused rebuild still lost the activity:\n%s", got) + } +} + +// Control: under mdl 1 the in-place splice keeps the unsupported activity, so +// an edit of the flow is not refused, and exec keeps the action. +func TestUnsupportedActionGuard_SpliceKeepsTheActivity(t *testing.T) { + exec, out := openTestAppCopy(t) + described := describeQualified(t, exec, "microflow", unsupportedFlow) + const old = "'It is not allowed to apply jump to activity for this workflow.'" + if !strings.Contains(described, old) { + t.Fatalf("control: the message to edit is gone:\n%s", described) + } + src := "mdl 1;\n" + strings.Replace(withoutHeader(described), old, "'Jump to is not allowed for this workflow.'", 1) + if msg, ok := verdictRules(t, exec, src)[FlowUnsupportedRule]; ok { + t.Fatalf("an in-place splice was refused:\n%s", msg) + } + if err := agreeExec(t, exec, src); err != nil { + t.Fatalf("exec: %v\n%s", err, out.String()) + } + if got := describeQualified(t, exec, "microflow", unsupportedFlow); !strings.Contains(got, "Jump to is not allowed") || + !strings.Contains(got, "Unsupported action: Microflows$GenerateJumpToOptionsAction") { + t.Fatalf("after the splice the flow lacks the edit or the unsupported activity:\n%s", got) + } +} + +// loopEditFlowOfQualified loads a stored microflow by its qualified name. +func loopEditFlowOfQualified(t *testing.T, exec *Executor, qn string) *microflows.Microflow { + t.Helper() + ctx := exec.newExecContext(context.Background()) + h, err := getHierarchy(ctx) + if err != nil { + t.Fatal(err) + } + all, err := ctx.Backend.ListMicroflows() + if err != nil { + t.Fatal(err) + } + for _, m := range all { + if h.GetModuleName(h.FindModuleID(m.ContainerID))+"."+m.Name == qn { + mf, err := ctx.Backend.GetMicroflow(m.ID) + if err != nil { + t.Fatal(err) + } + return mf + } + } + t.Fatalf("%s not found", qn) + return nil +} diff --git a/mdl/executor/identifier_quoting.go b/mdl/executor/identifier_quoting.go index 60e1cadfca..fafa831aee 100644 --- a/mdl/executor/identifier_quoting.go +++ b/mdl/executor/identifier_quoting.go @@ -48,6 +48,37 @@ func mdlQuote(ctx *ExecContext, s string) string { return mdl0Quote(s) } +// mdlDollarQuote writes s as a `$$ … $$` literal where one can hold it, and as +// the single-quoted literal mdlQuote writes where it cannot: the lexer ends a +// DOLLAR_STRING at the first `$$` it meets and has no escape, so a text +// containing `$$` (a Teamcenter UID, ako/mxcli#1003) — or ending in `$`, whose +// last `$` the closing delimiter would pair with — would end the literal early +// and the rest would fail to lex. Every describer that emits a dollar-quoted +// value at a position that also accepts a STRING_LITERAL goes through here. +func mdlDollarQuote(ctx *ExecContext, s string) string { + if dollarQuotable(s) { + return "$$" + s + "$$" + } + return mdlQuote(ctx, s) +} + +// mdlSourceBlock writes a block of source code (a Java or JavaScript action +// body, its EXTRA CODE, a JSLT step) as `$$` on its own line, the code, and +// `$$` on its own line — or, for code that holds `$$`, as mdlQuote's string +// literal, which those positions accept as well (ako/mxcli#1003). +func mdlSourceBlock(ctx *ExecContext, code string) string { + if dollarQuotable("\n" + code + "\n") { + return "$$\n" + code + "\n$$" + } + return mdlQuote(ctx, code) +} + +// dollarQuotable reports whether `$$` + s + `$$` lexes as one DOLLAR_STRING +// whose content is s. +func dollarQuotable(s string) bool { + return !strings.Contains(s, "$$") && !strings.HasSuffix(s, "$") +} + // mdlIdent renders name as an MDL identifier suitable for DESCRIBE output, // double-quoting it when it would not lex as a bare IDENTIFIER — e.g. when it // collides with a reserved keyword ("List", "Column", "Template", …). This keeps diff --git a/mdl/executor/listview_database_over_association_test.go b/mdl/executor/listview_database_over_association_test.go index ea64f53fe3..e75534c1ae 100644 --- a/mdl/executor/listview_database_over_association_test.go +++ b/mdl/executor/listview_database_over_association_test.go @@ -130,7 +130,7 @@ func TestDatabaseOverAssociationRefusedOutsideListViewAndPluggable(t *testing.T) if _, _, err := pb.buildDataSourceV3(ds); err == nil { t.Error("buildDataSourceV3 accepted `database from $currentObject/…`") } - if _, _, err := pb.buildPluggableDataSourceV3(ds); err != nil { + if _, _, err := pb.buildPluggableDataSourceV3(ds, ""); err != nil { t.Errorf("a pluggable widget was refused: %v", err) } } diff --git a/mdl/executor/microflow_annotation_unattached_test.go b/mdl/executor/microflow_annotation_unattached_test.go new file mode 100644 index 0000000000..e489be0338 --- /dev/null +++ b/mdl/executor/microflow_annotation_unattached_test.go @@ -0,0 +1,102 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// ako/mxcli#1933 (mendixlabs/mxcli#1731): "annotations not linked to an +// activity are missing in MDL". The reporter's flow has two notes; Note2 is +// wired to two activities and is described, Note1 is not described at all. +// +// A note with no AnnotationFlow was already printed as a free note. What +// vanished was a note whose only links go where the describer never prints an +// @annotation: to the start event, or to another note. Such a note counted as +// "referenced", so it was not free, and its target emits nothing, so it was +// not attached either. +// +// Such a note is now described as a free note — which is also what makes the +// canonical describe keep the first statement's @position (useDerivedFlowLayout +// asks collectFreeAnnotations), so it comes back free — and the link MDL cannot +// state is flagged rather than dropped in silence. + +func describeWithFreeNotes(t *testing.T, mf *microflows.Microflow) string { + t.Helper() + ctx := &ExecContext{} + lines := formatMicroflowActivities(ctx, mf, nil, nil) + return strings.Join(prependFreeAnnotationLines(ctx, mf.ObjectCollection, lines), "\n") +} + +func reporterNotes() (*microflows.Annotation, *microflows.Annotation) { + return note("note1", "Note1", model.Point{X: 100, Y: 300}, model.Size{Width: 200, Height: 50}), + note("note2", "Note2", model.Point{X: 383, Y: 89}, model.Size{Width: 230, Height: 50}) +} + +func sharedNote2Flows() []*microflows.AnnotationFlow { + return []*microflows.AnnotationFlow{ + {BaseElement: model.BaseElement{ID: "af1"}, OriginID: "note2", DestinationID: "a1"}, + {BaseElement: model.BaseElement{ID: "af2"}, OriginID: "note2", DestinationID: "a2"}, + } +} + +func TestUnattachedNote_EveryNoteIsDescribed(t *testing.T) { + for name, extra := range map[string]*microflows.AnnotationFlow{ + "linked to the start event": {OriginID: "note1", DestinationID: "start"}, + "linked to another note": {OriginID: "note1", DestinationID: "note2"}, + "linked to nothing": nil, + } { + t.Run(name, func(t *testing.T) { + flows := sharedNote2Flows() + if extra != nil { + flows = append(flows, extra) + } + n1, n2 := reporterNotes() + body := describeWithFreeNotes(t, mfWithNotes(flows, n1, n2)) + if !strings.Contains(body, "'Note1'") { + t.Fatalf("Note1 is missing from the description:\n%s", body) + } + + notes, rebuiltFlows, errs := rebuildFromBody(t, body) + if len(errs) > 0 { + t.Fatalf("rebuild reported errors: %v\n%s", errs, body) + } + if len(notes) != 2 { + t.Fatalf("rebuilt %d notes, want 2:\n%s", len(notes), body) + } + // Note2 keeps its two links; Note1 comes back unattached. + byNote := map[string]int{} + captions := map[model.ID]string{} + for _, n := range notes { + captions[n.ID] = n.Caption + } + for _, f := range rebuiltFlows { + byNote[captions[f.OriginID]]++ + } + if byNote["Note2"] != 2 || byNote["Note1"] != 0 { + t.Errorf("links per note = %v, want Note2: 2, Note1: 0\n%s", byNote, body) + } + }) + } +} + +// A link MDL cannot state is not dropped in silence: the description says so. +func TestUnattachedNote_UnstatableLinkIsFlagged(t *testing.T) { + n1, n2 := reporterNotes() + flows := append(sharedNote2Flows(), µflows.AnnotationFlow{OriginID: "note1", DestinationID: "start"}) + body := describeWithFreeNotes(t, mfWithNotes(flows, n1, n2)) + if !strings.Contains(body, "-- WARNING") || !strings.Contains(body, "start") { + t.Errorf("the note's link to the start event is lost without a warning:\n%s", body) + } + + // Control: a note linked to nothing has nothing to lose, so no warning. + n1, n2 = reporterNotes() + body = describeWithFreeNotes(t, mfWithNotes(sharedNote2Flows(), n1, n2)) + if strings.Contains(body, "-- WARNING") { + t.Errorf("a free note was flagged:\n%s", body) + } +} diff --git a/mdl/executor/nanoflow_external_object_test.go b/mdl/executor/nanoflow_external_object_test.go new file mode 100644 index 0000000000..a41902fd66 --- /dev/null +++ b/mdl/executor/nanoflow_external_object_test.go @@ -0,0 +1,23 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// mendixlabs/mxcli#1735 (ako/mxcli#1939): measured on mxbuild 11.12.6, both +// external-object activities in a nanoflow are CE0009 "This action is not +// supported in nanoflows." — check passed them. +func TestNanoflowRefusesExternalObjectActivities(t *testing.T) { + for _, del := range []bool{false, true} { + body := []ast.MicroflowStatement{&ast.ExternalObjectStmt{Delete: del, Variable: "Customer"}} + errs := validateNanoflowBody(body) + if len(errs) != 1 || !strings.Contains(errs[0], "external object") { + t.Errorf("delete=%v: want one refusal naming the external object activity, got %v", del, errs) + } + } +} diff --git a/mdl/executor/nanoflow_validation.go b/mdl/executor/nanoflow_validation.go index 212495e0a7..3fa21ddac1 100644 --- a/mdl/executor/nanoflow_validation.go +++ b/mdl/executor/nanoflow_validation.go @@ -54,6 +54,14 @@ func validateNanoflowStatements(stmts []ast.MicroflowStatement, errors *[]string // The manual QA test plan (docs/15-testing/nanoflow-test-cases.md §4.1) lists // all disallowed actions and should be updated in parallel. func checkDisallowedNanoflowAction(stmt ast.MicroflowStatement) string { + // Measured on mxbuild 11.12.6: CE0009 "This action is not supported in + // nanoflows." on both external-object activities (mendixlabs/mxcli#1735). + if s, ok := stmt.(*ast.ExternalObjectStmt); ok { + if s.Delete { + return "delete external object is not allowed in nanoflows" + } + return "send external object is not allowed in nanoflows" + } switch stmt.(type) { case *ast.RaiseErrorStmt: return "ErrorEvent is not allowed in nanoflows" diff --git a/mdl/executor/pluggable_database_source.go b/mdl/executor/pluggable_database_source.go index d5e5123a0d..bee7493510 100644 --- a/mdl/executor/pluggable_database_source.go +++ b/mdl/executor/pluggable_database_source.go @@ -36,7 +36,32 @@ func (pb *pageBuilder) customWidgetDatabaseSourceSupported() bool { // CustomWidgetDatabaseSource, its WHERE stored as constraint rows // (mendixlabs/mxcli#1399). A widget with its own builder keeps // buildDataSourceV3's refusal. -func (pb *pageBuilder) buildPluggableDataSourceV3(ds *ast.DataSourceV3) (pages.DataSource, string, error) { +// +// widgetName is the widget the source belongs to. A direct-entity source +// (`database from E where …`) is written as an XPath source on every version — +// the two are equivalent — EXCEPT where the stored widget of that name already +// holds a 9.x database source: then the rewrite keeps it, when its WHERE fits +// constraint rows (ako/mxcli#1208, ADR-0008: a rewrite keeps what it was not +// asked to change; describe prints both alike, so the statement cannot say). +func (pb *pageBuilder) buildPluggableDataSourceV3(ds *ast.DataSourceV3, widgetName string) (pages.DataSource, string, error) { + if ds != nil && ds.Type == "database" && ds.AssociationPath == "" && + pb.customWidgetDatabaseSourceSupported() && pb.storedPluggables.storedUsesDatabaseSource(widgetName) { + built, entity, err := pb.buildDataSourceV3(ds) + if err != nil { + return nil, "", err + } + if src, ok := built.(*pages.DatabaseSource); ok { + if constraints, cerr := parseDatabaseConstraints(src.XPathConstraint, entity); cerr == nil { + src.TypeName = "CustomWidgets$CustomWidgetDatabaseSource" + src.UseConstraints = true + src.Constraints = constraints + src.XPathConstraint = "" + } + // A WHERE the rows cannot hold stays an XPath source: changing the + // type is the lesser drift than dropping a term. + } + return built, entity, nil + } if ds == nil || ds.Type != "database" || ds.AssociationPath == "" { return pb.buildDataSourceV3(ds) } diff --git a/mdl/executor/pluggable_database_source_carry_test.go b/mdl/executor/pluggable_database_source_carry_test.go new file mode 100644 index 0000000000..f6b4480bfb --- /dev/null +++ b/mdl/executor/pluggable_database_source_carry_test.go @@ -0,0 +1,39 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +// ako/mxcli#1208: on a 9.x project, rewriting a pluggable widget whose stored +// source is Studio Pro 9's "Database" source (CustomWidgetDatabaseSource) wrote +// it back as an XPath source, though describe prints both as `database from E +// where …` and the statement asked for no change of type. The rewrite now keeps +// the stored type; this pins the detection it rests on. The end-to-end run (a +// 9.24 project, a grid edited by its sort only) is in the PR: the base build +// writes CustomWidgetXPathSource, the fix keeps the DatabaseSource and its +// constraint rows. +func TestStoredUsesDatabaseSource(t *testing.T) { + db := bson.D{{Key: "Properties", Value: bson.A{int32(2), bson.D{ + {Key: "Value", Value: bson.D{{Key: "DataSource", Value: bson.D{{Key: "$Type", Value: "CustomWidgets$CustomWidgetDatabaseSource"}}}}}, + }}}} + xp := bson.D{{Key: "Properties", Value: bson.A{int32(2), bson.D{ + {Key: "Value", Value: bson.D{{Key: "DataSource", Value: bson.D{{Key: "$Type", Value: "CustomWidgets$CustomWidgetXPathSource"}}}}}, + }}}} + pt := &pluggablePassthrough{byName: map[string]*storedPluggable{ + "dg": {obj: db}, + "gal": {obj: xp}, + }} + if !pt.storedUsesDatabaseSource("dg") { + t.Error("a stored CustomWidgetDatabaseSource must be detected, nested in the Object") + } + if pt.storedUsesDatabaseSource("gal") { + t.Error("an XPath source is not a database source") + } + if pt.storedUsesDatabaseSource("missing") || (*pluggablePassthrough)(nil).storedUsesDatabaseSource("dg") { + t.Error("an unknown widget or no stored page is not a database source") + } +} diff --git a/mdl/executor/pluggable_xpath_over_association_test.go b/mdl/executor/pluggable_xpath_over_association_test.go index 0752e7cd10..33624efc33 100644 --- a/mdl/executor/pluggable_xpath_over_association_test.go +++ b/mdl/executor/pluggable_xpath_over_association_test.go @@ -91,7 +91,7 @@ func TestBuildPluggableDataSource_DatabaseOverAssociationOn10(t *testing.T) { t.Fatalf("parse: %v", errs) } w := prog.Statements[0].(*ast.CreatePageStmtV3).Widgets[0] - built, entity, err := pb.buildPluggableDataSourceV3(w.GetDataSource()) + built, entity, err := pb.buildPluggableDataSourceV3(w.GetDataSource(), "") if err != nil { t.Fatalf("buildPluggableDataSourceV3: %v", err) } diff --git a/mdl/executor/rule_validation.go b/mdl/executor/rule_validation.go index 6ce6fc20dd..fd93568272 100644 --- a/mdl/executor/rule_validation.go +++ b/mdl/executor/rule_validation.go @@ -49,6 +49,8 @@ func checkDisallowedRuleAction(stmt ast.MicroflowStatement) string { return "a rule cannot commit objects" case *ast.RollbackStmt: return "a rule cannot roll back objects" + case *ast.ExternalObjectStmt: + return "a rule cannot send or delete external objects" // Client interaction — a rule has no client to talk to. case *ast.ShowPageStmt: diff --git a/mdl/executor/validate.go b/mdl/executor/validate.go index 24a1329ee2..a50eeadd97 100644 --- a/mdl/executor/validate.go +++ b/mdl/executor/validate.go @@ -579,6 +579,12 @@ func validateProgramWithWarnings(ctx *ExecContext, prog *ast.Program) ([]error, // CE0069, after exec wrote it (ako/mxcli#1645). Replayed in statement // order, judged by the verdict exec gives. errors = append(errors, validateMemberNameClashes(ctx, prog)...) + // A drop + create of a flow whose stored body holds an activity MDL cannot + // express: the create writes it without that activity (mendixlabs/mxcli#1735). + errors = append(errors, validateDropRecreateKeepsActivities(ctx, prog)...) + // toString() into a String(n): builds, and overflows the attribute + // on commit (mendixlabs/mxcli#1515). Warns, never blocks. + validateToStringIntoShortAttribute(ctx, prog, sc) return errors, sc.warnings } @@ -1068,6 +1074,10 @@ func validateWithContext(ctx *ExecContext, stmt ast.Statement, sc *scriptContext } case *ast.CreatePublishedRestServiceStmt: return validatePublishedRestAuthMicroflow(ctx, s.Authentication, sc) + case *ast.CreateMessageDefinitionCollectionStmt: + return messageDefinitionMemberErrors(ctx, sc, s.Name.String(), s.Definitions...) + case *ast.CreateMessageDefinitionStmt: + return messageDefinitionMemberErrors(ctx, sc, s.Name.String(), s.Definition) case *ast.AlterPublishedRestServiceStmt: for _, a := range s.Actions { if set, ok := a.(*ast.PublishedRestSetAction); ok { diff --git a/mdl/executor/validate_assoc_hop_start.go b/mdl/executor/validate_assoc_hop_start.go new file mode 100644 index 0000000000..45e0cb4707 --- /dev/null +++ b/mdl/executor/validate_assoc_hop_start.go @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// MDL-ASSOCDS03 — a data source over associations whose first association does +// not touch the entity it starts from. +// +// `$Order/M.Account_Flag/M.Flag` names an association between Account and Flag +// from an Order. Every other check passes it — the association exists and Flag +// is one of its ends — so exec wrote an EntityRefStep that cannot be followed +// from the context object (ako/mxcli#834). exec now refuses it +// (resolveAssociationHops); this reports it at check time, from the same facts. +// +// Only what can be proved is reported: the association must be known (project +// or script), and every entity on the context entity's generalization chain +// must be loaded — an entity outside them (the System module's, say) could +// inherit from an end, so it ends the check silently. +func (v *attributeScopeValidator) checkAssocHopStart(w *ast.WidgetV3, enclosing dataContext) { + if v.assocs == nil { + return + } + ds := w.GetDataSource() + if ds == nil { + return + } + var path string + switch { + case ds.Type == "association": + path = ds.Reference + case ds.Type == "database" && ds.AssociationPath != "": + path = ds.AssociationPath + default: + return + } + start := v.dataSourceContextEntity(ds, enclosing) + if start == "" || path == "" { + return + } + first, _, _ := strings.Cut(path, "/") + qn := strings.TrimSpace(first) + if qn == "" { + return + } + if !strings.Contains(qn, ".") { + mod, _, _ := strings.Cut(start, ".") + qn = mod + "." + qn + } + shape, ok := v.assocs[strings.ToLower(qn)] + if !ok || !v.provablyOffAssociation(start, shape) { + return + } + v.errs = append(v.errs, fmt.Sprintf( + "%s data source %s: %s joins %s and %s, and does not start at %s — "+ + "the step it writes cannot be followed from the context object. Follow an association of %s first [MDL-ASSOCDS03]", + widgetLabel(w.Name, w.Type), dataSourceDisplay(ds), shape.qn, shape.from, shape.to, start, start)) +} + +// provablyOffAssociation reports whether entity is known to be neither end of +// the association nor a specialization of one: every entity on its +// generalization chain is in the script or the loaded domain models, and none +// of them is an end. +func (v *attributeScopeValidator) provablyOffAssociation(entity string, shape assocShape) bool { + seen := map[string]bool{} + for e := entity; e != ""; { + key := strings.ToLower(e) + if seen[key] { + return false + } + seen[key] = true + if strings.EqualFold(e, shape.from) || strings.EqualFold(e, shape.to) { + return false + } + if gen, ok := v.generalizations[key]; ok { + e = gen + continue + } + if v.pb == nil { + return false + } + gen, known := v.pb.entityGeneralization(e) + if !known { + return false + } + e = gen + } + return true +} diff --git a/mdl/executor/validate_datasource_args.go b/mdl/executor/validate_datasource_args.go index f3d582f924..38cba040b8 100644 --- a/mdl/executor/validate_datasource_args.go +++ b/mdl/executor/validate_datasource_args.go @@ -32,7 +32,10 @@ // page parameter of the exact type, no dataview → CE1571 // // So an enclosing data container of a compatible type DOES supply the argument, -// at any depth, and a page parameter does not. The rule now says exactly that; +// at any depth, and a page parameter does not — FROM MENDIX 10.1. Before it, a +// page's or snippet's own parameter supplies it too, for data sources and +// actions alike (bisected for ako/mxcli#1206: 9.24.30 and 10.0.0 build clean, +// 10.1.0 onwards CE1571); see paramsSupplyFlowArguments. The rule now says exactly that; // before this it rejected a page that builds, with a message asserting as fact // that Mendix "does not fill it in, even when an object of the right type is in // scope". @@ -211,17 +214,41 @@ func validateFlowArguments(ctx *ExecContext, params []ast.PageParameter, widgets if len(sigs) == 0 { return nil } - return validateFlowArgumentsIn(params, widgets, sigs, entityCompatibility(ctx)) + return validateFlowArgumentsScoped(params, widgets, sigs, entityCompatibility(ctx), + paramsSupplyFlowArguments(projectVersionAtLeast(ctx.Backend))) +} + +// paramsSupplyFlowArguments reports whether the page's or snippet's own +// parameters supply flow arguments, as an enclosing data container does. They +// do before Mendix 10.1 and not from it (ako/mxcli#1206), bisected with one +// script — snippet and page parameter, data source and button action — across +// 9.24.30, 10.0.0 (0 errors) and 10.1.0 … 11.12.6 (CE1571 on each). An unknown +// version gets the 10.1+ rule, the one every project used to get. +func paramsSupplyFlowArguments(atLeast func(major, minor int) bool) bool { + return atLeast != nil && !atLeast(10, 1) } // validateFlowArgumentsIn is the whole rule with its two project-dependent // inputs — the flow signatures and the entity-compatibility test — handed in, so -// it can be measured against the shapes `mx check` was run on. +// it can be measured against the shapes `mx check` was run on. It is the 10.1+ +// rule; validateFlowArgumentsScoped takes the version's answer. func validateFlowArgumentsIn( pageParams []ast.PageParameter, widgets []*ast.WidgetV3, sigs map[string]*flowSignature, compatible func(contextQN, paramQN string) bool, +) []string { + return validateFlowArgumentsScoped(pageParams, widgets, sigs, compatible, false) +} + +// validateFlowArgumentsScoped is validateFlowArgumentsIn with paramsInScope: +// when set, the parameters' entities are the root data context. +func validateFlowArgumentsScoped( + pageParams []ast.PageParameter, + widgets []*ast.WidgetV3, + sigs map[string]*flowSignature, + compatible func(contextQN, paramQN string) bool, + paramsInScope bool, ) []string { paramEntities := make(map[string]string, len(pageParams)) for _, p := range pageParams { @@ -262,8 +289,16 @@ func validateFlowArgumentsIn( walk(c, inner, childControlBarOf) } } + root := dataContext{} + if paramsInScope { + for _, p := range pageParams { + if qn := p.EntityType.String(); qn != "" && qn != "." { + root = root.with(qn) + } + } + } for _, w := range widgets { - walk(w, dataContext{}, "") + walk(w, root, "") } return errs } diff --git a/mdl/executor/validate_datasource_params_pre101_test.go b/mdl/executor/validate_datasource_params_pre101_test.go new file mode 100644 index 0000000000..5b67834af6 --- /dev/null +++ b/mdl/executor/validate_datasource_params_pre101_test.go @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// ako/mxcli#1206: before Mendix 10.1 a page's or snippet's own parameter +// supplies an argument the way an enclosing data container does. Measured with +// one script (a snippet and a page each with an $Order parameter; a nanoflow +// data source and a nanoflow button action, both without an argument), one +// project per version: +// +// 9.24.30, 10.0.0 → 0 errors +// 10.1.0, 10.2.0, 10.3.1, 10.6.25, 10.12.19, +// 10.24.24, 11.12.6 → CE1571 on all three +// +// check applied the 10.1+ rule to every project, so a 9.x / 10.0 page that +// builds was refused. +func TestFlowArgs_ParameterSuppliesTheArgumentBefore10_1(t *testing.T) { + widgets := []*ast.WidgetV3{dsWidget("dgParamScope", "DS.DS_Rows")} + params := []ast.PageParameter{ + {Name: "Ctx", EntityType: ast.QualifiedName{Module: "DS147", Name: "ReportContext"}}, + } + if errs := validateFlowArgumentsScoped(params, widgets, f147Sigs(), sameName, true); len(errs) != 0 { + t.Errorf("before 10.1 a parameter of the right type supplies the argument (0 errors measured); got %v", errs) + } + // Control: from 10.1 the same page is CE1571, as before. + if errs := validateFlowArgumentsScoped(params, widgets, f147Sigs(), sameName, false); len(errs) != 1 { + t.Errorf("from 10.1 the parameter alone does not supply it (CE1571 measured); got %v", errs) + } + // A parameter of ANOTHER type supplies nothing in either version. + other := []ast.PageParameter{{Name: "Row", EntityType: ast.QualifiedName{Module: "DS147", Name: "Row"}}} + if errs := validateFlowArgumentsScoped(other, widgets, f147Sigs(), sameName, true); len(errs) != 1 || + !strings.Contains(errs[0], "CE1571") { + t.Errorf("a parameter of a different entity must not supply it; got %v", errs) + } +} + +func TestParamsSupplyFlowArguments(t *testing.T) { + at := func(maj, min int) func(int, int) bool { + return func(a, b int) bool { return maj > a || (maj == a && min >= b) } + } + for _, c := range []struct { + name string + atLeast func(int, int) bool + want bool + }{ + {"9.24", at(9, 24), true}, {"10.0", at(10, 0), true}, {"10.1", at(10, 1), false}, + {"11.12", at(11, 12), false}, {"unknown", nil, false}, + } { + if got := paramsSupplyFlowArguments(c.atLeast); got != c.want { + t.Errorf("%s: got %v, want %v", c.name, got, c.want) + } + } +} diff --git a/mdl/executor/validate_dynamictext_content_1865_test.go b/mdl/executor/validate_dynamictext_content_1865_test.go new file mode 100644 index 0000000000..5c9f98e90b --- /dev/null +++ b/mdl/executor/validate_dynamictext_content_1865_test.go @@ -0,0 +1,55 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "os" + "path/filepath" + "testing" +) + +// ako/mxcli#1865 (mendixlabs/mxcli#1707): `mxcli check -p` on 0.28.0 warned +// +// MDL-WIDGET10 … property `content` is hidden when … — the value will be ignored +// +// on `dynamictext x (Content: '…')`, also inside a Data grid 2 column, although +// exec writes a Forms$DynamicText whose content renders and passes mx check. +// Without a project there was no warning — the visibility rule came from the +// project. +// +// `dynamictext` is a keyword the page builder builds with its own switch, so no +// pluggable definition may answer for it. On 0.28.0 one could: a +// `.mxcli/widgets/dynamictext.def.json` that `widget init` generated for a +// package widget whose id ends in `dynamictext` took over the keyword in the +// registry, and check validated the built-in against that widget's editorConfig +// rules. mendixlabs/mxcli#1463 closed both doors (the loader refuses a +// builder-owned name; lookupWidgetDef never resolves one), so current main does +// not reproduce. This pins it for `dynamictext` specifically, top-level and in a +// column, since the #1463 test only covers `image` and `radiobuttons`. +const shadowingDynamicTextDef = `{"widgetId":"com.example.widget.web.dynamictext.DynamicText","mdlName":"DYNAMICTEXT","widgetKind":"pluggable","templateFile":"dynamictext.json","defaultEditable":"Always", + "propertyMappings":[{"propertyKey":"mode","value":"auto","operation":"primitive"},{"propertyKey":"content","source":"Content","operation":"primitive"}], + "propertyVisibility":[{"propertyKey":"content","hiddenWhen":{"propertyKey":"mode","operator":"eq","value":"auto"}}]}` + +func TestCheck1865_DynamicTextContentIsNotHidden(t *testing.T) { + dir := t.TempDir() + widgets := filepath.Join(dir, ".mxcli", "widgets") + if err := os.MkdirAll(widgets, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(widgets, "dynamictext.def.json"), []byte(shadowingDynamicTextDef), 0o644); err != nil { + t.Fatal(err) + } + projectPath := filepath.Join(dir, "App.mpr") + + src := `create page M.P (Title: 'P', Layout: Atlas_Core.Atlas_Default) { + dynamictext top (Content: 'Hello') + datagrid dg (DataSource: database from M.E) { + column (Attribute: Name, Caption: 'Name') { dynamictext inCol (Content: 'plain') } + } +};` + for _, v := range ValidateWidgetProperties(parseMDL(t, src), projectPath) { + if v.RuleID == "MDL-WIDGET10" { + t.Errorf("false hidden-property warning on a built-in dynamictext: %s", v.Message) + } + } +} diff --git a/mdl/executor/validate_filter_attrchoice_1684_test.go b/mdl/executor/validate_filter_attrchoice_1684_test.go new file mode 100644 index 0000000000..2724b65dc5 --- /dev/null +++ b/mdl/executor/validate_filter_attrchoice_1684_test.go @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// ako/mxcli#1684: `mxcli check -p` warned +// +// MDL-WIDGET10 … property `attributes` is hidden when `attrChoice` is "auto" +// +// on every text/number/date filter that names attributes, e.g. +// `textfilter tfName (Attributes: [Name])`. False: the builder's +// SetAttributeObjects writes attrChoice = "linked" whenever it fills the list. +// The definitions declared attrChoice as the constant "auto", so the value map +// the check reads disagreed with what exec stores. The drop-down filter had the +// same shape and was fixed under mendixlabs/mxcli#1519 by an `attribute` mode; +// the three sibling filters now carry the same mode. +// +// The warning only appears with a project, because the visibility rule is lifted +// from the installed DataWidgets .mpk (whose editorConfig hides `attributes` +// under "auto"). The rule is supplied directly here, as the #1519 test advises. + +// filterDefWithAttributesRule returns a copy of a real filter definition whose +// visibility rules hide `attributes` when attrChoice equals hideWhen. +func filterDefWithAttributesRule(t *testing.T, mdlName, hideWhen string) (*WidgetRegistry, *WidgetDefinition) { + t.Helper() + reg, err := NewWidgetRegistry() + if err != nil { + t.Fatalf("registry: %v", err) + } + real, ok := reg.Get(mdlName) + if !ok { + t.Fatalf("no %s definition", mdlName) + } + def := *real + def.PropertyVisibility = []types.WidgetVisibilityRule{{ + PropertyKey: "attributes", + HiddenWhen: &types.WidgetVisibilityCondition{PropertyKey: "attrChoice", Operator: "eq", Value: hideWhen}, + }} + return &WidgetRegistry{byMDLName: map[string]*WidgetDefinition{mdlName: &def}}, &def +} + +func TestCheck1684_FilterAttributesAreNotHidden(t *testing.T) { + for _, kw := range []string{"textfilter", "numberfilter", "datefilter"} { + t.Run(kw, func(t *testing.T) { + reg, def := filterDefWithAttributesRule(t, strings.ToUpper(kw), "auto") + w := parseSnippetWidget(t, kw+` f (Attributes: [Name])`) + + values, _ := widgetValueMap(w, def) + if got := values["attrchoice"]; got != "linked" { + t.Errorf("attrChoice = %q, want linked (what SetAttributeObjects stores)", got) + } + if v := validateWidgetVisibility(w, reg, "page P"); len(v) != 0 { + t.Errorf("got %d violation(s) %+v, want none — the builder stores attrChoice linked", len(v), v) + } + }) + } +} + +// A bare filter still builds in the column-bound "auto" choice. +func TestWidgetValueMap1684_BareFilterStaysAuto(t *testing.T) { + _, def := filterDefWithAttributesRule(t, "TEXTFILTER", "auto") + values, _ := widgetValueMap(parseSnippetWidget(t, `textfilter f`), def) + if got := values["attrchoice"]; got != "auto" { + t.Errorf("attrChoice = %q, want auto", got) + } +} + +// Control: the rule still fires for a property really hidden under the +// configuration the builder writes. +func TestCheck1684_ReallyHiddenAttributesStillWarn(t *testing.T) { + reg, _ := filterDefWithAttributesRule(t, "TEXTFILTER", "linked") + w := parseSnippetWidget(t, `textfilter f (Attributes: [Name])`) + v := validateWidgetVisibility(w, reg, "page P") + if len(v) != 1 || v[0].RuleID != "MDL-WIDGET10" { + t.Fatalf("got %d violation(s) %+v, want 1 MDL-WIDGET10", len(v), v) + } +} diff --git a/mdl/executor/validate_flow_verdicts.go b/mdl/executor/validate_flow_verdicts.go index 8d70504bce..cf235e5e6c 100644 --- a/mdl/executor/validate_flow_verdicts.go +++ b/mdl/executor/validate_flow_verdicts.go @@ -173,6 +173,17 @@ func checkFlowVerdicts(ctx *ExecContext, prog *ast.Program) FlowVerdictReport { case ruled: vs, dep, apply := flowModifyViolations(ctx, d) r.Violations = append(r.Violations, vs...) + // A rebuild exec runs instead of the splice (mdl 0) would delete + // what MDL cannot express; exec refuses it when it gets there, + // so refuse the script now, before anything is written. + for _, v := range vs { + if v.RuleID == FlowRebuildRule && v.Severity != linter.SeverityError { + if types := storedFlowUnsupportedActions(ctx, d.kind(), d.name.String()); len(types) > 0 { + r.Violations = append(r.Violations, unsupportedRebuildViolation(d.kind(), d.name, types)) + } + break + } + } if dep != "" { unpredicted(d.nanoflow, d.name, dep) } diff --git a/mdl/executor/validate_member_refs.go b/mdl/executor/validate_member_refs.go index 58f60edfad..e68970e2d5 100644 --- a/mdl/executor/validate_member_refs.go +++ b/mdl/executor/validate_member_refs.go @@ -491,6 +491,17 @@ func checkMembers(ctx *ExecContext, sc *scriptContext, authored map[string]bool, "Retrieve the associated object and change that instead (`retrieve $obj from $var/%s;` then `change $obj (%s = ...)`); "+ "mxbuild rejects this form with CE1613", flowQN, name, entityQN, name[:dot], name[dot+1:])) + } else if strings.Count(name, ".") == 1 && moduleLacksAssociation(ctx, sc, name[:dot], name[dot+1:]) { + // `Module.Assoc` over a real module is an association, and only + // one: exec's builder refuses a name the module does not have, + // so check must too, or the two disagree (ako/mxcli#1810 — on a + // 9.x project, System.WorkflowUserTask_Assignees passed check + // and stopped exec; on a version-blind System model it wrote + // and failed the build with CE1613). + out = append(out, fmt.Sprintf( + "%s: %q on %s is not an association the project has (and a one-qualifier name cannot be "+ + "an attribute) — mxbuild reports CE1613 \"no longer exists\"", + flowQN, name, entityQN)) } continue } @@ -625,3 +636,40 @@ func nearMembers(ctx *ExecContext, entityQN string) string { } return nearMembersFor(b, entityQN) } + +// moduleLacksAssociation reports whether module provably has no association +// named assoc: neither the project's domain model of that module nor the +// script declares one. A module the project cannot be read for answers false +// (could-not-establish stays silent). +func moduleLacksAssociation(ctx *ExecContext, sc *scriptContext, module, assoc string) bool { + if sc != nil { + if qn, ok := sc.associations[assoc]; ok && strings.EqualFold(qn, module+"."+assoc) { + return false + } + if sc.modules[module] { + return false // created by the script: its domain model is not readable yet + } + } + if ctx == nil || ctx.Backend == nil || !ctx.Connected() { + return false + } + mod, err := ctx.Backend.GetModuleByName(module) + if err != nil || mod == nil { + return false + } + dm, err := ctx.Backend.GetDomainModel(mod.ID) + if err != nil || dm == nil { + return false + } + for _, a := range dm.Associations { + if a != nil && a.Name == assoc { + return false + } + } + for _, a := range dm.CrossAssociations { + if a != nil && a.Name == assoc { + return false + } + } + return true +} diff --git a/mdl/executor/validate_member_refs_test.go b/mdl/executor/validate_member_refs_test.go index e4bea209ee..9ad741082f 100644 --- a/mdl/executor/validate_member_refs_test.go +++ b/mdl/executor/validate_member_refs_test.go @@ -445,3 +445,23 @@ func TestValidateMemberReferences_ScriptCreatedModuleIsAModule(t *testing.T) { t.Errorf("control: got %d messages without the script's module, want 1: %v", len(msgs), msgs) } } + +// ako/mxcli#1810: `Module.Assoc` over a real module names an association, and +// exec's builder refuses one the module does not have. check passed it — on a +// 9.x project, System.WorkflowUserTask_Assignees (10.0's name) passed check and +// then stopped exec. The existing associations, same-module and cross-module, +// are the control. +func TestValidateMemberReferences_RefusesAnUnknownQualifiedAssociation(t *testing.T) { + ctx := memberFixture(t) + msgs := validateMemberReferences(ctx, + flowProgram(orderParam(), changeStmt("Order", "Shop.Order_Nope")), newScriptContext()) + if len(msgs) != 1 || !strings.Contains(msgs[0], "Shop.Order_Nope") || !strings.Contains(msgs[0], "not an association") { + t.Fatalf("want one refusal naming Shop.Order_Nope, got %v", msgs) + } + for _, ok := range []string{"Shop.Order_Customer", "Shop.Order_Bin"} { + if msgs := validateMemberReferences(ctx, + flowProgram(orderParam(), changeStmt("Order", ok)), newScriptContext()); len(msgs) != 0 { + t.Errorf("%s exists and must pass: %v", ok, msgs) + } + } +} diff --git a/mdl/executor/validate_message_definition_members.go b/mdl/executor/validate_message_definition_members.go new file mode 100644 index 0000000000..95655f5beb --- /dev/null +++ b/mdl/executor/validate_message_definition_members.go @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" +) + +// messageDefinitionMemberErrors reports the attribute members exec would refuse +// with " has no attribute …" (buildMessageMember). check --references +// used to pass them, so a describe that printed a renamed attribute's old name +// was "Check passed!" and then failed on exec (mendixlabs/mxcli#1733). +// +// It uses the DataGrid column judge, which answers only "provably not an +// attribute": an entity the script alters, imports or declares without its +// members is left alone, so a script that adds the attribute first still passes. +func messageDefinitionMemberErrors(ctx *ExecContext, sc *scriptContext, label string, defs ...*ast.MessageDefinitionDef) error { + missing := newColumnAttrJudge(ctx, sc) + var errs []string + var walk func(holder, where string, members []*ast.MessageMemberDef) + walk = func(holder, where string, members []*ast.MessageMemberDef) { + for _, m := range members { + if m == nil { + continue + } + if m.Association.Name != "" { + walk(m.Entity.String(), where, m.Members) + continue + } + if missing(holder, m.Attribute) { + errs = append(errs, fmt.Sprintf("%s: %s has no attribute %q%s", + where, holder, m.Attribute, availableAttributes(ctx, holder))) + } + } + } + for _, d := range defs { + if d == nil { + continue + } + where := label + if d.Name != "" { + where = label + "." + d.Name + } + walk(d.Entity.String(), "message definition "+where, d.Members) + } + if len(errs) == 0 { + return nil + } + return mdlerrors.NewValidation(strings.Join(errs, "\n - ")) +} diff --git a/mdl/executor/validate_message_definition_members_test.go b/mdl/executor/validate_message_definition_members_test.go new file mode 100644 index 0000000000..69524674b9 --- /dev/null +++ b/mdl/executor/validate_message_definition_members_test.go @@ -0,0 +1,98 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +// mendixlabs/mxcli#1733 (ako/mxcli#1935): `check --references` passed a message +// definition member naming an attribute the entity no longer has, and exec then +// refused it — `MyFirstModule.Booking has no attribute "Reference"; available: +// Code, CustomerReference`. check is meant to be the stronger gate. +func messageMemberCtx(t *testing.T) *ExecContext { + t.Helper() + mod := mkModule("MyFirstModule") + booking := mkEntity(mod.ID, "Booking") + booking.Attributes = []*domainmodel.Attribute{{Name: "CustomerReference"}, {Name: "Code"}} + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListFoldersFunc: func() ([]*types.FolderInfo, error) { return nil, nil }, + ListDomainModelsFunc: func() ([]*domainmodel.DomainModel, error) { + return []*domainmodel.DomainModel{mkDomainModel(mod.ID, booking)}, nil + }, + GetModuleByNameFunc: func(string) (*model.Module, error) { return mod, nil }, + GetDomainModelFunc: func(model.ID) (*domainmodel.DomainModel, error) { + return mkDomainModel(mod.ID, booking), nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy(mod))) + return ctx +} + +func checkMessageMembers(t *testing.T, src string) string { + t.Helper() + ctx := messageMemberCtx(t) + prog := parseMDL(t, src) + sc := newScriptContext() + sc.collectDefinitions(prog) + var out []string + for _, stmt := range prog.Statements { + if err := validateWithContext(ctx, stmt, sc); err != nil { + out = append(out, err.Error()) + } + } + return strings.Join(out, "\n") +} + +func TestMessageDefinitionUnknownAttributeIsReportedByCheck(t *testing.T) { + got := checkMessageMembers(t, `create or modify message definition collection MyFirstModule.MD_Booking +{ + definition BookingMessage for MyFirstModule.Booking { + Reference, + Code + } +};`) + for _, want := range []string{`"Reference"`, "MyFirstModule.Booking", "CustomerReference"} { + if !strings.Contains(got, want) { + t.Errorf("check should refuse the unknown member and name %s; got:\n%s", want, got) + } + } + if strings.Contains(got, `"Code"`) { + t.Errorf("Code is an attribute and must not be reported:\n%s", got) + } +} + +func TestMessageDefinitionKnownAttributesPassCheck(t *testing.T) { + for name, src := range map[string]string{ + "renamed attribute with stored exposed name": `create or modify message definition collection MyFirstModule.MD_Booking +{ + definition BookingMessage for MyFirstModule.Booking { + CustomerReference as 'Reference', + Code + } +};`, + // the script adds the attribute itself: exec runs the alter first + "attribute added earlier in the script": `alter entity MyFirstModule.Booking add attribute Extra: String(10); +create message definition collection MyFirstModule.MD_B2 +{ + definition B for MyFirstModule.Booking { + Extra + } +};`, + "11.15 document form": `create message definition MyFirstModule.MD_Doc for MyFirstModule.Booking { + Code +};`, + } { + if got := checkMessageMembers(t, src); got != "" { + t.Errorf("%s: check refused a valid member:\n%s", name, got) + } + } +} diff --git a/mdl/executor/validate_microflow.go b/mdl/executor/validate_microflow.go index dd01bec70a..07c7009778 100644 --- a/mdl/executor/validate_microflow.go +++ b/mdl/executor/validate_microflow.go @@ -1264,6 +1264,11 @@ func stmtActivityName(stmt ast.MicroflowStatement) string { return "delete" case *ast.MfCommitStmt: return "commit" + case *ast.ExternalObjectStmt: + if s := stmt.(*ast.ExternalObjectStmt); s.Delete { + return "delete external object" + } + return "send external object" case *ast.RetrieveStmt: return "retrieve" case *ast.CallMicroflowStmt: @@ -1601,6 +1606,8 @@ func referencedVars(stmt ast.MicroflowStatement) []string { refs = append(refs, s.Variable) case *ast.DeleteObjectStmt: refs = append(refs, s.Variable) + case *ast.ExternalObjectStmt: + refs = append(refs, s.Variable) case *ast.AddToListStmt: if s.Value != nil { refs = append(refs, exprVarRefs(s.Value)...) diff --git a/mdl/executor/validate_microflow_error_handling.go b/mdl/executor/validate_microflow_error_handling.go index d67ffa9798..590c8d95df 100644 --- a/mdl/executor/validate_microflow_error_handling.go +++ b/mdl/executor/validate_microflow_error_handling.go @@ -58,6 +58,11 @@ var continueUnsupportedOn = map[string]string{ "close page": "Close page activity", "show message": "Show message activity", "validation feedback": "Validation feedback activity", + // Measured on 11.12.6, one microflow per project: Continue is CE6035 on + // both; the default, and a custom handler with or without rollback, are + // accepted (mendixlabs/mxcli#1735). + "send external object": "Send external object activity", + "delete external object": "Delete external object activity", // mendixlabs/mxcli#175, measured on 11.14.0 in ako/TestApp: continue is // CE6035 on a Call workflow activity; no clause, rollback and both custom // handlers build. @@ -152,6 +157,11 @@ func continueUnsupportedStatement(stmt ast.MicroflowStatement) (keyword, activit keyword = "cast" case *ast.ShowHomePageStmt: keyword = "show home page" + case *ast.ExternalObjectStmt: + keyword = "send external object" + if s.Delete { + keyword = "delete external object" + } default: // DeclareStmt and MfSetStmt are deliberately absent: create-variable and // change-variable accept Continue on 11.14.0. So is RollbackStmt, on diff --git a/mdl/executor/validate_microflow_loop_scope.go b/mdl/executor/validate_microflow_loop_scope.go index 12b981ace0..75875ba24c 100644 --- a/mdl/executor/validate_microflow_loop_scope.go +++ b/mdl/executor/validate_microflow_loop_scope.go @@ -286,6 +286,8 @@ func loopRefVars(stmt ast.MicroflowStatement) []string { add(s.Variable) case *ast.RollbackStmt: add(s.Variable) + case *ast.ExternalObjectStmt: + add(s.Variable) case *ast.RetrieveStmt: add(s.StartVariable) addExpr(s.Where) diff --git a/mdl/executor/validate_page_context.go b/mdl/executor/validate_page_context.go index 2af20091e5..691fc0f6d2 100644 --- a/mdl/executor/validate_page_context.go +++ b/mdl/executor/validate_page_context.go @@ -30,7 +30,7 @@ func validatePageContextTree(ctx *ExecContext, params []ast.PageParameter, widge // Walk the widget tree with context tracking var errors []string errors = append(errors, checkDuplicateWidgetNames(widgets, pageContextWidgetRegistry(ctx))...) - walkWidgetsWithContext(widgets, paramNames, widgetNames, false, &errors) + walkWidgetsWithContext(widgets, paramNames, widgetNames, false, pageContextWidgetRegistry(ctx), nil, &errors) return errors } @@ -173,7 +173,16 @@ func collectWidgetNames(widgets []*ast.WidgetV3, names map[string]bool) { // walkWidgetsWithContext validates each widget's DataSource and attribute bindings, // tracking whether the current position is inside a data container (DataView, // DataGrid, ListView, etc.) that provides entity context. -func walkWidgetsWithContext(widgets []*ast.WidgetV3, paramNames map[string]bool, widgetNames map[string]bool, hasEntityContext bool, errors *[]string) { +// +// itemKinds are the container keywords the enclosing widget's definition +// declares (unstoredContainerKinds): a child of one of those types is an +// object-list ITEM or a slot block, not a widget, and its properties are the +// item's own. An item property may be named `attribute` without being an +// attribute binding — AccessibilityHelper's attributesList.attribute is the +// HTML attribute's name, a string — and reading it as one refused the +// describe output of every such widget (ako/mxcli#904). registry may be nil; +// then no item resolves and the rule behaves as before. +func walkWidgetsWithContext(widgets []*ast.WidgetV3, paramNames map[string]bool, widgetNames map[string]bool, hasEntityContext bool, registry *WidgetRegistry, itemKinds map[string]bool, errors *[]string) { for _, w := range widgets { ds := w.GetDataSource() childHasContext := hasEntityContext @@ -212,7 +221,7 @@ func walkWidgetsWithContext(widgets []*ast.WidgetV3, paramNames map[string]bool, } // Validate attribute binding: needs entity context - if attr := w.GetAttribute(); attr != "" { + if attr := w.GetAttribute(); attr != "" && !itemKinds[strings.ToLower(w.Type)] { // `$Param.Attr` names its object; validateNamedObjectBindings // judges the name. if _, _, named := namedObjectBinding(attr); !hasEntityContext && !named { @@ -222,7 +231,7 @@ func walkWidgetsWithContext(widgets []*ast.WidgetV3, paramNames map[string]bool, } // Recurse into children - walkWidgetsWithContext(w.Children, paramNames, widgetNames, childHasContext, errors) + walkWidgetsWithContext(w.Children, paramNames, widgetNames, childHasContext, registry, unstoredContainerKinds(registry, w), errors) } } diff --git a/mdl/executor/validate_page_context_accessibilityhelper_904_test.go b/mdl/executor/validate_page_context_accessibilityhelper_904_test.go new file mode 100644 index 0000000000..cbb7304c30 --- /dev/null +++ b/mdl/executor/validate_page_context_accessibilityhelper_904_test.go @@ -0,0 +1,105 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// ako/mxcli#904: DESCRIBE of an AccessibilityHelper did not re-execute. +// +// Two defects were reported. (1) describe emitted `content { … }` for the +// widget's child slot, which exec refused with MDL-WIDGET26 "`content` is not a +// container of `accessibilityhelper` — it declares: attr, template". That half +// was fixed by mendixlabs/mxcli#1400 (describe derives the slot keyword with the +// definition's function); TestAccessibilityHelper_SlotKeywordIsTemplate pins it +// for this widget. (2) The widget's attributesList item has a property literally +// named `attribute` (the HTML attribute's name, a string), and `check +// --references` — which exec runs first — read it as an attribute BINDING: +// +// widget '': Attribute 'aria-label' is bound but there is no enclosing data +// container providing entity context +// +// so the describe output of any AccessibilityHelper with an attribute refused to +// execute. Measured on 11.12.1: exec --no-check of the same script writes the +// item and describes it back unchanged. + +const accessibilityHelperID = "com.mendix.widget.web.accessibilityhelper.AccessibilityHelper" + +// accessibilityHelperRegistry is the shape of the widget's .def.json: an +// attributesList object list (container `attr`) and a `content` child slot +// (container `template`). In memory, as chartRegistry is, because the embedded +// definitions declare no object list. +func accessibilityHelperRegistry() *WidgetRegistry { + def := &WidgetDefinition{ + WidgetID: accessibilityHelperID, + MDLName: "accessibilityhelper", + ObjectLists: []ObjectListMapping{ + {PropertyKey: "attributesList", MDLContainer: deriveObjectListKeyword("attributesList")}, + }, + ChildSlots: []ChildSlotMapping{ + {PropertyKey: "content", MDLContainer: mdlContainerForWidgetSlot(accessibilityHelperID, "content")}, + }, + } + return &WidgetRegistry{ + byMDLName: map[string]*WidgetDefinition{"ACCESSIBILITYHELPER": def}, + byWidgetID: map[string]*WidgetDefinition{def.WidgetID: def}, + } +} + +// describedAccessibilityHelper is what `describe page` prints for one, measured. +const describedAccessibilityHelper = `create or modify page Probe904.P (Title: 'P', Layout: Atlas_Core.Atlas_Default) { + accessibilityhelper acc1 (targetSelector: '.mx-name-acc1') { + attr (attribute: 'aria-label', valueSourceType: 'text', valueText: 'Hello', attributeCondition: 'true') + template { + dynamictext accContent (Content: 'Accessible content area') + } + } +};` + +func pageWidgetsOf(t *testing.T, src string) []*ast.WidgetV3 { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + return prog.Statements[0].(*ast.CreatePageStmtV3).Widgets +} + +func ctxWithRegistry(r *WidgetRegistry) *ExecContext { + return &ExecContext{widgetRegistry: r, widgetRegistryLoaded: true} +} + +func TestAccessibilityHelper_SlotKeywordIsTemplate(t *testing.T) { + if kw := mdlContainerForWidgetSlot(accessibilityHelperID, "content"); kw != "TEMPLATE" { + t.Errorf("content slot keyword = %q, want TEMPLATE (what the definition declares)", kw) + } + if kw := types.WidgetSlotKeyword(accessibilityHelperID, "content"); kw != "TEMPLATE" { + t.Errorf("describe's slot keyword = %q, want TEMPLATE", kw) + } +} + +func TestAccessibilityHelper_ItemAttributeIsNotABinding(t *testing.T) { + errs := validatePageContextTree(ctxWithRegistry(accessibilityHelperRegistry()), nil, + pageWidgetsOf(t, describedAccessibilityHelper)) + if len(errs) != 0 { + t.Errorf("describe output of an AccessibilityHelper refused: %v", errs) + } +} + +// Control: a real attribute binding outside every data container is still +// reported, beside the helper — the rule has not gone quiet. +func TestAccessibilityHelper_RealUnboundAttributeStillReported(t *testing.T) { + src := strings.Replace(describedAccessibilityHelper, + " accessibilityhelper acc1", + " textbox txtLoose (Attribute: Name)\n accessibilityhelper acc1", 1) + errs := validatePageContextTree(ctxWithRegistry(accessibilityHelperRegistry()), nil, pageWidgetsOf(t, src)) + if len(errs) != 1 || !strings.Contains(errs[0], "txtLoose") { + t.Errorf("got %v, want exactly the unbound textbox txtLoose", errs) + } +} diff --git a/mdl/executor/validate_program.go b/mdl/executor/validate_program.go index 6a45f371c1..75b72ffef5 100644 --- a/mdl/executor/validate_program.go +++ b/mdl/executor/validate_program.go @@ -259,7 +259,9 @@ func ValidateProgram(prog *ast.Program, projectPath string) []linter.Violation { // assigned. TARGETING decides who may SEE a task; it does not assign it, and // completing an unassigned one fails only at runtime, with the button // appearing to do nothing. - violations = append(violations, ValidateTaskClaims(prog)...) + violations = append(violations, ValidateTaskClaimsIn(prog, func() func(major, minor int) bool { + return projectVersionAtLeast(project()) + })...) // Warn (MDL-WF20) when a user task's page has no button that completes one // of its outcomes: the instance waits at the task and nothing reports it. diff --git a/mdl/executor/validate_queued_calls.go b/mdl/executor/validate_queued_calls.go index e51e8638da..a45032f033 100644 --- a/mdl/executor/validate_queued_calls.go +++ b/mdl/executor/validate_queued_calls.go @@ -33,9 +33,11 @@ import ( // - A stored queue the new script does not name. Restating some and dropping // others is far more likely a mistake than an intent. // - A stored QueueSettings carrying a Retry (Queues$QueueFixedRetry / -// Queues$QueueExponentialRetry). MDL has no syntax for a retry policy, so -// the rewrite cannot preserve one, and re-running an "unchanged" script -// would quietly reset it. +// Queues$QueueExponentialRetry) that the script does not restate with a +// `retry …` clause on a call to that queue: the rebuild would write the +// call without it, so re-running an "unchanged" script would quietly reset +// it. A script that states a retry for that queue — the stored one, or a +// new one — is saying what it wants, and goes through (mendixlabs/mxcli#1729). func checkNoQueuedCalls(ctx *ExecContext, microflowID model.ID, qualifiedName string, stmt *ast.CreateMicroflowStmt) error { raw, err := ctx.Backend.GetRawUnit(microflowID) if err != nil { @@ -48,13 +50,8 @@ func checkNoQueuedCalls(ctx *ExecContext, microflowID model.ID, qualifiedName st return nil } - if retries := storedQueueRetries(raw); len(retries) > 0 { - sort.Strings(retries) - return mdlerrors.NewUnsupported(fmt.Sprintf( - "microflow %s has %d queued call(s) with a retry policy (%s), and MDL cannot express one — "+ - "rewriting the microflow would reset it.\n"+ - " Change the microflow in Studio Pro, or remove the retry from the call first.", - qualifiedName, len(retries), strings.Join(retries, ", "))) + if err := checkNoDroppedQueueRetries(ctx, microflowID, qualifiedName, stmt, raw); err != nil { + return err } restated := authoredQueueTargets(stmt) @@ -76,6 +73,49 @@ func checkNoQueuedCalls(ctx *ExecContext, microflowID model.ID, qualifiedName st qualifiedName, len(lost), strings.Join(lost, ", "), lost[0])) } +// checkNoDroppedQueueRetries refuses a rewrite that still binds a call to a +// queue whose stored call carries a retry policy, but states no retry for that +// queue: the call would be written without it (mendixlabs/mxcli#1729). +// +// It runs on BOTH paths of create or modify. The splice path (an in-place edit +// of the stored flow) does not reach checkNoQueuedCalls, and once the retry is +// part of the model a call restated without one is an edit — so without this +// the splice would quietly reset the retry. A script that no longer binds any +// call to the queue has removed those calls on purpose, and is let through +// here (the rebuild path refuses the lost binding separately). +// +// raw is the stored unit when the caller already read it, or nil. +func checkNoDroppedQueueRetries(ctx *ExecContext, microflowID model.ID, qualifiedName string, stmt *ast.CreateMicroflowStmt, raw map[string]any) error { + if raw == nil { + var err error + if raw, err = ctx.Backend.GetRawUnit(microflowID); err != nil { + return nil + } + } + stored := storedQueueRetries(raw) + if len(stored) == 0 { + return nil + } + queues := authoredQueueTargets(stmt) + retries := authoredQueueRetries(stmt) + var dropped []string + for _, q := range stored { + if queues[strings.ToLower(q)] && !retries[strings.ToLower(q)] { + dropped = append(dropped, q) + } + } + if len(dropped) == 0 { + return nil + } + sort.Strings(dropped) + return mdlerrors.NewUnsupported(fmt.Sprintf( + "microflow %s has %d queued call(s) with a retry policy (%s) that this script does not restate, "+ + "and rewriting it would reset the retry.\n"+ + " Add the retry to the call — `IN QUEUE %s RETRY FIXED (Retries: n, Interval: n, IntervalType: Minutes)` "+ + "or `RETRY EXPONENTIAL (…)`; `describe microflow %s` prints the stored one.", + qualifiedName, len(dropped), strings.Join(dropped, ", "), dropped[0], qualifiedName)) +} + // authoredQueueTargets returns the lower-cased queue names the incoming // statement binds calls to, found by walking the whole statement tree — call // statements nest inside IF / LOOP / error handlers, and a hand-written switch @@ -85,11 +125,13 @@ func authoredQueueTargets(stmt *ast.CreateMicroflowStmt) map[string]bool { if stmt == nil { return out } - collectAuthoredQueues(reflect.ValueOf(stmt), out, map[uintptr]bool{}) + collectQueueBindings(reflect.ValueOf(stmt), out, map[string]bool{}, map[uintptr]bool{}) return out } -func collectAuthoredQueues(v reflect.Value, out map[string]bool, seen map[uintptr]bool) { +// collectQueueBindings records each call's queue in queues, and in retries too +// when the call states a retry policy. +func collectQueueBindings(v reflect.Value, queues, retries map[string]bool, seen map[uintptr]bool) { switch v.Kind() { case reflect.Ptr, reflect.Interface: if v.IsNil() { @@ -102,26 +144,43 @@ func collectAuthoredQueues(v reflect.Value, out map[string]bool, seen map[uintpt seen[v.Pointer()] = true switch s := v.Interface().(type) { case *ast.CallMicroflowStmt: - addQueueName(s.Queue, out) + addQueueName(s.Queue, queues) + if s.QueueRetry != nil { + addQueueName(s.Queue, retries) + } case *ast.CallJavaActionStmt: - addQueueName(s.Queue, out) + addQueueName(s.Queue, queues) + if s.QueueRetry != nil { + addQueueName(s.Queue, retries) + } } } - collectAuthoredQueues(v.Elem(), out, seen) + collectQueueBindings(v.Elem(), queues, retries, seen) case reflect.Slice, reflect.Array: for i := 0; i < v.Len(); i++ { - collectAuthoredQueues(v.Index(i), out, seen) + collectQueueBindings(v.Index(i), queues, retries, seen) } case reflect.Struct: for i := 0; i < v.NumField(); i++ { if v.Type().Field(i).PkgPath != "" { continue // unexported } - collectAuthoredQueues(v.Field(i), out, seen) + collectQueueBindings(v.Field(i), queues, retries, seen) } } } +// authoredQueueRetries returns the lower-cased queue names the statement binds +// at least one call to WITH a retry clause, found by the same whole-tree walk. +func authoredQueueRetries(stmt *ast.CreateMicroflowStmt) map[string]bool { + out := map[string]bool{} + if stmt == nil { + return out + } + collectQueueBindings(reflect.ValueOf(stmt), map[string]bool{}, out, map[uintptr]bool{}) + return out +} + func addQueueName(q *ast.QualifiedName, out map[string]bool) { if q == nil { return diff --git a/mdl/executor/validate_queued_calls_test.go b/mdl/executor/validate_queued_calls_test.go index 947e9550ab..f1053f1322 100644 --- a/mdl/executor/validate_queued_calls_test.go +++ b/mdl/executor/validate_queued_calls_test.go @@ -232,8 +232,8 @@ func TestAuthoredQueueTargets_FindsNestedCalls(t *testing.T) { } } -// A stored retry policy has no MDL spelling, so restating the queue is not -// enough — the rewrite would reset the retry. That must still refuse. +// Restating the queue without the retry is not enough — the rewrite would +// reset the retry. That must still refuse, and say how to restate it. func TestCheckNoQueuedCalls_RefusesStoredRetry(t *testing.T) { mb := &mock.MockBackend{ IsConnectedFunc: func() bool { return true }, @@ -321,3 +321,63 @@ func TestStoredQueueRetries_HandlesBsonArrays(t *testing.T) { t.Fatalf("storedQueueRetries = %v, want [Q.MyQueue]", got) } } + +// mendixlabs/mxcli#1729: a flow with a retry was locked for MDL. A script that +// restates a retry on that queue is saying what it wants, and goes through. +func TestCheckNoQueuedCalls_AllowsRestatedRetry(t *testing.T) { + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + GetRawUnitFunc: func(id model.ID) (map[string]any, error) { + return storedCall(map[string]any{ + "$Type": "Queues$QueueSettings", + "Queue": "Q.MyQueue", + "Retry": map[string]any{"$Type": "Queues$QueueFixedRetry", "Retries": 3}, + }, nil), nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb)) + stmt := &ast.CreateMicroflowStmt{ + Name: ast.QualifiedName{Module: "Q", Name: "ACT_Caller"}, + Body: []ast.MicroflowStatement{ + &ast.IfStmt{ThenBody: []ast.MicroflowStatement{ // nested: the walk must reach it + &ast.CallMicroflowStmt{ + MicroflowName: ast.QualifiedName{Module: "Q", Name: "Target"}, + Queue: &ast.QualifiedName{Module: "Q", Name: "MyQueue"}, + QueueRetry: &ast.QueueRetry{Kind: "Fixed", Retries: 3, Interval: 10, IntervalType: "Minutes"}, + }, + }}, + }, + } + if err := checkNoQueuedCalls(ctx, "mf-1", "Q.ACT_Caller", stmt); err != nil { + t.Fatalf("a script restating the retry must be allowed: %v", err) + } +} + +// The splice path calls checkNoDroppedQueueRetries directly: a call still bound +// to the queue without its retry is refused; a script that removed every call +// to that queue has said so, and is let through. +func TestCheckNoDroppedQueueRetries(t *testing.T) { + raw := storedCall(map[string]any{ + "$Type": "Queues$QueueSettings", + "Queue": "Q.MyQueue", + "Retry": map[string]any{"$Type": "Queues$QueueExponentialRetry", "Retries": 3}, + }, nil) + ctx, _ := newMockCtx(t, withBackend(&mock.MockBackend{IsConnectedFunc: func() bool { return true }})) + call := func(retry *ast.QueueRetry) *ast.CreateMicroflowStmt { + return &ast.CreateMicroflowStmt{Body: []ast.MicroflowStatement{&ast.CallMicroflowStmt{ + MicroflowName: ast.QualifiedName{Module: "Q", Name: "Target"}, + Queue: &ast.QualifiedName{Module: "Q", Name: "MyQueue"}, + QueueRetry: retry, + }}} + } + if err := checkNoDroppedQueueRetries(ctx, "mf", "Q.F", call(nil), raw); err == nil { + t.Error("a call restated without its stored retry must be refused") + } + if err := checkNoDroppedQueueRetries(ctx, "mf", "Q.F", + call(&ast.QueueRetry{Kind: "Fixed", Retries: 1, Interval: 1, IntervalType: "Hours"}), raw); err != nil { + t.Errorf("a changed, stated retry must be allowed: %v", err) + } + if err := checkNoDroppedQueueRetries(ctx, "mf", "Q.F", &ast.CreateMicroflowStmt{}, raw); err != nil { + t.Errorf("removing the queued call must be allowed: %v", err) + } +} diff --git a/mdl/executor/validate_task_claim.go b/mdl/executor/validate_task_claim.go index dba25ab3eb..fc9f20c6f4 100644 --- a/mdl/executor/validate_task_claim.go +++ b/mdl/executor/validate_task_claim.go @@ -43,14 +43,54 @@ import ( // unquoted, and with or without the System prefix, all of which parse. const assigneesAssociation = "workflowusertask_assignees" -// ValidateTaskClaims reports SET TASK OUTCOME statements whose task was not +// assigneeAssociation is the same association's name before Mendix 10.0: a +// Reference (one user) named WorkflowUserTask_Assignee, renamed to the +// ReferenceSet Assignees in 10.0 with its GUID kept (ako/mxcli#1810). A 9.x +// claim writes this one, so it claims just the same. +const assigneeAssociation = "workflowusertask_assignee" + +// isClaimAssociation reports whether a member name (unqualified) is the +// association a claim writes, under either version's name. +func isClaimAssociation(name string) bool { + return strings.EqualFold(name, assigneesAssociation) || strings.EqualFold(name, assigneeAssociation) +} + +// taskClaimSuggestion is MDL-WORKFLOW10's fix for the project's version. +// atLeast is the project version's IsAtLeast, nil when it is not known; then +// both spellings are given, each with its versions, rather than one that does +// not build on the other (CE1613 on 9.x for Assignees, ako/mxcli#1810). +func taskClaimSuggestion(task string, atLeast func(major, minor int) bool) string { + var claim string + switch { + case atLeast == nil: + claim = fmt.Sprintf(" change $%s (System.WorkflowUserTask_Assignees = [%%CurrentUser%%]); -- Mendix 10.0+\n"+ + " change $%s (System.WorkflowUserTask_Assignee = [%%CurrentUser%%]); -- Mendix 9.x\n", task, task) + case atLeast(10, 0): + claim = fmt.Sprintf(" change $%s (System.WorkflowUserTask_Assignees = [%%CurrentUser%%]);\n", task) + default: + claim = fmt.Sprintf(" change $%s (System.WorkflowUserTask_Assignee = [%%CurrentUser%%]);\n"+ + " (Mendix 9 names it WorkflowUserTask_Assignee, one user; Assignees is 10.0+ and CE1613 here.)\n", task) + } + return "Claim the task before setting the outcome:\n" + claim + + fmt.Sprintf(" commit $%s;\n", task) + + " TARGETING XPATH / TARGETING MICROFLOW decides who may SEE a task; it does not " + + "assign it. A microflow this one calls first, passing the task, counts when it claims " + + "it. Ignore this if the task is claimed elsewhere — in a nanoflow on the button, or " + + "earlier in the process." +} + +// ValidateTaskClaimsIn reports SET TASK OUTCOME statements whose task was not // assigned earlier in the same microflow — or in a microflow it calls first, // passing the task (ako/mxcli#943). A callee the script itself creates is read; // one it does not create cannot be, so a call that passes the task to it counts // as a possible claim and the outcome is not reported. With a project, // StoredTaskClaimViolations reads those callees too. -func ValidateTaskClaims(prog *ast.Program) []linter.Violation { - return validateTaskClaims(prog, nil) +// +// version returns the project version's IsAtLeast (nil when unknown), asked +// only when a warning is written, so the suggested claim is the one that +// builds on the project's version (ako/mxcli#1810). +func ValidateTaskClaimsIn(prog *ast.Program, version func() func(major, minor int) bool) []linter.Violation { + return validateTaskClaims(prog, nil, version) } // storedClaimSource reads a microflow the script does not create: whether it @@ -65,11 +105,11 @@ type storedClaimSource interface { // call is treated as a possible claim — the quiet direction. const maxClaimCallDepth = 8 -func validateTaskClaims(prog *ast.Program, stored storedClaimSource) []linter.Violation { +func validateTaskClaims(prog *ast.Program, stored storedClaimSource, version func() func(major, minor int) bool) []linter.Violation { if prog == nil { return nil } - r := &taskClaimResolver{script: map[string]*ast.CreateMicroflowStmt{}, stored: stored} + r := &taskClaimResolver{script: map[string]*ast.CreateMicroflowStmt{}, stored: stored, version: version} for _, stmt := range prog.Statements { if mf, ok := stmt.(*ast.CreateMicroflowStmt); ok { r.script[strings.ToLower(mf.Name.String())] = mf @@ -86,22 +126,24 @@ func validateTaskClaims(prog *ast.Program, stored storedClaimSource) []linter.Vi return out } -// StoredTaskClaimViolations is the MDL-WORKFLOW10 warnings ValidateTaskClaims +// StoredTaskClaimViolations is the MDL-WORKFLOW10 warnings ValidateTaskClaimsIn // stays quiet about because the claiming callee is not in the script: with the // project's microflows readable, a call to a stored microflow that does not // claim the task it is passed no longer hides the outcome. Only the warnings -// ValidateTaskClaims did not already report are returned, so the two can be +// ValidateTaskClaimsIn did not already report are returned, so the two can be // printed together. func StoredTaskClaimViolations(prog *ast.Program, b backend.FullBackend) []linter.Violation { if prog == nil || b == nil { return nil } base := map[string]bool{} - for _, v := range validateTaskClaims(prog, nil) { + for _, v := range validateTaskClaims(prog, nil, nil) { base[v.Location.DocumentName+"\x00"+v.Message] = true } var out []linter.Violation - for _, v := range validateTaskClaims(prog, &storedTaskClaims{b: b}) { + for _, v := range validateTaskClaims(prog, &storedTaskClaims{b: b}, func() func(major, minor int) bool { + return projectVersionAtLeast(b) + }) { if !base[v.Location.DocumentName+"\x00"+v.Message] { out = append(out, v) } @@ -114,6 +156,17 @@ func StoredTaskClaimViolations(prog *ast.Program, b backend.FullBackend) []linte type taskClaimResolver struct { script map[string]*ast.CreateMicroflowStmt stored storedClaimSource + // version returns the project version's IsAtLeast, or nil when unknown. + // Asked only when a warning is written, so no project is opened for a + // script that has nothing to report. + version func() func(major, minor int) bool +} + +func (r *taskClaimResolver) versionAtLeast() func(major, minor int) bool { + if r.version == nil { + return nil + } + return r.version() } // callClaims reports whether calling flow with the task bound to param may @@ -246,15 +299,7 @@ func (r *taskClaimResolver) taskClaimViolations(name ast.QualifiedName, body []a "RUNTIME (\"You can't complete this user task, it is not assigned to you\") with "+ "the button appearing to do nothing. Neither mxcli check nor mx check catches it.", flowName, s.WorkflowTaskVariable), - Suggestion: fmt.Sprintf( - "Claim the task before setting the outcome:\n"+ - " change $%s (System.WorkflowUserTask_Assignees = [%%CurrentUser%%]);\n"+ - " commit $%s;\n"+ - " TARGETING XPATH / TARGETING MICROFLOW decides who may SEE a task; it does not "+ - "assign it. A microflow this one calls first, passing the task, counts when it claims "+ - "it. Ignore this if the task is claimed elsewhere — in a nanoflow on the button, or "+ - "earlier in the process.", - s.WorkflowTaskVariable, s.WorkflowTaskVariable), + Suggestion: taskClaimSuggestion(s.WorkflowTaskVariable, r.versionAtLeast()), }) } // Nested bodies (loops, decisions, error handlers) can hold either half, @@ -274,7 +319,7 @@ func changeClaimsTask(s *ast.ChangeObjectStmt) bool { if i := strings.LastIndex(name, "."); i >= 0 { name = name[i+1:] } - if strings.EqualFold(strings.Trim(name, `"`), assigneesAssociation) { + if isClaimAssociation(strings.Trim(name, `"`)) { return true } } @@ -297,3 +342,15 @@ func nestedStatements(st ast.MicroflowStatement) []ast.MicroflowStatement { } return nil } + +// projectVersionAtLeast is b's project version as IsAtLeast, nil when unknown. +func projectVersionAtLeast(b backend.FullBackend) func(major, minor int) bool { + if b == nil { + return nil + } + pv := b.ProjectVersion() + if pv == nil || pv.MajorVersion == 0 { + return nil + } + return pv.IsAtLeast +} diff --git a/mdl/executor/validate_task_claim_stored.go b/mdl/executor/validate_task_claim_stored.go index 9accbd452b..902479d077 100644 --- a/mdl/executor/validate_task_claim_stored.go +++ b/mdl/executor/validate_task_claim_stored.go @@ -74,7 +74,7 @@ func storedChangeClaims(act *microflows.ChangeObjectAction) bool { if i := strings.LastIndex(name, "."); i >= 0 { name = name[i+1:] } - if strings.EqualFold(name, assigneesAssociation) { + if isClaimAssociation(name) { return true } } diff --git a/mdl/executor/validate_task_claim_test.go b/mdl/executor/validate_task_claim_test.go index 29b076d167..0df553cf15 100644 --- a/mdl/executor/validate_task_claim_test.go +++ b/mdl/executor/validate_task_claim_test.go @@ -21,7 +21,7 @@ func taskClaimWarnings(t *testing.T, body string) []string { t.Fatalf("parse errors: %v", errs) } var msgs []string - for _, v := range ValidateTaskClaims(prog) { + for _, v := range ValidateTaskClaimsIn(prog, nil) { msgs = append(msgs, v.Message) } return msgs @@ -105,7 +105,7 @@ func taskClaimWarningsIn(t *testing.T, src string, stored storedClaimSource) []s t.Fatalf("parse errors: %v", errs) } var msgs []string - for _, v := range validateTaskClaims(prog, stored) { + for _, v := range validateTaskClaims(prog, stored, nil) { msgs = append(msgs, v.Message) } return msgs @@ -200,3 +200,31 @@ func TestStoredCalleeIsRead(t *testing.T) { t.Errorf("a stored callee that does not claim: got %v, want 1 warning", got) } } + +// ako/mxcli#1810: on Mendix 9 a claim writes WorkflowUserTask_Assignee (the +// association is renamed Assignees in 10.0). It must count as a claim, and the +// suggestion must be the spelling that builds on the project's version — +// Assignees is CE1613 on 9.24. +func TestTaskClaim_NineXAssigneeClaims(t *testing.T) { + got := taskClaimWarnings(t, ` change $Task (System.WorkflowUserTask_Assignee = [%CurrentUser%]); + commit $Task; + set task outcome $Task 'Approve';`) + if len(got) != 0 { + t.Errorf("a 9.x claim through WorkflowUserTask_Assignee must count, got %v", got) + } +} + +func TestTaskClaim_SuggestionFollowsVersion(t *testing.T) { + nine := taskClaimSuggestion("T", func(maj, min int) bool { return 9 > maj || (9 == maj && 24 >= min) }) + if !strings.Contains(nine, "WorkflowUserTask_Assignee =") || strings.Contains(nine, "WorkflowUserTask_Assignees =") { + t.Errorf("9.24 suggestion must use Assignee only:\n%s", nine) + } + eleven := taskClaimSuggestion("T", func(maj, min int) bool { return 11 > maj || (11 == maj && 12 >= min) }) + if !strings.Contains(eleven, "WorkflowUserTask_Assignees =") || strings.Contains(eleven, "WorkflowUserTask_Assignee =") { + t.Errorf("11.12 suggestion must use Assignees only:\n%s", eleven) + } + unknown := taskClaimSuggestion("T", nil) + if !strings.Contains(unknown, "Assignees =") || !strings.Contains(unknown, "Assignee =") || !strings.Contains(unknown, "9.x") { + t.Errorf("unknown version must give both spellings with their versions:\n%s", unknown) + } +} diff --git a/mdl/executor/validate_tostring_short_attribute.go b/mdl/executor/validate_tostring_short_attribute.go new file mode 100644 index 0000000000..9ea18fec5e --- /dev/null +++ b/mdl/executor/validate_tostring_short_attribute.go @@ -0,0 +1,130 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/exprcheck" + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +// validateToStringIntoShortAttribute warns about a create/change member that +// writes `toString()` into a String attribute with a length limit +// (mendixlabs/mxcli#1515 item 2, ako/mxcli#1397). +// +// A Decimal such as minutesBetween($a, $b) or `$x div 3` is rendered with its +// whole fraction: 43 characters were seen at runtime, which overflows a +// String(10) on commit. It builds — mxbuild does not know the value's length — +// so this warns rather than refuses, and only where both halves are +// established: the value is provably a Decimal, and the attribute is provably +// bounded. Anything unknown stays silent. +// +// It runs in the --references pass because the bound is the attribute's, which +// only the project or the script's own CREATE ENTITY knows. +func validateToStringIntoShortAttribute(ctx *ExecContext, prog *ast.Program, sc *scriptContext) { + if prog == nil || sc == nil { + return + } + for _, stmt := range prog.Statements { + var flowQN string + var params []ast.MicroflowParam + var body []ast.MicroflowStatement + switch s := stmt.(type) { + case *ast.CreateMicroflowStmt: + flowQN, params, body = s.Name.String(), s.Parameters, s.Body + case *ast.CreateNanoflowStmt: + flowQN, params, body = s.Name.String(), s.Parameters, s.Body + default: + continue + } + kinds := map[string]exprcheck.TypeKind{} + for _, p := range params { + if k, ok := astKindToExprKind(p.Type.Kind); ok { + kinds[p.Name] = k + } + } + walkTypedFlow(ctx, sc, params, body, func(st ast.MicroflowStatement, vars map[string]string) { + var entity, what string + var changes []ast.ChangeItem + switch s := st.(type) { + case *ast.DeclareStmt: + if k, ok := astKindToExprKind(s.Type.Kind); ok { + kinds[s.Variable] = k + } + return + case *ast.CreateObjectStmt: + entity, what, changes = s.EntityType.String(), "create "+s.EntityType.String(), s.Changes + case *ast.ChangeObjectStmt: + entity, what, changes = vars[s.Variable], "change $"+s.Variable, s.Changes + default: + return + } + if entity == "" { + return + } + for _, ch := range changes { + if ch.Op != ast.MemberSet { + continue + } + src := microflowExprSource(ch.Value) + if !exprcheck.ToStringOfDecimal(src, kinds) { + continue + } + length := boundedStringLength(ctx, sc, entity, ch.Attribute) + if length <= 0 { + continue + } + sc.warnings = append(sc.warnings, fmt.Sprintf( + "%s: %s sets %s, a String(%d), to %s — a Decimal is rendered with its whole "+ + "fraction (dozens of characters), which overflows the attribute when it is "+ + "committed. Round it first, e.g. toString(round(…)) or formatDecimal(…, '#0.00').", + flowQN, what, ch.Attribute, length, src)) + } + }) + } +} + +// boundedStringLength returns the length limit of entity's String attribute +// attr, or 0 when it is unlimited, not a String, or cannot be established. The +// script's own declaration wins over the stored entity, as it does on exec. +func boundedStringLength(ctx *ExecContext, sc *scriptContext, entityQN, attr string) int { + if decl := sc.entityDecls[entityQN]; decl != nil { + for _, a := range decl.Attributes { + if a.Name == attr { + if a.Type.Kind == ast.TypeString && a.Type.Length > 0 { + return a.Type.Length + } + return 0 + } + } + if decl.Generalization == nil { + return 0 + } + entityQN = decl.Generalization.String() + } + b, ok := ctx.Backend.(entityLookupBackend) + if !ok { + return 0 + } + seen := map[string]bool{} + for qn := entityQN; qn != "" && !seen[qn]; { + seen[qn] = true + ent, found := findEntityByQN(b, qn) + if !found { + return 0 + } + for _, a := range ent.Attributes { + if a == nil || a.Name != attr { + continue + } + if st, isString := a.Type.(*domainmodel.StringAttributeType); isString && st.Length > 0 { + return st.Length + } + return 0 + } + qn = ent.GeneralizationRef + } + return 0 +} diff --git a/mdl/executor/validate_tostring_short_attribute_test.go b/mdl/executor/validate_tostring_short_attribute_test.go new file mode 100644 index 0000000000..c644dd5962 --- /dev/null +++ b/mdl/executor/validate_tostring_short_attribute_test.go @@ -0,0 +1,81 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +// mendixlabs/mxcli#1515 item 2 (ako/mxcli#1397): `toString` of a Decimal-valued +// expression (minutesBetween, div) "can produce a long fraction string (43 +// characters seen at runtime) and overflow a bounded String attribute. +// Assigning a Decimal to an Integer is caught (MDL041), toString of one is not. +// Expected: a warning." +func toStringShortAttrWarnings(t *testing.T, src string) string { + t.Helper() + mod := mkModule("Tmp") + item := mkEntity(mod.ID, "Item") + item.Attributes = []*domainmodel.Attribute{ + {Name: "Lbl", Type: &domainmodel.StringAttributeType{Length: 10}}, + {Name: "Note", Type: &domainmodel.StringAttributeType{Length: 0}}, // unlimited + } + dm := mkDomainModel(mod.ID, item) + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListFoldersFunc: func() ([]*types.FolderInfo, error) { return nil, nil }, + ListDomainModelsFunc: func() ([]*domainmodel.DomainModel, error) { return []*domainmodel.DomainModel{dm}, nil }, + GetModuleByNameFunc: func(string) (*model.Module, error) { return mod, nil }, + GetDomainModelFunc: func(model.ID) (*domainmodel.DomainModel, error) { return dm, nil }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy(mod))) + prog := parseMDL(t, src) + sc := newScriptContext() + sc.collectDefinitions(prog) + validateToStringIntoShortAttribute(ctx, prog, sc) + return strings.Join(sc.warnings, "\n") +} + +func TestToStringOfDecimalIntoShortAttributeWarns(t *testing.T) { + got := toStringShortAttrWarnings(t, `create microflow Tmp.F ($a: DateTime, $b: DateTime, $i: Tmp.Item) +begin + change $i (Lbl = toString(minutesBetween($a, $b))); +end;`) + for _, want := range []string{"Lbl", "String(10)", "toString(minutesBetween($a, $b))", "round"} { + if !strings.Contains(got, want) { + t.Errorf("want a warning naming %q; got:\n%s", want, got) + } + } +} + +func TestToStringIntoAttributeStaysQuietWhenSafe(t *testing.T) { + for name, body := range map[string]string{ + "unlimited attribute": `change $i (Note = toString(minutesBetween($a, $b)));`, + "rounded first": `change $i (Lbl = toString(round(minutesBetween($a, $b))));`, + "integer value": `change $i (Lbl = toString($n));`, + "create, unlimited attr": `$x = create Tmp.Item (Note = toString($d));`, + } { + got := toStringShortAttrWarnings(t, `create microflow Tmp.F ($a: DateTime, $b: DateTime, $n: Integer, $d: Decimal, $i: Tmp.Item) +begin + `+body+` +end;`) + if got != "" { + t.Errorf("%s: unexpected warning:\n%s", name, got) + } + } + // script-declared entity, create form, Decimal parameter + got := toStringShortAttrWarnings(t, `create persistent entity Tmp.Other (Code: String(5)); +create microflow Tmp.G ($d: Decimal) +begin + $o = create Tmp.Other (Code = toString($d)); +end;`) + if !strings.Contains(got, "Code") || !strings.Contains(got, "String(5)") { + t.Errorf("script-declared String(5) should warn; got:\n%s", got) + } +} diff --git a/mdl/executor/validate_unsupported_actions.go b/mdl/executor/validate_unsupported_actions.go new file mode 100644 index 0000000000..7b53f5d589 --- /dev/null +++ b/mdl/executor/validate_unsupported_actions.go @@ -0,0 +1,126 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// A stored activity MDL has no statement for is read as an UnsupportedAction and +// described as a `-- Unsupported action: ` comment, which parses to +// nothing. The splice path (create or modify of an existing flow) keeps it, +// because it does not rewrite it. Every path that builds the flow from the +// statement instead — a drop followed by a create, the route MDL-V1-REBUILD +// itself recommends, or a whole-flow rebuild — writes the flow without it, and +// reports success: the model is valid without the activity, so nothing fails at +// build time either. mendixlabs/mxcli#1735 lost a Send external object that way, +// and with it every save of the external entity. +// +// These guards refuse instead, naming what would be lost. + +// unsupportedActionTypes lists the storage types of the activities in a stored +// flow body that MDL cannot express, loop bodies included, sorted and deduped. +func unsupportedActionTypes(oc *microflows.MicroflowObjectCollection) []string { + seen := map[string]bool{} + var walk func(*microflows.MicroflowObjectCollection) + walk = func(c *microflows.MicroflowObjectCollection) { + if c == nil { + return + } + for _, obj := range c.Objects { + switch o := obj.(type) { + case *microflows.ActionActivity: + if u, ok := o.Action.(*microflows.UnsupportedAction); ok && u != nil { + seen[u.StorageType] = true + } + case *microflows.LoopedActivity: + walk(o.ObjectCollection) + } + } + } + walk(oc) + out := make([]string, 0, len(seen)) + for t := range seen { + out = append(out, t) + } + sort.Strings(out) + return out +} + +// unsupportedActionLossError is the refusal both guards give. +func unsupportedActionLossError(kind, qn, how string, types []string) error { + return mdlerrors.NewUnsupported(fmt.Sprintf( + "%s %s holds activities MDL cannot express (%s); %s would write the %s without them, and nothing "+ + "would report the loss. Change the %s with `create or modify` (which keeps them) or `alter %s %s { … }`, "+ + "or remove them in Studio Pro first if losing them is intended.", + kind, qn, strings.Join(types, ", "), how, kind, kind, kind, qn)) +} + +// storedFlowUnsupportedActions returns the unexpressible activities of the +// stored microflow or nanoflow qn, or nil when there is none or it cannot be read. +func storedFlowUnsupportedActions(ctx *ExecContext, kind, qn string) []string { + h, err := getHierarchy(ctx) + if err != nil { + return nil + } + switch kind { + case "microflow": + if mf := findMicroflowByQualifiedName(ctx, qn); mf != nil { + return unsupportedActionTypes(mf.ObjectCollection) + } + case "nanoflow": + nfs, err := ctx.Backend.ListNanoflows() + if err != nil { + return nil + } + for _, nf := range nfs { + if !nf.Excluded && h.GetQualifiedName(nf.ContainerID, nf.Name) == qn { + return unsupportedActionTypes(nf.ObjectCollection) + } + } + } + return nil +} + +// validateDropRecreateKeepsActivities refuses a script that drops a flow and +// creates it again when the stored flow holds an activity MDL cannot express. +// It runs before anything is written (check -p and exec's pre-flight), because +// once the drop has run there is nothing left to keep. +func validateDropRecreateKeepsActivities(ctx *ExecContext, prog *ast.Program) []error { + if prog == nil || !ctx.Connected() { + return nil + } + dropped := map[string]bool{} // "kind\x00qn" dropped earlier in the script + var errs []error + for _, stmt := range prog.Statements { + var kind, qn string + switch s := stmt.(type) { + case *ast.DropMicroflowStmt: + dropped["microflow\x00"+s.Name.String()] = true + continue + case *ast.DropNanoflowStmt: + dropped["nanoflow\x00"+s.Name.String()] = true + continue + case *ast.CreateMicroflowStmt: + kind, qn = "microflow", s.Name.String() + case *ast.CreateNanoflowStmt: + kind, qn = "nanoflow", s.Name.String() + default: + continue + } + if !dropped[kind+"\x00"+qn] { + continue + } + delete(dropped, kind+"\x00"+qn) + if types := storedFlowUnsupportedActions(ctx, kind, qn); len(types) > 0 { + errs = append(errs, unsupportedActionLossError(kind, qn, "dropping it and creating it again", types)) + } + } + return errs +} diff --git a/mdl/executor/validate_unsupported_actions_test.go b/mdl/executor/validate_unsupported_actions_test.go new file mode 100644 index 0000000000..fc45b1fb53 --- /dev/null +++ b/mdl/executor/validate_unsupported_actions_test.go @@ -0,0 +1,96 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// mendixlabs/mxcli#1735 (ako/mxcli#1939): "the drop-and-create rebuild, which +// the MDL-V1-REBUILD refusal recommends, deletes an existing send activity +// silently. check passes, exec prints no warning, and the activity is gone." +// Any stored activity MDL cannot express is read as an UnsupportedAction; every +// path that builds the flow from the statement must refuse rather than drop it. + +func storedFlowWithUnsupported(t *testing.T) (*mock.MockBackend, *ExecContext) { + t.Helper() + mod := mkModule("Main") + existing := µflows.Microflow{Name: "ACT_Save"} + existing.ID = "mf-existing" + existing.ContainerID = mod.ID + loop := µflows.LoopedActivity{ObjectCollection: µflows.MicroflowObjectCollection{ + Objects: []microflows.MicroflowObject{ + µflows.ActionActivity{Action: µflows.UnsupportedAction{StorageType: "Microflows$CounterMeterAction"}}, + }, + }} + existing.ObjectCollection = µflows.MicroflowObjectCollection{Objects: []microflows.MicroflowObject{ + µflows.ActionActivity{Action: µflows.UnsupportedAction{StorageType: "Microflows$PushToClientAction"}}, + loop, // nested: the walk must reach loop bodies + }} + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListMicroflowsFunc: func() ([]*microflows.Microflow, error) { + return []*microflows.Microflow{existing}, nil + }, + UpdateMicroflowFunc: func(mf *microflows.Microflow) error { + t.Error("UpdateMicroflow must not run: the rewrite would drop the unexpressible activities") + return nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy(mod))) + return mb, ctx +} + +func TestUnsupportedActionTypes_WalksLoops(t *testing.T) { + _, ctx := storedFlowWithUnsupported(t) + got := storedFlowUnsupportedActions(ctx, "microflow", "Main.ACT_Save") + want := "Microflows$CounterMeterAction,Microflows$PushToClientAction" + if strings.Join(got, ",") != want { + t.Errorf("got %v, want %s", got, want) + } +} + +func TestDropRecreateOfFlowWithUnsupportedActionIsRefused(t *testing.T) { + _, ctx := storedFlowWithUnsupported(t) + prog := parseMDL(t, `drop microflow Main.ACT_Save; +create microflow Main.ACT_Save () +begin + return; +end;`) + errs := validateDropRecreateKeepsActivities(ctx, prog) + if len(errs) != 1 || !strings.Contains(errs[0].Error(), "Microflows$PushToClientAction") { + t.Fatalf("want one refusal naming the activity, got %v", errs) + } + + // controls: a drop alone is a deliberate deletion, and a create without a + // preceding drop is not this guard's case + for _, src := range []string{ + "drop microflow Main.ACT_Save;", + "create or modify microflow Main.ACT_Save ()\nbegin\n return;\nend;", + } { + if errs := validateDropRecreateKeepsActivities(ctx, parseMDL(t, src)); len(errs) != 0 { + t.Errorf("%q: unexpected refusal %v", src, errs) + } + } +} + +// The whole-flow rebuild (what a create or modify the splice cannot make falls +// back to under mdl 0) is buildMicroflowFromStmt; the guard sits inside it, so +// this fails if it is removed. +func TestRebuildOfFlowWithUnsupportedActionIsRefused(t *testing.T) { + _, ctx := storedFlowWithUnsupported(t) + _, err := buildMicroflowFromStmt(ctx, &ast.CreateMicroflowStmt{ + Name: ast.QualifiedName{Module: "Main", Name: "ACT_Save"}, + CreateOrModify: true, + }, buildFlowOpts{AllowCreate: true}) + if err == nil || !strings.Contains(err.Error(), "Microflows$CounterMeterAction") { + t.Fatalf("expected the rebuild to be refused naming the activities, got %v", err) + } +} diff --git a/mdl/executor/validate_widget_attribute_scope.go b/mdl/executor/validate_widget_attribute_scope.go index e4a46fa189..f1ffe47e6d 100644 --- a/mdl/executor/validate_widget_attribute_scope.go +++ b/mdl/executor/validate_widget_attribute_scope.go @@ -76,6 +76,9 @@ func validatePluggableAttributeScopes(ctx *ExecContext, layout string, params [] assocs: checkAssociationShapes(ctx, sc), pb: &pageBuilder{ctx: ctx, backend: ctx.Backend, execCache: ctx.Cache}, } + if sc != nil { + v.generalizations = sc.entityGeneralizations + } for _, w := range widgets { v.walk(w, dataContext{}) } @@ -97,6 +100,11 @@ type attributeScopeValidator struct { // assocs feeds MDL-ASSOCDS01 (validate_assoc_list_source.go). assocs map[string]assocShape + // generalizations are the script's entities' EXTENDS targets (lower-cased + // name -> generalization, "" for none), ahead of the project's, for + // MDL-ASSOCDS03 (validate_assoc_hop_start.go). + generalizations map[string]string + // pb answers the association-as-attribute question with exec's own // function (validate_assoc_as_attribute.go). pb *pageBuilder @@ -115,6 +123,7 @@ func (v *attributeScopeValidator) walk(w *ast.WidgetV3, enclosing dataContext) { v.checkInputBinding(w, enclosing) v.checkAssocListSource(w, enclosing) v.checkViewAssocReverse(w, enclosing) + v.checkAssocHopStart(w, enclosing) v.checkReactUnsupported(w) if v.registry == nil { // No registry: the pluggable half cannot run, and a pluggable widget's diff --git a/mdl/executor/validate_widgets.go b/mdl/executor/validate_widgets.go index dc0e3c01b4..8b8bd4b780 100644 --- a/mdl/executor/validate_widgets.go +++ b/mdl/executor/validate_widgets.go @@ -320,6 +320,8 @@ func validateWidgetTreeIn(widgets []*ast.WidgetV3, registry *WidgetRegistry, loc // The widget's OWN action is judged in the context IT establishes, not the // one it sits in — a list widget's onClick is row-scoped (ako/mxcli#552). out = append(out, validateShowPageArguments(w, argContextForOwnAction(w, argCtx), locationPrefix)...) + // A bare `delete` with no object to delete is CE7005 (ako/mxcli#1932). + out = append(out, validateDeleteSource(w, argContextForOwnAction(w, argCtx), locationPrefix)...) // An `Attribute:` binding with no object to bind to is written empty. out = append(out, validateInputBindingContext(w, argCtx, locationPrefix)...) // Unknown-property warning applies only to built-in widgets; pluggable diff --git a/mdl/executor/version_gates_flow_actions.go b/mdl/executor/version_gates_flow_actions.go index 18cd77c096..f920358b8c 100644 --- a/mdl/executor/version_gates_flow_actions.go +++ b/mdl/executor/version_gates_flow_actions.go @@ -87,6 +87,24 @@ var flowActionGates = []typeGate{ return ok && len(s.Arguments) > 0 }, }, + { + versionGate: versionGate{"microflows", "send_external_object", "send external object", + "this version has no Microflows$SendExternalObject: upgrade the project"}, + types: []string{"Microflows$SendExternalObject"}, + match: func(st ast.MicroflowStatement) bool { + s, ok := st.(*ast.ExternalObjectStmt) + return ok && !s.Delete + }, + }, + { + versionGate: versionGate{"microflows", "delete_external_object", "delete external object", + "this version has no Microflows$DeleteExternalObject: upgrade the project"}, + types: []string{"Microflows$DeleteExternalObject"}, + match: func(st ast.MicroflowStatement) bool { + s, ok := st.(*ast.ExternalObjectStmt) + return ok && s.Delete + }, + }, { versionGate: versionGate{"microflows", "call_external_action", "call external action", "this version has no Microflows$CallExternalAction: upgrade the project"}, @@ -111,7 +129,9 @@ var flowActionGates = []typeGate{ { versionGate: versionGate{"microflows", "queued_calls", "call … in queue", "this version has no task queues on a call (Queues$QueueSettings): leave out `in queue`, or upgrade the project"}, - types: []string{"Queues$QueueSettings"}, + // The retry types arrived with QueueSettings (9.10.0) and are only written + // under it, so the same gate covers `in queue … retry …` (#1931). + types: []string{"Queues$QueueSettings", "Queues$QueueFixedRetry", "Queues$QueueExponentialRetry"}, properties: []string{"Microflows$MicroflowCall.QueueSettings", "Microflows$JavaActionCallAction.QueueSettings"}, match: func(st ast.MicroflowStatement) bool { switch s := st.(type) { diff --git a/mdl/executor/widget_container_keyword_test.go b/mdl/executor/widget_container_keyword_test.go index 78d3110743..5691874084 100644 --- a/mdl/executor/widget_container_keyword_test.go +++ b/mdl/executor/widget_container_keyword_test.go @@ -22,6 +22,8 @@ import ( "regexp" "strings" "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" ) // containerKeywordsInGrammar reads the keywords widgetTypeV3 accepts straight @@ -42,15 +44,15 @@ func containerKeywordsInGrammar(t *testing.T) map[string]bool { return out } -// knownUnsupportedContainers are derived keywords that collide with an existing -// token (ATTRIBUTE, ATTR and EVENT are already lexed for other purposes), so they -// cannot simply be added to widgetTypeV3. Their widgets' object lists are not -// authorable until the collision is resolved — tracked in issue #239, not -// silently tolerated. -var knownUnsupportedContainers = map[string]string{ - "ATTRIBUTE": "collides with the Attribute: property keyword (htmlelement attributes)", - "ATTR": "collides with the ATTR token (accessibilityhelper)", - "EVENT": "collides with the EVENT token (htmlelement events)", +// parsesAsContainer reports whether kw parses as a widget-body container. The +// grammar's `| keyword` alternative of widgetTypeV3 admits a keyword token that +// is not enumerated — `attr`, `attribute`, `event`. Those three were listed here +// as knowingly unsupported (#239) after that alternative had made them writable +// (ako/mxcli#904); asking the parser keeps the test honest without a second list. +func parsesAsContainer(kw string) bool { + _, errs := visitor.Build("create page M.P (Title: 'T', Layout: A.L) {\n" + + " pluggablewidget 'w.W' w1 {\n " + strings.ToLower(kw) + " (x: 'y')\n }\n}") + return len(errs) == 0 } func TestDerivedContainerKeywordsExistInTheGrammar(t *testing.T) { @@ -61,7 +63,7 @@ func TestDerivedContainerKeywordsExistInTheGrammar(t *testing.T) { for _, propertyKey := range []string{ "columns", "groups", "series", "lines", "markers", "dynamicMarkers", "scaleColors", "customItems", "basicItems", "customButtons", - "allowedFileFormats", + "allowedFileFormats", "attributesList", "attributes", "events", } { kw := strings.ToUpper(deriveObjectListKeyword(propertyKey)) if kw == "" { @@ -70,12 +72,10 @@ func TestDerivedContainerKeywordsExistInTheGrammar(t *testing.T) { if known[kw] { continue } - if why, ok := knownUnsupportedContainers[kw]; ok { - t.Logf("%s (from %q) is knowingly unsupported: %s", kw, propertyKey, why) + if parsesAsContainer(kw) { continue } t.Errorf("object-list property %q derives container keyword %q, which widgetTypeV3 does not accept — "+ - "DESCRIBE will emit MDL that does not parse; add it to MDLLexer.g4 + MDLPage.g4, or record it in "+ - "knownUnsupportedContainers with the reason", propertyKey, kw) + "DESCRIBE will emit MDL that does not parse; add it to MDLLexer.g4 + MDLPage.g4", propertyKey, kw) } } diff --git a/mdl/executor/widget_engine.go b/mdl/executor/widget_engine.go index ec45ea930b..b19e20e509 100644 --- a/mdl/executor/widget_engine.go +++ b/mdl/executor/widget_engine.go @@ -517,7 +517,7 @@ func (e *PluggableWidgetEngine) Build(def *WidgetDefinition, w *ast.WidgetV3) (* w.Name, def.MDLName, len(dsProps), strings.Join(dsProps, ", ")) } for _, propKey := range dsProps { - dataSource, entityName, err := e.pageBuilder.buildPluggableDataSourceV3(ds) + dataSource, entityName, err := e.pageBuilder.buildPluggableDataSourceV3(ds, w.Name) if err != nil { return nil, mdlerrors.NewBackend("auto datasource for "+propKey, err) } @@ -1760,7 +1760,7 @@ func (e *PluggableWidgetEngine) resolveMapping(mapping PropertyMapping, w *ast.W ds = w.GetDataSource() } if ds != nil { - dataSource, entityName, err := e.pageBuilder.buildPluggableDataSourceV3(ds) + dataSource, entityName, err := e.pageBuilder.buildPluggableDataSourceV3(ds, w.Name) if err != nil { return nil, mdlerrors.NewBackend("build datasource", err) } diff --git a/mdl/exprcheck/adapters/adapter_scope.go b/mdl/exprcheck/adapters/adapter_scope.go index b49631178d..02a608865b 100644 --- a/mdl/exprcheck/adapters/adapter_scope.go +++ b/mdl/exprcheck/adapters/adapter_scope.go @@ -198,6 +198,8 @@ func StatementErrorHandling(stmt ast.MicroflowStatement) *ast.ErrorHandlingClaus return s.ErrorHandling case *ast.MfCommitStmt: return s.ErrorHandling + case *ast.ExternalObjectStmt: + return s.ErrorHandling case *ast.RetrieveStmt: return s.ErrorHandling case *ast.CallMicroflowStmt: diff --git a/mdl/exprcheck/signatures.go b/mdl/exprcheck/signatures.go new file mode 100644 index 0000000000..08c1ed3fd5 --- /dev/null +++ b/mdl/exprcheck/signatures.go @@ -0,0 +1,88 @@ +// SPDX-License-Identifier: Apache-2.0 + +package exprcheck + +import ( + "sort" + "strconv" + "strings" +) + +// FunctionSignature is one built-in expression function as the checker knows +// it, for documentation. `mxcli syntax microflow expressions functions` is +// rendered from these, so the reference and the checker cannot disagree about +// which functions exist or what they take (mendixlabs/mxcli#1708). +type FunctionSignature struct { + Name string + // Params holds each parameter's kind name; a parameter that accepts more + // than one kind is written "Decimal|DateTime". + Params []string + // MinArgs is how many leading Params are required. + MinArgs int + Variadic bool // the last parameter repeats + Return string + // ReturnNote explains a result kind that is not fixed: round()'s depends on + // its arity, abs/max/min follow their operands. + ReturnNote string +} + +// String renders the signature as `name(A, B[, C]) -> R`. +func (s FunctionSignature) String() string { + var b strings.Builder + b.WriteString(s.Name) + b.WriteByte('(') + for i, p := range s.Params { + switch { + case i == 0: + case i >= s.MinArgs: + b.WriteString("[, ") + default: + b.WriteString(", ") + } + b.WriteString(p) + } + if s.Variadic { + b.WriteString(", …") + } + for i := len(s.Params) - 1; i >= s.MinArgs && i > 0; i-- { + b.WriteByte(']') + } + b.WriteString(") -> ") + b.WriteString(s.Return) + return b.String() +} + +// Functions returns every verified built-in, sorted by name. Rows marked +// unverified are left out: each was measured as CE0117 (mendixlabs/mxcli#1515) +// and is only kept in the table so MDL044 does not change behaviour by itself. +func Functions() []FunctionSignature { + out := make([]FunctionSignature, 0, len(funcTable)) + for name, sig := range funcTable { + if sig.unverified { + continue + } + fs := FunctionSignature{ + Name: name, + MinArgs: sig.min(), + Variadic: sig.variadic, + Return: typeKindName(sig.ret), + } + for i := range sig.args { + fs.Params = append(fs.Params, strings.Join(typeKindNames(sig.accepts(i)), "|")) + } + switch { + case sig.retFromArgs: + fs.ReturnNote = "Integer when every operand is Integer, else Decimal" + case len(sig.retByArity) > 0: + var notes []string + for argc, k := range sig.retByArity { + notes = append(notes, typeKindName(k)+" with "+strconv.Itoa(argc)+" argument(s)") + } + sort.Strings(notes) + fs.ReturnNote = strings.Join(notes, "; ") + } + out = append(out, fs) + } + sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name }) + return out +} diff --git a/mdl/exprcheck/tostring_decimal_test.go b/mdl/exprcheck/tostring_decimal_test.go new file mode 100644 index 0000000000..d2456d3704 --- /dev/null +++ b/mdl/exprcheck/tostring_decimal_test.go @@ -0,0 +1,23 @@ +// SPDX-License-Identifier: Apache-2.0 + +package exprcheck + +import "testing" + +func TestToStringOfDecimal(t *testing.T) { + vars := map[string]TypeKind{"d": KindDecimal, "i": KindInteger, "a": KindDateTime, "b": KindDateTime} + for src, want := range map[string]bool{ + "toString(minutesBetween($a, $b))": true, + "toString($d)": true, + "toString($i div 3)": true, + "toString(round(minutesBetween($a, $b)))": false, // one-arg round is whole + "toString($i)": false, + "toString($unknown)": false, + "minutesBetween($a, $b)": false, + "'x' + toString($d)": false, // not the whole value + } { + if got := ToStringOfDecimal(src, vars); got != want { + t.Errorf("ToStringOfDecimal(%q) = %v, want %v", src, got, want) + } + } +} diff --git a/mdl/exprcheck/unknown_funcs.go b/mdl/exprcheck/unknown_funcs.go index 89b07dc6f5..62e88905db 100644 --- a/mdl/exprcheck/unknown_funcs.go +++ b/mdl/exprcheck/unknown_funcs.go @@ -187,3 +187,25 @@ func SourceRejectedForIntegerTarget(src string, vars map[string]TypeKind) bool { } return false } + +// ToStringOfDecimal reports whether src is `toString(x)` with x provably a +// Decimal: a Decimal-returning built-in such as minutesBetween(), Decimal +// arithmetic, or a Decimal variable from vars. Mendix renders such a value with +// its full fraction — 43 characters were seen at runtime for a minutesBetween — +// so writing it to a short String attribute overflows (mendixlabs/mxcli#1515). +// Unknown inference answers false. +func ToStringOfDecimal(src string, vars map[string]TypeKind) bool { + if strings.TrimSpace(src) == "" { + return false + } + ctx := Context{} + if vars != nil { + ctx.Scope = mapScope(vars) + } + root, _ := (&parserImpl{}).Parse(src, ctx) + call, ok := unwrapParens(root).(*CallExpr) + if !ok || call.Name != "toString" || call.Qualified || len(call.Args) != 1 { + return false + } + return inferKind(call.Args[0], ctx) == KindDecimal +} diff --git a/mdl/grammar/domains/MDLCatalog.g4 b/mdl/grammar/domains/MDLCatalog.g4 index 0d16d0d2bb..a0e84a0457 100644 --- a/mdl/grammar/domains/MDLCatalog.g4 +++ b/mdl/grammar/domains/MDLCatalog.g4 @@ -119,9 +119,9 @@ showStatement | showOrList PUBLISHED REST SERVICES (IN (qualifiedName | IDENTIFIER))? | showOrList DATA TRANSFORMERS (IN (qualifiedName | IDENTIFIER))? | showOrList LANGUAGES - | showOrList FEATURES (IN IDENTIFIER)? - | showOrList FEATURES FOR VERSION NUMBER_LITERAL - | showOrList FEATURES ADDED SINCE NUMBER_LITERAL + | showOrList FEATURES (IN identifierOrKeyword)? // area names are keywords: workflows, pages, … (#1367) + | showOrList FEATURES FOR VERSION featureVersion + | showOrList FEATURES ADDED SINCE featureVersion | showOrList JAR DEPENDENCIES (IN (qualifiedName | IDENTIFIER))? ; @@ -399,3 +399,9 @@ limitOffsetClause : LIMIT NUMBER_LITERAL (OFFSET NUMBER_LITERAL)? | OFFSET NUMBER_LITERAL (LIMIT NUMBER_LITERAL)? ; + +// A Mendix version as written everywhere else: 10.24 or 10.24.3. NUMBER_LITERAL +// takes one dot, so a patch arrives as DOT NUMBER_LITERAL (ako/mxcli#1367). +featureVersion + : NUMBER_LITERAL (DOT NUMBER_LITERAL)? + ; diff --git a/mdl/grammar/domains/MDLMicroflow.g4 b/mdl/grammar/domains/MDLMicroflow.g4 index 63484356ef..70c62ef104 100644 --- a/mdl/grammar/domains/MDLMicroflow.g4 +++ b/mdl/grammar/domains/MDLMicroflow.g4 @@ -65,14 +65,22 @@ createJavaActionStatement LPAREN javaActionParameterList? RPAREN javaActionReturnType? javaActionExposedClause? - AS DOLLAR_STRING javaActionExtraCode? SEMICOLON? + AS sourceCodeText javaActionExtraCode? SEMICOLON? + ; + +/** A Java or JavaScript action's source: `$$ … $$`, or a string literal for a + * text holding `$$`, which no dollar-quoted literal can contain — describe + * writes that form for it (ako/mxcli#1003). */ +sourceCodeText + : DOLLAR_STRING + | STRING_LITERAL ; /** The .java file's EXTRA CODE section — helper methods, constants and nested * classes beside executeAction(), which a regeneration keeps. Omitted, the * stored section is kept; `extra code $$ $$` clears it (ako/mxcli#1702). */ javaActionExtraCode - : EXTRA CODE DOLLAR_STRING + : EXTRA CODE sourceCodeText ; javaActionParameterList @@ -144,7 +152,7 @@ createJavaScriptActionStatement javaActionReturnType? javaActionExposedClause? javaScriptPlatformClause? - AS DOLLAR_STRING SEMICOLON? + AS sourceCodeText SEMICOLON? ; javaScriptPlatformClause @@ -284,6 +292,7 @@ microflowStatement | annotation* commitStatement SEMICOLON | annotation* deleteObjectStatement SEMICOLON | annotation* rollbackStatement SEMICOLON + | annotation* externalObjectStatement SEMICOLON | annotation* retrieveStatement SEMICOLON | annotation* ifStatement SEMICOLON | annotation* loopStatement SEMICOLON @@ -459,6 +468,14 @@ rollbackStatement : ROLLBACK VARIABLE REFRESH? onErrorClause? ; +// SEND EXTERNAL OBJECT $District [REFRESH]; and DELETE EXTERNAL OBJECT $District +// [REFRESH]; — Studio Pro's "Send external object" and "Delete external object" +// (mendixlabs/mxcli#1735). A plain commit or delete on an external entity is a +// Studio Pro error; these are how an OData write is saved or removed. +externalObjectStatement + : (SEND | DELETE) EXTERNAL OBJECT VARIABLE REFRESH? onErrorClause? + ; + // RETRIEVE $Product FROM MfTest.Product WHERE Code = $SearchCode SORT BY Name ASC FIRST; // RETRIEVE $Top FROM MfTest.Product SORT BY Price DESC LIMIT 10 OFFSET 20; // @@ -602,7 +619,20 @@ callMicroflowStatement // on the call activity). Valid on CALL MICROFLOW and CALL JAVA ACTION only; // Mendix has no queued nanoflow or JavaScript action. queueClause - : IN QUEUE qualifiedName + : IN QUEUE qualifiedName queueRetryClause? + ; + +// RETRY FIXED (Retries: 5, Interval: 10, IntervalType: Minutes) +// RETRY EXPONENTIAL (Retries: 20, InitialInterval: 1, MaximumInterval: 60, IntervalType: Minutes) +// The retry policy of a queued call (Queues$QueueFixedRetry / +// Queues$QueueExponentialRetry, mendixlabs/mxcli#1729). FIXED / EXPONENTIAL and +// the property names are words, checked by the visitor, so they reserve nothing. +queueRetryClause + : RETRY identifierOrKeyword LPAREN queueRetryProperty (COMMA queueRetryProperty)* RPAREN + ; + +queueRetryProperty + : identifierOrKeyword COLON (NUMBER_LITERAL | identifierOrKeyword) ; callNanoflowStatement @@ -758,7 +788,8 @@ callArgumentList // its expression is empty, which is what Studio Pro stores when a call's // argument field is left blank (`Argument: ""`). It is not `empty` — that is // a Mendix value, stored as the text `empty` — and not an omitted argument, -// which writes no mapping at all. Only call microflow / call nanoflow take it +// which writes no mapping at all. Only call microflow / call nanoflow / call +// external action take it // (the visitor refuses it elsewhere). NOTHING is listed before expression so // the keyword wins over reading `nothing` as a name. // diff --git a/mdl/grammar/domains/MDLPage.g4 b/mdl/grammar/domains/MDLPage.g4 index a8c28e4ceb..15e09a2dcf 100644 --- a/mdl/grammar/domains/MDLPage.g4 +++ b/mdl/grammar/domains/MDLPage.g4 @@ -730,8 +730,10 @@ actionExprV3 | SAVE_CHANGES closePageV3? actionSettingsV3? // save changes [close page] | CANCEL_CHANGES closePageV3? actionSettingsV3? // cancel changes [close page] | closePageV3 actionSettingsV3? // close page - | DELETE closePageV3? actionSettingsV3? // delete [close page] - | DELETE_OBJECT /* @alias MDL-DEPR020 */ closePageV3? actionSettingsV3? + // `delete $grid1` deletes grid1's selection, `delete $Param` a page or + // snippet parameter; bare `delete` the context object (ako/mxcli#1932). + | DELETE VARIABLE? closePageV3? actionSettingsV3? // delete [$source] [close page] + | DELETE_OBJECT /* @alias MDL-DEPR020 */ VARIABLE? closePageV3? actionSettingsV3? | CREATE OBJECT qualifiedName (THEN actionExprV3 | actionSettingsV3)? // create object Entity then show page ... | CREATE_OBJECT /* @alias MDL-DEPR020 */ qualifiedName (THEN actionExprV3 | actionSettingsV3)? | SHOW PAGE qualifiedName microflowArgsV3? actionSettingsV3? // show page Module.Page (Param: val) diff --git a/mdl/linter/rules/mpr008_overlapping_activities.go b/mdl/linter/rules/mpr008_overlapping_activities.go index 3e3da4dddf..6d3c4a07bb 100644 --- a/mdl/linter/rules/mpr008_overlapping_activities.go +++ b/mdl/linter/rules/mpr008_overlapping_activities.go @@ -77,22 +77,18 @@ func (r *OverlappingActivitiesRule) Check(ctx *linter.LintContext) []linter.Viol continue } reported[key] = true + msg, suggestion := overlapViolationText(a, b, mf.DocumentNoun(), mf.ModuleName, mf.Name) violations = append(violations, linter.Violation{ RuleID: r.ID(), Severity: r.DefaultSeverity(), - Message: fmt.Sprintf( - "Activities '%s' (%d,%d) and '%s' (%d,%d) overlap in %s '%s.%s'. "+ - "Each MDL statement that creates a canvas activity needs its own @position annotation.", - a.caption, a.x, a.y, b.caption, b.x, b.y, - mf.DocumentNoun(), mf.ModuleName, mf.Name, - ), + Message: msg, Location: linter.Location{ Module: mf.ModuleName, DocumentType: mf.DocumentNoun(), DocumentName: mf.Name, DocumentID: mf.ID, }, - Suggestion: "Add a separate @position(x, y) annotation before each statement. Use 190px spacing between activities.", + Suggestion: suggestion, }) } } @@ -103,12 +99,49 @@ func (r *OverlappingActivitiesRule) Check(ctx *linter.LintContext) []linter.Viol return violations } +// overlapViolationText words an MPR008 finding for one overlapping pair. +// +// When one box is a loop, the overlap is almost never two statements sharing one +// @position: a loop's box grows with its body, so a loop placed at an ordinary +// activity spacing reaches into the activity before or after it. Saying "each +// statement needs its own @position" there sent users adding annotations that +// were already present (ako/mxcli#1839, mendixlabs/mxcli#1690), so the message +// names the loop, its size, and the neighbour it covers instead. +func overlapViolationText(a, b actInfo, noun, module, name string) (message, suggestion string) { + loop, other := a, b + if !loop.loop { + loop, other = b, a + } + if loop.loop { + message = fmt.Sprintf( + "The box of loop '%s' (%d,%d, %dx%d) overlaps '%s' (%d,%d) in %s '%s.%s'. "+ + "A loop's box grows with its body, so a gap sized for an ordinary activity is not enough.", + loop.caption, loop.x, loop.y, loop.w, loop.h, other.caption, other.x, other.y, + noun, module, name, + ) + suggestion = fmt.Sprintf( + "Widen the gap: the @position x values of the loop and '%s' must differ by more than half of both widths "+ + "(%dpx here). Or drop the @position annotations around the loop and let the layout place it.", + other.caption, (loop.w+other.w)/2, + ) + return message, suggestion + } + message = fmt.Sprintf( + "Activities '%s' (%d,%d) and '%s' (%d,%d) overlap in %s '%s.%s'. "+ + "Each MDL statement that creates a canvas activity needs its own @position annotation.", + a.caption, a.x, a.y, b.caption, b.x, b.y, noun, module, name, + ) + return message, "Add a separate @position(x, y) annotation before each statement. Use 190px spacing between activities." +} + // actInfo is one positioned node on a canvas: its middle point (Mendix stores -// RelativeMiddlePoint) and its box size. +// RelativeMiddlePoint) and its box size. loop marks a LoopedActivity, whose box +// is sized by its body. type actInfo struct { x, y int w, h int caption string + loop bool } // boxesOverlap reports whether two nodes' boxes intersect. Each node is @@ -174,7 +207,9 @@ func overlapPlanes(objects []microflows.MicroflowObject) [][]actInfo { if caption == "" { caption = "(loop)" } - plane = append(plane, newActInfo(act, caption)) + info := newActInfo(act, caption) + info.loop = true + plane = append(plane, info) if act.ObjectCollection != nil { nested = append(nested, act.ObjectCollection.Objects) } diff --git a/mdl/linter/rules/mpr008_overlapping_activities_test.go b/mdl/linter/rules/mpr008_overlapping_activities_test.go index 3d9751308a..da8f60ae1f 100644 --- a/mdl/linter/rules/mpr008_overlapping_activities_test.go +++ b/mdl/linter/rules/mpr008_overlapping_activities_test.go @@ -3,6 +3,7 @@ package rules import ( + "strings" "testing" "github.com/mendixlabs/mxcli/mdl/linter" @@ -204,3 +205,53 @@ func TestOverlapComparesEvents(t *testing.T) { t.Errorf("%d of %d nodes are on the canvas", got, len(nodes)) } } + +// ako/mxcli#1839 (mendixlabs/mxcli#1690): when one of the overlapping boxes is a +// loop, the overlap is the loop's BOX — sized by its content — reaching into the +// activity before or after it, not two statements sharing one @position. The +// generic "each statement needs its own @position" sent users adding +// annotations that were already there. +func TestOverlapMessageNamesTheLoop(t *testing.T) { + act := µflows.ActionActivity{} + act.Position = model.Point{X: 300, Y: 200} + act.Caption = "Retrieve orders" + loop := µflows.LoopedActivity{} + loop.Position = model.Point{X: 420, Y: 200} + loop.Size = model.Size{Width: 400, Height: 200} + loop.Caption = "Iterate orders" + + planes := overlapPlanes([]microflows.MicroflowObject{act, loop}) + a, b := planes[0][0], planes[0][1] + if !boxesOverlap(a, b) { + t.Fatal("fixture does not overlap") + } + msg, sug := overlapViolationText(a, b, "microflow", "M", "ProcessOrders") + for _, want := range []string{"loop 'Iterate orders'", "400x200", "overlaps 'Retrieve orders'"} { + if !strings.Contains(msg, want) { + t.Errorf("message %q does not contain %q", msg, want) + } + } + if strings.Contains(msg, "needs its own @position") { + t.Errorf("loop overlap still blames missing @position: %q", msg) + } + if !strings.Contains(sug, "gap") || !strings.Contains(sug, "@position") { + t.Errorf("suggestion %q should say to widen the gap or drop the @position", sug) + } + // Order of the pair must not matter. + if msg2, _ := overlapViolationText(b, a, "microflow", "M", "ProcessOrders"); !strings.Contains(msg2, "loop 'Iterate orders'") { + t.Errorf("loop hint lost when the loop is the first box: %q", msg2) + } +} + +// Control: two plain activities keep the generic @position message. +func TestOverlapMessageGenericForActivities(t *testing.T) { + a := actInfo{x: 100, y: 100, w: 120, h: 60, caption: "Declare"} + b := actInfo{x: 150, y: 100, w: 120, h: 60, caption: "Set"} + msg, sug := overlapViolationText(a, b, "microflow", "M", "MF") + if !strings.Contains(msg, "needs its own @position") || strings.Contains(msg, "loop") { + t.Errorf("generic overlap message changed: %q", msg) + } + if !strings.Contains(sug, "190px") { + t.Errorf("generic suggestion changed: %q", sug) + } +} diff --git a/mdl/visitor/datasource_args_error_site_test.go b/mdl/visitor/datasource_args_error_site_test.go new file mode 100644 index 0000000000..36b26b2e1c --- /dev/null +++ b/mdl/visitor/datasource_args_error_site_test.go @@ -0,0 +1,44 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "fmt" + "strings" + "testing" +) + +// ako/mxcli#533, the part left open after #926: a malformed data-source +// argument reported "line 3:19 no viable alternative at input 'datasource'" — +// the column of a keyword that is correct, while the mistake sat in the +// argument list further along the line. ANTLR predicts a widget property with +// full-LL lookahead, so its error always lands on the property's first token; +// the message must at least name the argument and its column. +func TestDataSourceArgumentErrorNamesTheArgument(t *testing.T) { + cases := []struct { + args, at, want string + }{ + {"Param = ", "Param", "`Param` has no value"}, + {"Param: ", "Param", "`Param` has no value"}, + {"Param $v", "Param", "`Param $v` is missing `=`"}, + {"A = $a, , B = $b", ", B", "an empty argument"}, + } + for _, c := range cases { + line := " dataview DVMain (datasource: microflow M.DS_X(" + c.args + ")) { }" + src := "create page M.P (title: 'x', layout: Atlas_Core.Atlas_Default) {\n" + line + "\n};" + _, errs := Build(src) + var all []string + for _, e := range errs { + all = append(all, e.Error()) + } + got := strings.Join(all, "\n") + col := strings.Index(line, "M.DS_X(") + len("M.DS_X(") + strings.Index(c.args, strings.TrimPrefix(c.at, ", ")) + if c.at == ", B" { + col = strings.Index(line, ", ,") + 2 // the empty slot after the first comma + } + site := fmt.Sprintf("line 2:%d", col) + if !strings.Contains(got, site) || !strings.Contains(got, c.want) { + t.Errorf("(%s): want %q at %q; got:\n%s", c.args, c.want, site, got) + } + } +} diff --git a/mdl/visitor/datasource_args_hint.go b/mdl/visitor/datasource_args_hint.go new file mode 100644 index 0000000000..eb8ae69e2a --- /dev/null +++ b/mdl/visitor/datasource_args_hint.go @@ -0,0 +1,102 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "fmt" + "regexp" + "strings" +) + +// flowArgsCallRe finds a `microflow M.F(` / `nanoflow M.F(` call on a line: the +// data-source and action forms whose argument lists ANTLR mis-locates. +var flowArgsCallRe = regexp.MustCompile(`(?i)\b(?:microflow|nanoflow)\s+[\w."]+\s*\(`) + +var ( + argNoValueRe = regexp.MustCompile(`^(\$?[\w"]+)\s*[=:]\s*$`) + argWithValueRe = regexp.MustCompile(`^\$?[\w"]+\s*[=:]`) + argMissingEqualRe = regexp.MustCompile(`^[\w"]+\s+\S`) +) + +// flowArgumentHint explains a "no viable alternative" that a malformed flow +// argument list caused (ako/mxcli#533). ANTLR predicts a widget property with +// full-LL lookahead, so an error anywhere inside `datasource: microflow +// M.F(Param = )` is reported at `datasource` — a keyword that is correct. The +// grammar cannot move the report without restructuring every widget property, +// so this reads the argument list on the offending line and names the +// argument at fault, with its own line:column. It returns "" when the line has +// no flow call or every argument looks well formed: the error is then +// somewhere else, and a guess would mislead. +func flowArgumentHint(msg, offendingLine string, line int) string { + if line <= 0 || !strings.Contains(msg, "no viable alternative") { + return "" + } + for _, loc := range flowArgsCallRe.FindAllStringIndex(offendingLine, -1) { + if col, problem := badFlowArgument(offendingLine, loc[1]); problem != "" { + return fmt.Sprintf(" The mistake is in the argument list, at line %d:%d: %s.\n"+ + " Each argument is `Param = `, separated by commas:\n"+ + " microflow Module.DS_Orders(Customer = $Customer, Status = 'Open')", line, col, problem) + } + } + return "" +} + +// badFlowArgument scans the argument list that opens just before index open +// and returns the column and description of the first malformed argument. +func badFlowArgument(line string, open int) (int, string) { + depth := 0 + start := open + var args [][2]int // [start, end) of each argument; end is its delimiter + closed := false + inQuote := false +scan: + for i := open; i < len(line); i++ { + c := line[i] + switch { + case inQuote: + if c == '\'' { + if i+1 < len(line) && line[i+1] == '\'' { + i++ + } else { + inQuote = false + } + } + case c == '\'': + inQuote = true + case c == '(' || c == '[': + depth++ + case (c == ')' || c == ']') && depth > 0: + depth-- + case c == ')': + args = append(args, [2]int{start, i}) + closed = true + break scan + case c == ',' && depth == 0: + args = append(args, [2]int{start, i}) + start = i + 1 + } + } + if !closed { + return 0, "" // runs onto the next line: not judged from one line + } + if len(args) == 1 && strings.TrimSpace(line[args[0][0]:args[0][1]]) == "" { + return 0, "" // `()` — no arguments, which is valid + } + for _, a := range args { + raw := line[a[0]:a[1]] + text := strings.TrimSpace(raw) + col := a[0] + (len(raw) - len(strings.TrimLeft(raw, " \t"))) + switch { + case text == "": + return a[1], "an empty argument (a stray comma)" + case argNoValueRe.MatchString(text): + name := argNoValueRe.FindStringSubmatch(text)[1] + return col, fmt.Sprintf("`%s` has no value", name) + case argWithValueRe.MatchString(text): + continue + case argMissingEqualRe.MatchString(text): + return col, fmt.Sprintf("`%s` is missing `=` between the parameter and its value", text) + } + } + return 0, "" +} diff --git a/mdl/visitor/show_features_for_version_test.go b/mdl/visitor/show_features_for_version_test.go index 3cfe7fcf4b..a1d98fcb36 100644 --- a/mdl/visitor/show_features_for_version_test.go +++ b/mdl/visitor/show_features_for_version_test.go @@ -37,3 +37,30 @@ func TestShowFeaturesForVersionRoutesToFeatures(t *testing.T) { t.Errorf("show version: got %#v, want ShowStmt{ShowVersion}", prog.Statements[0]) } } + +// ako/mxcli#1367: versions are written major.minor.patch everywhere else, and +// `show features for version 10.20.0` was a parse error. The area names are +// lexer keywords (workflows, microflows, pages, …), and `show features in +// workflows` was a parse error too — the documented `in ` form failed +// for most of the areas the registry has. +func TestShowFeaturesAcceptsFullVersionsAndKeywordAreas(t *testing.T) { + cases := map[string]ast.ShowFeaturesStmt{ + "show features for version 10.20.0;": {ForVersion: "10.20.0"}, + "show features added since 11.0.0;": {AddedSince: "11.0.0"}, + "show features for version 10.24;": {ForVersion: "10.24"}, + "show features in workflows;": {InArea: "workflows"}, + "show features in microflows;": {InArea: "microflows"}, + "show features in domain_model;": {InArea: "domain_model"}, + } + for src, want := range cases { + prog, errs := Build(src) + if len(errs) > 0 { + t.Errorf("%q: parse errors: %v", src, errs) + continue + } + got, ok := prog.Statements[0].(*ast.ShowFeaturesStmt) + if !ok || *got != want { + t.Errorf("%q: got %#v, want %#v", src, prog.Statements[0], want) + } + } +} diff --git a/mdl/visitor/visitor.go b/mdl/visitor/visitor.go index 0f4e49afea..d6dd850639 100644 --- a/mdl/visitor/visitor.go +++ b/mdl/visitor/visitor.go @@ -50,7 +50,7 @@ func (l *errorListener) SyntaxError(rec antlr.Recognizer, sym any, line, column if word, ok := unknownStatementWord(rec, sym, msg); ok { msg = unknownStatementMessage(word) } - site := errorSite{routineHeader: inRoutineHeader(rec), backslashEscapes: l.backslashEscapes} + site := errorSite{routineHeader: inRoutineHeader(rec), backslashEscapes: l.backslashEscapes, line: line} enhancedMsg := l.deduplicateHint(enhanceErrorMessageAt(msg, offending, site), line) l.errors = append(l.errors, fmt.Errorf("line %d:%d %s", line, column, enhancedMsg)) l.lines[line] = true @@ -133,6 +133,9 @@ type errorSite struct { // backslashEscapes: the script is mdl 0, where `\'` escapes an apostrophe. // The zero value is mdl 1's rule, where a backslash is an ordinary character. backslashEscapes bool + // line is the 1-based line the error is on, 0 when unknown; a hint that + // points elsewhere on the line names its own line:column with it. + line int } // inRoutineHeader reports whether the parser was inside the header of a create @@ -461,6 +464,12 @@ func enhanceErrorMessageAt(msg, offendingLine string, site errorSite) string { } } + // A malformed flow argument list in a widget property: ANTLR reports it at + // the property keyword, so name the argument (ako/mxcli#533). + if hint := flowArgumentHint(msg, offendingLine, site.line); hint != "" { + return msg + "\n\n" + hint + } + // Nothing matched a specific pattern — the location is precise but the fix // isn't spelled out. Point at the syntax reference so the correct form is one // command away. (Kept to one line since it can repeat across cascading errors.) diff --git a/mdl/visitor/visitor_argument_binding.go b/mdl/visitor/visitor_argument_binding.go index 32309f232b..3ca39d40ed 100644 --- a/mdl/visitor/visitor_argument_binding.go +++ b/mdl/visitor/visitor_argument_binding.go @@ -38,10 +38,10 @@ func (b *Builder) ExitCallArgument(ctx *parser.CallArgumentContext) { b.recordColonArgument(ctx.ParameterName(), ctx.COLON(), ctx.Expression()) return } - if n := ctx.NOTHING(); n != nil && !isFlowCallArgument(ctx) { + if n := ctx.NOTHING(); n != nil && !isBlankableCallArgument(ctx) { tok := n.GetSymbol() - b.addError(fmt.Errorf("line %d:%d `= nothing` (a blank argument) is only valid in call microflow and "+ - "call nanoflow; bind the parameter to a value, or to `empty`", + b.addError(fmt.Errorf("line %d:%d `= nothing` (a blank argument) is only valid in call microflow, "+ + "call nanoflow and call external action; bind the parameter to a value, or to `empty`", tok.GetLine(), tok.GetColumn())) } if tp := ctx.TemplateParams(); tp != nil && !isJavaActionCallArgument(ctx) { @@ -65,17 +65,20 @@ func isJavaActionCallArgument(ctx *parser.CallArgumentContext) bool { return ok } -// isFlowCallArgument reports whether a call argument belongs to a call -// microflow or call nanoflow — the two calls whose parameter mappings store a -// bare expression, where a blank one (`Argument: ""`) is a state Studio Pro -// writes when an argument field is left empty. -func isFlowCallArgument(ctx *parser.CallArgumentContext) bool { +// isBlankableCallArgument reports whether a call argument belongs to a call +// whose parameter mappings store a bare expression, where a blank one +// (`Argument: ""`) is a state Studio Pro writes when an argument field is left +// empty: call microflow, call nanoflow, and call external action — Studio Pro +// 11.16 stores an optional OData action parameter left blank that way +// (ako/mxcli#1915). +func isBlankableCallArgument(ctx *parser.CallArgumentContext) bool { list := ctx.GetParent() if list == nil { return false } switch list.GetParent().(type) { - case *parser.CallMicroflowStatementContext, *parser.CallNanoflowStatementContext: + case *parser.CallMicroflowStatementContext, *parser.CallNanoflowStatementContext, + *parser.CallExternalActionStatementContext: return true } return false diff --git a/mdl/visitor/visitor_javaaction.go b/mdl/visitor/visitor_javaaction.go index 7b569576b7..df3f1562d6 100644 --- a/mdl/visitor/visitor_javaaction.go +++ b/mdl/visitor/visitor_javaaction.go @@ -66,15 +66,10 @@ func (b *Builder) ExitCreateJavaActionStatement(ctx *parser.CreateJavaActionStat } } - // Get Java code from dollar-quoted string - if dollarStr := ctx.DOLLAR_STRING(); dollarStr != nil { - code := dollarStr.GetText() - // Remove the $$ delimiters - if len(code) >= 4 && strings.HasPrefix(code, "$$") && strings.HasSuffix(code, "$$") { - code = code[2 : len(code)-2] - } + // Get Java code from the dollar-quoted (or string) body + if body := ctx.SourceCodeText(); body != nil { // Trim leading/trailing whitespace but preserve internal formatting - code = strings.TrimSpace(code) + code := strings.TrimSpace(sourceCodeTextValue(body)) // Extract import lines so they go into the file-level import section, // not into the executeAction() method body (a common AI agent mistake). stmt.JavaCode, stmt.Imports = extractJavaImports(code) @@ -84,9 +79,9 @@ func (b *Builder) ExitCreateJavaActionStatement(ctx *parser.CreateJavaActionStat // Kept as written apart from the blank lines around it — describe prints // the section as the file holds it, so trimming indentation would shift // every round trip. An import line goes to the import list, as in the body. - if ec := ctx.JavaActionExtraCode(); ec != nil && ec.DOLLAR_STRING() != nil { + if ec := ctx.JavaActionExtraCode(); ec != nil && ec.SourceCodeText() != nil { var imports []string - stmt.ExtraCode, imports = extractJavaExtraCode(ec.DOLLAR_STRING().GetText()) + stmt.ExtraCode, imports = extractJavaExtraCode(sourceCodeTextValue(ec.SourceCodeText())) stmt.Imports = append(stmt.Imports, imports...) stmt.ExtraCodeSet = true } @@ -166,12 +161,8 @@ func (b *Builder) ExitCreateJavaScriptActionStatement(ctx *parser.CreateJavaScri } } - if dollarStr := ctx.DOLLAR_STRING(); dollarStr != nil { - code := dollarStr.GetText() - if len(code) >= 4 && strings.HasPrefix(code, "$$") && strings.HasSuffix(code, "$$") { - code = code[2 : len(code)-2] - } - stmt.JavaScriptCode = strings.TrimSpace(code) + if body := ctx.SourceCodeText(); body != nil { + stmt.JavaScriptCode = strings.TrimSpace(sourceCodeTextValue(body)) } if parent, ok := ctx.GetParent().(*parser.CreateStatementContext); ok { @@ -260,12 +251,26 @@ func extractJavaImports(code string) (body string, imports []string) { return strings.TrimSpace(strings.Join(bodyLines, "\n")), imports } -// extractJavaExtraCode returns the text of an `extra code $$ … $$` clause, its -// import lines taken out, with the blank lines at either end dropped and every -// other line as written — apart from the spaces between a `$$` and text on its -// own line, which are not indentation. +// sourceCodeTextValue is the text of a Java or JavaScript action's source +// literal: a `$$ … $$` without its delimiters, or a string literal read under +// the escape rule it was lexed with — the form describe writes for a text +// holding `$$` (ako/mxcli#1003). +func sourceCodeTextValue(c parser.ISourceCodeTextContext) string { + if sl := c.STRING_LITERAL(); sl != nil { + return unquoteStringLit(sl) + } + if ds := c.DOLLAR_STRING(); ds != nil { + return unquoteDollarString(ds.GetText()) + } + return "" +} + +// extractJavaExtraCode returns the text of an `extra code $$ … $$` clause +// (its delimiters already removed), its import lines taken out, with the +// blank lines at either end dropped and every other line as written — apart +// from the spaces between a `$$` and text on its own line, which are not +// indentation. func extractJavaExtraCode(dollar string) (code string, imports []string) { - dollar = strings.TrimSuffix(strings.TrimPrefix(dollar, "$$"), "$$") dollar = strings.TrimRight(strings.TrimLeft(dollar, " \t"), " \t") var lines []string for _, line := range strings.Split(strings.ReplaceAll(dollar, "\r\n", "\n"), "\n") { diff --git a/mdl/visitor/visitor_microflow_actions.go b/mdl/visitor/visitor_microflow_actions.go index 67e9f4349f..3742e81dc6 100644 --- a/mdl/visitor/visitor_microflow_actions.go +++ b/mdl/visitor/visitor_microflow_actions.go @@ -417,8 +417,9 @@ func buildCallMicroflowStatement(ctx parser.ICallMicroflowStatementContext) *ast stmt.Arguments = buildCallArgumentList(argList) } - // IN QUEUE Module.Name — runs the call on a task queue. + // IN QUEUE Module.Name [RETRY …] — runs the call on a task queue. stmt.Queue = buildQueueClause(callCtx.QueueClause()) + stmt.QueueRetry = buildQueueRetry(callCtx.QueueClause()) // Check for ON ERROR clause if errClause := callCtx.OnErrorClause(); errClause != nil { @@ -486,8 +487,9 @@ func buildCallJavaActionStatement(ctx parser.ICallJavaActionStatementContext) *a stmt.Arguments = buildCallArgumentList(argList) } - // IN QUEUE Module.Name — runs the call on a task queue. + // IN QUEUE Module.Name [RETRY …] — runs the call on a task queue. stmt.Queue = buildQueueClause(callCtx.QueueClause()) + stmt.QueueRetry = buildQueueRetry(callCtx.QueueClause()) // Check for ON ERROR clause if errClause := callCtx.OnErrorClause(); errClause != nil { @@ -730,7 +732,7 @@ func buildCallArgumentList(ctx parser.ICallArgumentListContext) []ast.CallArgume } if arg.NOTHING() != nil { // `Param = nothing`: a blank argument, stored as "" (ExitCallArgument - // refuses it outside call microflow / call nanoflow). + // refuses it outside call microflow / call nanoflow / call external action). ca.Value = &ast.BlankExpr{} } else if expr := arg.Expression(); expr != nil { value := buildSourceExpression(expr) diff --git a/mdl/visitor/visitor_microflow_statements.go b/mdl/visitor/visitor_microflow_statements.go index 77febb190b..cf8f7a7376 100644 --- a/mdl/visitor/visitor_microflow_statements.go +++ b/mdl/visitor/visitor_microflow_statements.go @@ -65,6 +65,8 @@ func buildMicroflowStatement(ctx parser.IMicroflowStatementContext) ast.Microflo stmt = buildDeleteObjectStatement(del) } else if rollback := mfCtx.RollbackStatement(); rollback != nil { stmt = buildRollbackStatement(rollback) + } else if ext := mfCtx.ExternalObjectStatement(); ext != nil { + stmt = buildExternalObjectStatement(ext) } else if retr := mfCtx.RetrieveStatement(); retr != nil { stmt = buildRetrieveStatement(retr) } else if ifStmt := mfCtx.IfStatement(); ifStmt != nil { @@ -1374,6 +1376,24 @@ func buildDeleteObjectStatement(ctx parser.IDeleteObjectStatementContext) *ast.D return stmt } +// buildExternalObjectStatement converts SEND/DELETE EXTERNAL OBJECT to an +// ExternalObjectStmt. +func buildExternalObjectStatement(ctx parser.IExternalObjectStatementContext) *ast.ExternalObjectStmt { + if ctx == nil { + return nil + } + c := ctx.(*parser.ExternalObjectStatementContext) + stmt := &ast.ExternalObjectStmt{Delete: c.DELETE() != nil} + if v := c.VARIABLE(); v != nil { + stmt.Variable = strings.TrimPrefix(v.GetText(), "$") + } + stmt.RefreshInClient = c.REFRESH() != nil + if errClause := c.OnErrorClause(); errClause != nil { + stmt.ErrorHandling = buildOnErrorClause(errClause) + } + return stmt +} + // buildRollbackStatement converts ROLLBACK statement context to RollbackStmt. func buildRollbackStatement(ctx parser.IRollbackStatementContext) *ast.RollbackStmt { if ctx == nil { diff --git a/mdl/visitor/visitor_page_v3.go b/mdl/visitor/visitor_page_v3.go index d4ff932c3e..a9d8a0efb4 100644 --- a/mdl/visitor/visitor_page_v3.go +++ b/mdl/visitor/visitor_page_v3.go @@ -1194,7 +1194,8 @@ func buildActionV3(ctx parser.IActionExprV3Context) *ast.ActionV3 { // snake-case one (R8, ako/mxcli#752); both build the same action, and // ExitActionExprV3 reports the old one. closePage := actCtx.ClosePageV3() != nil - if v := actCtx.VARIABLE(); v != nil && actCtx.OpenLinkV3() == nil { + isDelete := actCtx.DELETE_OBJECT() != nil || actCtx.DELETE() != nil + if v := actCtx.VARIABLE(); v != nil && actCtx.OpenLinkV3() == nil && !isDelete { // $handler — a fragment action parameter; resolved at expansion. // (open link $currentObject/Attr also carries a VARIABLE.) action.Type = "param" @@ -1212,9 +1213,12 @@ func buildActionV3(ctx parser.IActionExprV3Context) *ast.ActionV3 { } else if actCtx.CANCEL_CHANGES() != nil { action.Type = "cancel" action.ClosePage = closePage - } else if actCtx.DELETE_OBJECT() != nil || actCtx.DELETE() != nil { + } else if isDelete { action.Type = "delete" action.ClosePage = closePage + if v := actCtx.VARIABLE(); v != nil { + action.SourceVariable = strings.TrimPrefix(v.GetText(), "$") + } } else if closePage { action.Type = "close" } else if actCtx.CREATE_OBJECT() != nil || actCtx.CREATE() != nil { diff --git a/mdl/visitor/visitor_query.go b/mdl/visitor/visitor_query.go index c156a0657d..d39ddccf22 100644 --- a/mdl/visitor/visitor_query.go +++ b/mdl/visitor/visitor_query.go @@ -700,15 +700,15 @@ func (b *Builder) ExitShowStatement(ctx *parser.ShowStatementContext) { // SHOW FEATURES [IN area] | SHOW FEATURES FOR VERSION x.y | SHOW FEATURES ADDED SINCE x.y stmt := &ast.ShowFeaturesStmt{} if ctx.ADDED() != nil && ctx.SINCE() != nil { - if nl := ctx.NUMBER_LITERAL(); nl != nil { - stmt.AddedSince = nl.GetText() + if fv := ctx.FeatureVersion(); fv != nil { + stmt.AddedSince = fv.GetText() } } else if ctx.FOR() != nil && ctx.VERSION() != nil { - if nl := ctx.NUMBER_LITERAL(); nl != nil { - stmt.ForVersion = nl.GetText() + if fv := ctx.FeatureVersion(); fv != nil { + stmt.ForVersion = fv.GetText() } } else if ctx.IN() != nil { - if id := ctx.IDENTIFIER(); id != nil { + if id := ctx.IdentifierOrKeyword(); id != nil { stmt.InArea = id.GetText() } } diff --git a/mdl/visitor/visitor_queue_retry.go b/mdl/visitor/visitor_queue_retry.go new file mode 100644 index 0000000000..dc5a0d9752 --- /dev/null +++ b/mdl/visitor/visitor_queue_retry.go @@ -0,0 +1,159 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "fmt" + "sort" + "strconv" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/grammar/parser" +) + +// queueRetryProps lists, per retry kind, the properties the model stores +// (Queues$QueueFixedRetry / Queues$QueueExponentialRetry). Every one is +// required: the clause states the whole policy, so nothing is defaulted and a +// describe of it reads back the same. +var queueRetryProps = map[string][]string{ + "Fixed": {"Retries", "Interval", "IntervalType"}, + "Exponential": {"Retries", "InitialInterval", "MaximumInterval", "IntervalType"}, +} + +// queueIntervalTypes are Queues$IntervalType's values. +var queueIntervalTypes = []string{"Seconds", "Minutes", "Hours"} + +// queueRetryKind maps the word after RETRY to the kind, case-insensitively. +func queueRetryKind(word string) string { + switch strings.ToLower(word) { + case "fixed": + return "Fixed" + case "exponential": + return "Exponential" + } + return "" +} + +// buildQueueRetry reads the RETRY clause of an IN QUEUE clause, or nil. It is +// lenient — ExitQueueRetryClause reports what is wrong with it. +func buildQueueRetry(ctx parser.IQueueClauseContext) *ast.QueueRetry { + qc, ok := ctx.(*parser.QueueClauseContext) + if !ok || qc == nil || qc.QueueRetryClause() == nil { + return nil + } + rc := qc.QueueRetryClause().(*parser.QueueRetryClauseContext) + r := &ast.QueueRetry{} + if k := rc.IdentifierOrKeyword(); k != nil { + r.Kind = queueRetryKind(k.GetText()) + } + for _, p := range rc.AllQueueRetryProperty() { + pc := p.(*parser.QueueRetryPropertyContext) + name, value := queueRetryProperty(pc) + n, _ := strconv.Atoi(value) + switch strings.ToLower(name) { + case "retries": + r.Retries = n + case "interval": + r.Interval = n + case "initialinterval": + r.InitialInterval = n + case "maximuminterval": + r.MaximumInterval = n + case "intervaltype": + r.IntervalType = canonicalIntervalType(value) + } + } + return r +} + +func queueRetryProperty(pc *parser.QueueRetryPropertyContext) (name, value string) { + ids := pc.AllIdentifierOrKeyword() + if len(ids) > 0 { + name = ids[0].GetText() + } + if num := pc.NUMBER_LITERAL(); num != nil { + value = num.GetText() + } else if len(ids) > 1 { + value = ids[1].GetText() + } + return name, value +} + +func canonicalIntervalType(v string) string { + for _, t := range queueIntervalTypes { + if strings.EqualFold(t, v) { + return t + } + } + return "" +} + +// ExitQueueRetryClause refuses a retry clause that does not state a whole, +// valid policy: an unknown kind, a property the kind does not have, a missing +// or repeated one, a count that is not a whole number, or an unknown unit. +func (b *Builder) ExitQueueRetryClause(ctx *parser.QueueRetryClauseContext) { + tok := ctx.GetStart() + at := fmt.Sprintf("line %d:%d", tok.GetLine(), tok.GetColumn()) + word := "" + if k := ctx.IdentifierOrKeyword(); k != nil { + word = k.GetText() + } + kind := queueRetryKind(word) + if kind == "" { + b.addError(fmt.Errorf("%s retry %q: a queued call's retry is `retry fixed (…)` or `retry exponential (…)`. "+ + "See: mxcli syntax microflow call", at, word)) + return + } + want := queueRetryProps[kind] + seen := map[string]bool{} + for _, p := range ctx.AllQueueRetryProperty() { + pc := p.(*parser.QueueRetryPropertyContext) + name, value := queueRetryProperty(pc) + canon := "" + for _, w := range want { + if strings.EqualFold(w, name) { + canon = w + } + } + if canon == "" { + b.addError(fmt.Errorf("%s retry %s has no property %q; it takes %s", + at, strings.ToLower(kind), name, strings.Join(want, ", "))) + continue + } + if seen[canon] { + b.addError(fmt.Errorf("%s retry %s: %s is given twice", at, strings.ToLower(kind), canon)) + } + seen[canon] = true + if canon == "IntervalType" { + if canonicalIntervalType(value) == "" { + b.addError(fmt.Errorf("%s retry %s: IntervalType is %s, not %q", + at, strings.ToLower(kind), strings.Join(queueIntervalTypes, ", "), value)) + } + continue + } + n, err := strconv.Atoi(value) + switch { + case err != nil || n < 0: + b.addError(fmt.Errorf("%s retry %s: %s must be a whole number of 0 or more, not %q", + at, strings.ToLower(kind), canon, value)) + case canon == "Retries" && n < 1: + // Measured on mxbuild 11.12.6: CE7044 "Retries should be at least 1." + // on the call activity. An interval of 0 builds clean. + b.addError(fmt.Errorf("%s retry %s: Retries must be at least 1 — mxbuild rejects 0 "+ + "(CE7044 \"Retries should be at least 1.\"); leave out the retry clause for no retries", + at, strings.ToLower(kind))) + } + } + var missing []string + for _, w := range want { + if !seen[w] { + missing = append(missing, w) + } + } + if len(missing) > 0 { + sort.Strings(missing) + b.addError(fmt.Errorf("%s retry %s needs %s — the clause states the whole policy, so nothing is defaulted", + at, strings.ToLower(kind), strings.Join(missing, ", "))) + } +} diff --git a/mdl/visitor/visitor_queue_retry_test.go b/mdl/visitor/visitor_queue_retry_test.go new file mode 100644 index 0000000000..0e393ce4ad --- /dev/null +++ b/mdl/visitor/visitor_queue_retry_test.go @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// mendixlabs/mxcli#1729 (ako/mxcli#1931): "No spelling is accepted" — try1 was +// `line 4:81 mismatched input 'retry' expecting ';'`. +func TestQueueRetryClauseParses(t *testing.T) { + src := `create microflow Ops.ACT () +begin + call microflow Ops.POST_Message(Body = $Body) in queue Ops.SendQueue + retry exponential (Retries: 20, InitialInterval: 1, MaximumInterval: 60, IntervalType: Minutes); + call java action Ops.Refresh(Url = $Url) in queue Ops.SendQueue + retry fixed (Retries: 5, Interval: 10, IntervalType: minutes); + call microflow Ops.Plain() in queue Ops.SendQueue; +end;` + prog, errs := Build(src) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + body := prog.Statements[0].(*ast.CreateMicroflowStmt).Body + mf := body[0].(*ast.CallMicroflowStmt) + want := ast.QueueRetry{Kind: "Exponential", Retries: 20, InitialInterval: 1, MaximumInterval: 60, IntervalType: "Minutes"} + if mf.QueueRetry == nil || *mf.QueueRetry != want { + t.Errorf("exponential = %+v, want %+v", mf.QueueRetry, want) + } + ja := body[1].(*ast.CallJavaActionStmt) + wantF := ast.QueueRetry{Kind: "Fixed", Retries: 5, Interval: 10, IntervalType: "Minutes"} + if ja.QueueRetry == nil || *ja.QueueRetry != wantF { + t.Errorf("fixed = %+v, want %+v", ja.QueueRetry, wantF) + } + if body[2].(*ast.CallMicroflowStmt).QueueRetry != nil { + t.Error("a call without a retry clause must have no retry") + } +} + +func TestQueueRetryClauseRefusals(t *testing.T) { + for clause, want := range map[string]string{ + "retry linear (Retries: 3)": `retry "linear"`, + "retry fixed (Retries: 3, Interval: 10)": "needs IntervalType", + "retry fixed (Retries: 3, Interval: 10, IntervalType: Days)": "IntervalType is Seconds, Minutes, Hours", + "retry fixed (Retries: 3, InitialInterval: 1, IntervalType: Hours)": `no property "InitialInterval"`, + "retry fixed (Retries: 0, Interval: 10, IntervalType: Hours)": "CE7044", + } { + _, errs := Build("create microflow Ops.ACT ()\nbegin\n call microflow Ops.F() in queue Ops.Q " + clause + ";\nend;") + var got []string + for _, e := range errs { + got = append(got, e.Error()) + } + if !strings.Contains(strings.Join(got, "\n"), want) { + t.Errorf("%s: want an error containing %q, got %v", clause, want, got) + } + } +} diff --git a/modelsdk/gen/workflows/annotation_storagename_test.go b/modelsdk/gen/workflows/annotation_storagename_test.go new file mode 100644 index 0000000000..ba19cfcd98 --- /dev/null +++ b/modelsdk/gen/workflows/annotation_storagename_test.go @@ -0,0 +1,29 @@ +// SPDX-License-Identifier: Apache-2.0 + +package workflows + +import ( + "testing" + + "github.com/mendixlabs/mxcli/modelsdk/codec" +) + +// ako/mxcli#1784: the generated constructor named a workflow annotation +// Workflows$AnnotationActivity, the writer overrode it to Workflows$Annotation, +// and the decoder registered both. Measured, one workflow with a start and an +// activity annotation per project: Workflows$Annotation is 0 errors on +// mxbuild 9.24.30 and 11.12.6, and Workflows$AnnotationActivity is +// TypeCacheUnknownTypeException on both — no project can store it. So the +// constructor must produce the name Mendix loads, and the unknown name must not +// be decodable as if it were one. +func TestAnnotationStorageName(t *testing.T) { + if got := NewAnnotation().TypeName(); got != "Workflows$Annotation" { + t.Errorf("NewAnnotation().TypeName() = %q, want Workflows$Annotation (AnnotationActivity is TypeCacheUnknownTypeException)", got) + } + if _, ok := codec.DefaultRegistry.Lookup("Workflows$Annotation"); !ok { + t.Error("Workflows$Annotation must decode") + } + if _, ok := codec.DefaultRegistry.Lookup("Workflows$AnnotationActivity"); ok { + t.Error("Workflows$AnnotationActivity is not a Mendix storage name; registering it hides a wrong write") + } +} diff --git a/modelsdk/gen/workflows/types.go b/modelsdk/gen/workflows/types.go index 6b1240ab8f..83489d1962 100644 --- a/modelsdk/gen/workflows/types.go +++ b/modelsdk/gen/workflows/types.go @@ -6321,7 +6321,11 @@ func NewAllUserInput() *AllUserInput { // When a storage alias exists, init uses the STORAGE name (what Mendix uses in BSON). func initAnnotation() *Annotation { o := &Annotation{} - o.SetTypeName("Workflows$AnnotationActivity") + // STORAGE-NAME OVERRIDE (ako/mxcli#1784): the out-of-tree generator emitted + // Workflows$AnnotationActivity, which mxbuild 9.24.30 and 11.12.6 both + // refuse with TypeCacheUnknownTypeException; Workflows$Annotation is what + // Mendix loads (and what the writer always wrote). + o.SetTypeName("Workflows$Annotation") o.description = property.NewPrimitive[string]("Description", property.DecodeString) o.description.Bind(&o.Base, 0) o.SetProperties([]element.Property{o.description}) @@ -8305,13 +8309,10 @@ func init() { return initAllUserInput() }) codec.DefaultRegistry.Register("Workflows$Annotation", func() element.Element { - o := initAnnotation() - o.SetTypeName("Workflows$Annotation") - return o - }) - codec.DefaultRegistry.Register("Workflows$AnnotationActivity", func() element.Element { return initAnnotation() }) + // No Workflows$AnnotationActivity registration: it is not a Mendix storage + // name (TypeCacheUnknownTypeException on 9.24.30 and 11.12.6, #1784). codec.DefaultRegistry.Register("Workflows$BezierCurve", func() element.Element { return initBezierCurve() }) diff --git a/modelsdk/meta/system_module.go b/modelsdk/meta/system_module.go index b64843dc72..0881dd28ae 100644 --- a/modelsdk/meta/system_module.go +++ b/modelsdk/meta/system_module.go @@ -2,7 +2,10 @@ package meta -import "strings" +import ( + "fmt" + "strings" +) // System module constants — deterministic IDs for the virtual System module. const ( @@ -72,6 +75,36 @@ type SystemAssocDef struct { Child string // child entity name (without module prefix) Type string // "Reference" or "ReferenceSet" Owner string // "Default" or "Both" + + // Introduced and Removed bound the Mendix versions whose System module has + // this association, as "major.minor" (Removed is the first version WITHOUT + // it); empty means unbounded. The list otherwise describes the current + // platform, so only MEASURED differences are recorded here — read from the + // modeler assembly of each version (ako/mxcli#1810). + Introduced string + Removed string +} + +// InVersion reports whether the association exists in a project whose Mendix +// version satisfies atLeast(major, minor). A nil atLeast (version unknown) +// answers for the current platform: everything not Removed. +func (a SystemAssocDef) InVersion(atLeast func(major, minor int) bool) bool { + if atLeast == nil { + return a.Removed == "" + } + if a.Introduced != "" && !atLeast(parseMajorMinor(a.Introduced)) { + return false + } + if a.Removed != "" && atLeast(parseMajorMinor(a.Removed)) { + return false + } + return true +} + +func parseMajorMinor(v string) (int, int) { + var major, minor int + fmt.Sscanf(v, "%d.%d", &major, &minor) + return major, minor } // SystemEntityDef defines a System entity with name, persistability, and attributes. @@ -195,6 +228,12 @@ func ModelerSystemEntities() []SystemEntityDef { // in step, and the one that fell behind would put a dangling by-name reference // into a model — which is the failure this whole distinction exists to prevent. func ModelerSystemAssociations() []SystemAssocDef { + return ModelerSystemAssociationsIn(nil) +} + +// ModelerSystemAssociationsIn is ModelerSystemAssociations for one project +// version, given as its IsAtLeast; nil means the current platform. +func ModelerSystemAssociationsIn(atLeast func(major, minor int) bool) []SystemAssocDef { runtimeOnly := make(map[string]bool, len(SystemEntities)) for _, e := range SystemEntities { if e.RuntimeOnly { @@ -203,7 +242,7 @@ func ModelerSystemAssociations() []SystemAssocDef { } out := make([]SystemAssocDef, 0, len(SystemAssociations)) for _, a := range SystemAssociations { - if runtimeOnly[a.Parent] || runtimeOnly[a.Child] { + if runtimeOnly[a.Parent] || runtimeOnly[a.Child] || !a.InVersion(atLeast) { continue } out = append(out, a) @@ -554,7 +593,12 @@ var SystemAssociations = []SystemAssocDef{ {Name: "WorkflowUserTaskDefinition_WorkflowDefinition", Parent: "WorkflowUserTaskDefinition", Child: "WorkflowDefinition", Type: "Reference", Owner: "Default"}, {Name: "Workflow_WorkflowDefinition", Parent: "Workflow", Child: "WorkflowDefinition", Type: "Reference", Owner: "Default"}, {Name: "WorkflowUserTask_TargetUsers", Parent: "WorkflowUserTask", Child: "User", Type: "ReferenceSet", Owner: "Default"}, - {Name: "WorkflowUserTask_Assignees", Parent: "WorkflowUserTask", Child: "User", Type: "ReferenceSet", Owner: "Default"}, + // One association, renamed in 10.0 with the same GUID and ends: before it + // is WorkflowUserTask_Assignee, a Reference (one user). Read from the + // modeler assembly of 9.24.30 (Assignee), 10.0.0 and 11.12.6 (Assignees). + // A 9.x model naming Assignees is CE1613 "no longer exists" (ako/mxcli#1810). + {Name: "WorkflowUserTask_Assignees", Parent: "WorkflowUserTask", Child: "User", Type: "ReferenceSet", Owner: "Default", Introduced: "10.0"}, + {Name: "WorkflowUserTask_Assignee", Parent: "WorkflowUserTask", Child: "User", Type: "Reference", Owner: "Default", Removed: "10.0"}, {Name: "WorkflowUserTask_Workflow", Parent: "WorkflowUserTask", Child: "Workflow", Type: "Reference", Owner: "Default"}, {Name: "WorkflowUserTask_WorkflowUserTaskDefinition", Parent: "WorkflowUserTask", Child: "WorkflowUserTaskDefinition", Type: "Reference", Owner: "Default"}, {Name: "WorkflowJumpToDetails_Workflow", Parent: "WorkflowJumpToDetails", Child: "Workflow", Type: "Reference", Owner: "Default"}, diff --git a/sdk/microflows/microflows_actions.go b/sdk/microflows/microflows_actions.go index a071a98adc..6675e91fa2 100644 --- a/sdk/microflows/microflows_actions.go +++ b/sdk/microflows/microflows_actions.go @@ -84,6 +84,29 @@ type RollbackObjectAction struct { func (RollbackObjectAction) isMicroflowAction() {} +// SendExternalObjectAction is Studio Pro's "Send external object" +// (Microflows$SendExternalObject): it saves a changed external (OData) object +// through its consumed service, which a plain commit on one cannot do. +type SendExternalObjectAction struct { + model.BaseElement + Variable string `json:"variable"` + RefreshInClient bool `json:"refreshInClient"` + ErrorHandlingType ErrorHandlingType `json:"errorHandlingType,omitempty"` +} + +func (SendExternalObjectAction) isMicroflowAction() {} + +// DeleteExternalObjectAction is Studio Pro's "Delete external object" +// (Microflows$DeleteExternalObject). +type DeleteExternalObjectAction struct { + model.BaseElement + Variable string `json:"variable"` + RefreshInClient bool `json:"refreshInClient"` + ErrorHandlingType ErrorHandlingType `json:"errorHandlingType,omitempty"` +} + +func (DeleteExternalObjectAction) isMicroflowAction() {} + // MemberChange represents a change to a member (attribute or association). type MemberChange struct { model.BaseElement @@ -744,13 +767,35 @@ type MicroflowCall struct { // QueueSettings does (CE1613 when the named queue is missing). `generated/metamodel` // — the arbiter — has no top-level `Queue` on either call type, only this. // -// Retry is a Queues$QueueRetry (fixed or exponential) that MDL cannot author. -// It is carried as raw storage so a rewrite preserves whatever Studio Pro wrote -// rather than silently dropping it (guard-don't-drop, ADR-0005). +// Retry is the stored Queues$QueueRetry as raw storage, so the rewrite guard can +// tell one is there even when it cannot be decoded (guard-don't-drop, ADR-0005). +// RetryPolicy is the same value decoded — set by the reader for the two retry +// types Mendix has, and by the builder for `in queue … retry …` +// (mendixlabs/mxcli#1729). The writer writes RetryPolicy; nil writes no retry. type QueueSettings struct { model.BaseElement - Queue string `json:"queue,omitempty"` // Qualified name of the Queues$Queue - Retry any `json:"retry,omitempty"` // Opaque Queues$QueueRetry, preserved verbatim + Queue string `json:"queue,omitempty"` // Qualified name of the Queues$Queue + Retry any `json:"retry,omitempty"` // Stored Queues$QueueRetry, raw + RetryPolicy *QueueRetry `json:"retryPolicy,omitempty"` +} + +// Retry policy kinds: the two concrete Queues$QueueRetry types (Mendix 9.10+). +const ( + QueueRetryFixed = "Fixed" // Queues$QueueFixedRetry + QueueRetryExponential = "Exponential" // Queues$QueueExponentialRetry +) + +// QueueRetry is a queued call's retry policy. Fixed uses Interval; Exponential +// uses InitialInterval and MaximumInterval. IntervalType is the unit of all of +// them: Seconds, Minutes or Hours (Queues$IntervalType). +type QueueRetry struct { + model.BaseElement + Kind string `json:"kind"` + Retries int `json:"retries"` + Interval int `json:"interval,omitempty"` + InitialInterval int `json:"initialInterval,omitempty"` + MaximumInterval int `json:"maximumInterval,omitempty"` + IntervalType string `json:"intervalType,omitempty"` } // MicroflowCallParameterMapping maps a parameter to an argument. diff --git a/sdk/pages/pages_widgets_action.go b/sdk/pages/pages_widgets_action.go index abbedbdb1e..da2dc37a6c 100644 --- a/sdk/pages/pages_widgets_action.go +++ b/sdk/pages/pages_widgets_action.go @@ -372,6 +372,11 @@ type DeleteClientAction struct { model.BaseElement ActionExecution ClosePage bool `json:"closePage"` + // SourceVariable is the object deleted when it is not the context + // object: a widget's selection ({Widget: "grid1"}, a button in a grid's + // control bar) or a page/snippet parameter. nil is the context object, + // stored as a null SourceVariable (ako/mxcli#1932). + SourceVariable *WidgetVariable `json:"sourceVariable,omitempty"` } func (DeleteClientAction) isClientAction() {} diff --git a/sdk/versions/mendix-10.yaml b/sdk/versions/mendix-10.yaml index b4cdf3ed24..2519b0a365 100644 --- a/sdk/versions/mendix-10.yaml +++ b/sdk/versions/mendix-10.yaml @@ -94,6 +94,14 @@ features: min_version: "9.0.3" mdl: "$Found = FIND $List WHERE expression; $Kept = FILTER $List WHERE expression;" notes: "Microflows$FindByExpression and FilterByExpression introduced in 9.0.3 (mendixmodelsdk 4.117.0); `BY Member = value` writes Microflows$Find/Filter, which every version has. The write refuses each type below it (canon.UndeclaredTypeError); check -p names the statement first (ako/mxcli#1811)." + send_external_object: + min_version: "9.6.0" + mdl: "SEND EXTERNAL OBJECT $Obj [REFRESH];" + notes: "Microflows$SendExternalObject introduced in 9.6.0 (mendixmodelsdk; ako/mxcli#1939)." + delete_external_object: + min_version: "9.12.0" + mdl: "DELETE EXTERNAL OBJECT $Obj [REFRESH];" + notes: "Microflows$DeleteExternalObject introduced in 9.12.0 (mendixmodelsdk; ako/mxcli#1939)." call_external_action: min_version: "9.19.0" mdl: "CALL EXTERNAL ACTION Module.Service.Action ();" diff --git a/sdk/versions/mendix-11.yaml b/sdk/versions/mendix-11.yaml index 5c64a4c400..db79e1b984 100644 --- a/sdk/versions/mendix-11.yaml +++ b/sdk/versions/mendix-11.yaml @@ -97,6 +97,14 @@ features: min_version: "9.0.3" mdl: "$Found = FIND $List WHERE expression; $Kept = FILTER $List WHERE expression;" notes: "Microflows$FindByExpression and FilterByExpression introduced in 9.0.3 (mendixmodelsdk 4.117.0); `BY Member = value` writes Microflows$Find/Filter, which every version has. The write refuses each type below it (canon.UndeclaredTypeError); check -p names the statement first (ako/mxcli#1811)." + send_external_object: + min_version: "9.6.0" + mdl: "SEND EXTERNAL OBJECT $Obj [REFRESH];" + notes: "Microflows$SendExternalObject introduced in 9.6.0 (mendixmodelsdk; ako/mxcli#1939)." + delete_external_object: + min_version: "9.12.0" + mdl: "DELETE EXTERNAL OBJECT $Obj [REFRESH];" + notes: "Microflows$DeleteExternalObject introduced in 9.12.0 (mendixmodelsdk; ako/mxcli#1939)." call_external_action: min_version: "9.19.0" mdl: "CALL EXTERNAL ACTION Module.Service.Action ();" diff --git a/sdk/versions/mendix-9.yaml b/sdk/versions/mendix-9.yaml index d7be45525c..7b7c79a308 100644 --- a/sdk/versions/mendix-9.yaml +++ b/sdk/versions/mendix-9.yaml @@ -39,6 +39,14 @@ features: min_version: "9.0.3" mdl: "$Found = FIND $List WHERE expression; $Kept = FILTER $List WHERE expression;" notes: "Microflows$FindByExpression and FilterByExpression introduced in 9.0.3 (mendixmodelsdk 4.117.0); `BY Member = value` writes Microflows$Find/Filter, which every version has. The write refuses each type below it (canon.UndeclaredTypeError); check -p names the statement first (ako/mxcli#1811)." + send_external_object: + min_version: "9.6.0" + mdl: "SEND EXTERNAL OBJECT $Obj [REFRESH];" + notes: "Microflows$SendExternalObject introduced in 9.6.0 (mendixmodelsdk; ako/mxcli#1939)." + delete_external_object: + min_version: "9.12.0" + mdl: "DELETE EXTERNAL OBJECT $Obj [REFRESH];" + notes: "Microflows$DeleteExternalObject introduced in 9.12.0 (mendixmodelsdk; ako/mxcli#1939)." call_external_action: min_version: "9.19.0" mdl: "CALL EXTERNAL ACTION Module.Service.Action ();" diff --git a/sdk/versions/registry.go b/sdk/versions/registry.go index c668e4b59a..2962997277 100644 --- a/sdk/versions/registry.go +++ b/sdk/versions/registry.go @@ -230,6 +230,34 @@ func (r *Registry) IsAvailable(area, name string, projectVersion SemVer) bool { return false } +// Lookup returns the feature area/name, or false when the registry has none: +// the entry with the lowest MinVersion when the YAML files disagree, since that +// is the version below which none of it exists. `mxcli syntax` reads its +// "Requires Mendix" lines from here (ako/mxcli#1367). +func (r *Registry) Lookup(area, name string) (FeatureEntry, bool) { + entries, ok := r.index[area+"."+name] + if !ok || len(entries) == 0 { + return FeatureEntry{}, false + } + best := entries[0] + for _, e := range entries[1:] { + if best.MinVersion.AtLeast(e.MinVersion) && best.MinVersion != e.MinVersion { + best = e + } + } + return best, true +} + +// Keys returns every feature as "area/name", sorted. +func (r *Registry) Keys() []string { + out := make([]string, 0, len(r.index)) + for k := range r.index { + out = append(out, strings.Replace(k, ".", "/", 1)) + } + sort.Strings(out) + return out +} + // FeaturesForVersion returns all features and their availability status for a // given project version. Results are sorted alphabetically by display name. func (r *Registry) FeaturesForVersion(projectVersion SemVer) []FeatureStatus { diff --git a/sdk/widgets/definitions/datefilter.def.json b/sdk/widgets/definitions/datefilter.def.json index b454bae239..9c5f375a1c 100644 --- a/sdk/widgets/definitions/datefilter.def.json +++ b/sdk/widgets/definitions/datefilter.def.json @@ -3,10 +3,54 @@ "mdlName": "DATEFILTER", "templateFile": "datagrid-date-filter.json", "defaultEditable": "Always", - "propertyMappings": [ - {"propertyKey": "attrChoice", "value": "auto", "operation": "primitive"}, - {"propertyKey": "attributes", "source": "Attributes", "operation": "attributeObjects"}, - {"propertyKey": "defaultFilter", "source": "FilterType", "operation": "primitive"}, - {"propertyKey": "onChange", "source": "OnChange", "operation": "action"} + "modes": [ + { + "name": "attributes", + "condition": "hasAttributes", + "description": "Custom attributes — `attributes: [A, …]` names the attributes the filter compares, stored on the `attributes` list, which the widget's editorConfig shows only when attrChoice is \"linked\" (Custom). The builder's SetAttributeObjects writes \"linked\" whenever it fills the list; the mode states it so check reads the configuration exec stores (ako/mxcli#1684 — the constant \"auto\" made MDL-WIDGET10 call the list hidden). Same shape as the drop-down filter's `attribute` mode (mendixlabs/mxcli#1519).", + "propertyMappings": [ + { + "propertyKey": "attrChoice", + "value": "linked", + "operation": "primitive" + }, + { + "propertyKey": "attributes", + "source": "Attributes", + "operation": "attributeObjects" + }, + { + "propertyKey": "defaultFilter", + "source": "FilterType", + "operation": "primitive" + }, + { + "propertyKey": "onChange", + "source": "OnChange", + "operation": "action" + } + ] + }, + { + "name": "default", + "description": "Auto — no attributes given: the filter takes the attribute of the Data grid 2 column it sits in (attrChoice \"auto\"), so it only works inside a column.", + "propertyMappings": [ + { + "propertyKey": "attrChoice", + "value": "auto", + "operation": "primitive" + }, + { + "propertyKey": "defaultFilter", + "source": "FilterType", + "operation": "primitive" + }, + { + "propertyKey": "onChange", + "source": "OnChange", + "operation": "action" + } + ] + } ] } diff --git a/sdk/widgets/definitions/numberfilter.def.json b/sdk/widgets/definitions/numberfilter.def.json index fbf1f7519d..34829cb0fe 100644 --- a/sdk/widgets/definitions/numberfilter.def.json +++ b/sdk/widgets/definitions/numberfilter.def.json @@ -3,10 +3,54 @@ "mdlName": "NUMBERFILTER", "templateFile": "datagrid-number-filter.json", "defaultEditable": "Always", - "propertyMappings": [ - {"propertyKey": "attrChoice", "value": "auto", "operation": "primitive"}, - {"propertyKey": "attributes", "source": "Attributes", "operation": "attributeObjects"}, - {"propertyKey": "defaultFilter", "source": "FilterType", "operation": "primitive"}, - {"propertyKey": "onChange", "source": "OnChange", "operation": "action"} + "modes": [ + { + "name": "attributes", + "condition": "hasAttributes", + "description": "Custom attributes — `attributes: [A, …]` names the attributes the filter compares, stored on the `attributes` list, which the widget's editorConfig shows only when attrChoice is \"linked\" (Custom). The builder's SetAttributeObjects writes \"linked\" whenever it fills the list; the mode states it so check reads the configuration exec stores (ako/mxcli#1684 — the constant \"auto\" made MDL-WIDGET10 call the list hidden). Same shape as the drop-down filter's `attribute` mode (mendixlabs/mxcli#1519).", + "propertyMappings": [ + { + "propertyKey": "attrChoice", + "value": "linked", + "operation": "primitive" + }, + { + "propertyKey": "attributes", + "source": "Attributes", + "operation": "attributeObjects" + }, + { + "propertyKey": "defaultFilter", + "source": "FilterType", + "operation": "primitive" + }, + { + "propertyKey": "onChange", + "source": "OnChange", + "operation": "action" + } + ] + }, + { + "name": "default", + "description": "Auto — no attributes given: the filter takes the attribute of the Data grid 2 column it sits in (attrChoice \"auto\"), so it only works inside a column.", + "propertyMappings": [ + { + "propertyKey": "attrChoice", + "value": "auto", + "operation": "primitive" + }, + { + "propertyKey": "defaultFilter", + "source": "FilterType", + "operation": "primitive" + }, + { + "propertyKey": "onChange", + "source": "OnChange", + "operation": "action" + } + ] + } ] } diff --git a/sdk/widgets/definitions/textfilter.def.json b/sdk/widgets/definitions/textfilter.def.json index 0c1d1ece12..3425de7ffc 100644 --- a/sdk/widgets/definitions/textfilter.def.json +++ b/sdk/widgets/definitions/textfilter.def.json @@ -3,10 +3,54 @@ "mdlName": "TEXTFILTER", "templateFile": "datagrid-text-filter.json", "defaultEditable": "Always", - "propertyMappings": [ - {"propertyKey": "attrChoice", "value": "auto", "operation": "primitive"}, - {"propertyKey": "attributes", "source": "Attributes", "operation": "attributeObjects"}, - {"propertyKey": "defaultFilter", "source": "FilterType", "operation": "primitive"}, - {"propertyKey": "onChange", "source": "OnChange", "operation": "action"} + "modes": [ + { + "name": "attributes", + "condition": "hasAttributes", + "description": "Custom attributes — `attributes: [A, …]` names the attributes the filter compares, stored on the `attributes` list, which the widget's editorConfig shows only when attrChoice is \"linked\" (Custom). The builder's SetAttributeObjects writes \"linked\" whenever it fills the list; the mode states it so check reads the configuration exec stores (ako/mxcli#1684 — the constant \"auto\" made MDL-WIDGET10 call the list hidden). Same shape as the drop-down filter's `attribute` mode (mendixlabs/mxcli#1519).", + "propertyMappings": [ + { + "propertyKey": "attrChoice", + "value": "linked", + "operation": "primitive" + }, + { + "propertyKey": "attributes", + "source": "Attributes", + "operation": "attributeObjects" + }, + { + "propertyKey": "defaultFilter", + "source": "FilterType", + "operation": "primitive" + }, + { + "propertyKey": "onChange", + "source": "OnChange", + "operation": "action" + } + ] + }, + { + "name": "default", + "description": "Auto — no attributes given: the filter takes the attribute of the Data grid 2 column it sits in (attrChoice \"auto\"), so it only works inside a column.", + "propertyMappings": [ + { + "propertyKey": "attrChoice", + "value": "auto", + "operation": "primitive" + }, + { + "propertyKey": "defaultFilter", + "source": "FilterType", + "operation": "primitive" + }, + { + "propertyKey": "onChange", + "source": "OnChange", + "operation": "action" + } + ] + } ] }