Repository navigation
New non-shipping graph store - #6053
William Wong (compulim) wants to merge 31 commits into
Conversation
…ing, and append performance work (#6034) * Store: transactional edit, upsert(id, updater), delete with reference cleanup, tests Co-authored-by: compulim <[email protected]> * Store: track forward/backward node references, test node reference validation Co-authored-by: compulim <[email protected]> * Store: delete returns void, deep freeze nodes, verify referenced blank nodes on commit Co-authored-by: compulim <[email protected]> * Store: immutable Set via Proxy, require all referenced nodes to exist on commit Co-authored-by: compulim <[email protected]> * Store: auto-fix hasPart/isPartOf on commit, disallow self part, unique position per part Co-authored-by: compulim <[email protected]> * Store: make hasPart/isPartOf auto-fix opt-in via edit init, always validate inverse relationship Co-authored-by: compulim <[email protected]> * Store: split assertions/auto-relationship/deepFreeze into modules, notify listeners with affected node IDs Co-authored-by: compulim <[email protected]> * Rename to Source and Controller * Store: cache part IDs in a hidden Symbol property on each node via createNode() Co-authored-by: compulim <[email protected]> * Store: lazily build part IDs cache on first getPartIds() call, validated by reference equality Co-authored-by: compulim <[email protected]> * Revert lazy part IDs cache, build cache up front in createNode() Co-authored-by: compulim <[email protected]> * Store: replace Proxy-based readonly Set with a class wrapping a private Set Co-authored-by: compulim <[email protected]> * Store: move deepFreeze into createNode, attach part IDs before the single top-level freeze Co-authored-by: compulim <[email protected]> * Store: batch part membership auto-fix right before commit, document and test inverse assertion Co-authored-by: compulim <[email protected]> * Store: replace autoRelationship init option with transaction.autoFixRelationship() Co-authored-by: compulim <[email protected]> * Store: check inverse part relationship on changed links only, add createTrustedNode for auto-fix Co-authored-by: compulim <[email protected]> * Store: combine createNode and createTrustedNode into editNode Co-authored-by: compulim <[email protected]> * Store: revert editNode and createTrustedNode, auto-fix uses createNode again Co-authored-by: compulim <[email protected]> * Store: use mutate/commit overlays instead of cloning state and reference maps Co-authored-by: compulim <[email protected]> * Store: derive hasPart from isPartOf, mirror parent hasPart to children, remove auto-fix Co-authored-by: compulim <[email protected]> * Store: add getSortedChildren sorted by dateCreated Co-authored-by: compulim <[email protected]> * Store: sorted children as versioned skip list, getSortedChildren returns Iterable Co-authored-by: compulim <[email protected]> * Store: getSortedChildren takes parent ID Co-authored-by: compulim <[email protected]> * Store: getSortedChildren requires init with sortBy Co-authored-by: compulim <[email protected]> * Store: allow duplicate position in the same part Co-authored-by: compulim <[email protected]> * Store: skip list follows mutate()/commit() pattern, remove position from Node Co-authored-by: compulim <[email protected]> * Store: store hasPart/isPartOf as ReadonlyNodeSet backed by IDs, getPartIds() is O(1) Co-authored-by: compulim <[email protected]> * Store: createNode() reuses ReadonlyNodeSet and copies IDs via iterator Co-authored-by: compulim <[email protected]> * Store: forbid nested edit(), move data structures to src/structure, rename getSortedChildren to getSortedParts Co-authored-by: compulim <[email protected]> * Store: JSON value follows JSON.stringify(), honors toJSON() and rejects recursive values Co-authored-by: compulim <[email protected]> * Store: JSON value allows plain objects/arrays only, no toJSON(), still rejects recursive Co-authored-by: compulim <[email protected]> * Store: dateCreated as ISO timestamp string, sorted as Date Co-authored-by: compulim <[email protected]> * Store: cache parsed dateCreated on node to avoid parsing on every comparison Co-authored-by: compulim <[email protected]> * Store: brand nodes from createNode() as InternalNode via valibot brand Co-authored-by: compulim <[email protected]> * Update * Store: getSortedParts() supports order, rename sortBy to by Co-authored-by: compulim <[email protected]> * Update JSON literal type to @JSON * Clarify IRI * Add performance characteristics * Typo * Typo * Store: seal NodeReference with Exact, reuse deeply frozen values in createNode() Co-authored-by: compulim <[email protected]> * Store: add npm run benchmark with aligned Markdown table of average per node Co-authored-by: compulim <[email protected]> * Store: move deepFreeze to own file, transpose benchmark table Co-authored-by: compulim <[email protected]> * Store: add createCachedIterable (not yet used, slower than array in deepFreeze) Co-authored-by: compulim <[email protected]> * Store: single-pass deepFreeze, remove createCachedIterable, merge ExactNodeReference into NodeReference Co-authored-by: compulim <[email protected]> * Store: skip dangerous keys (__proto__, constructor, prototype) when deep freezing Co-authored-by: compulim <[email protected]> * Store: VersionedStorage mutate()/commit() API, recursive deepFreeze Co-authored-by: compulim <[email protected]> * More * Store: document why VersionedStorage uses undo log Co-authored-by: compulim <[email protected]> * More * Plain object * Store: add clone option to deepFreeze and upsert Co-authored-by: compulim <[email protected]> * Store: benchmark clone vs no clone, faster extra-key check Co-authored-by: compulim <[email protected]> * Revert "Store: benchmark clone vs no clone, faster extra-key check" This reverts commit db766b8. Co-authored-by: compulim <[email protected]> * Revert "Store: add clone option to deepFreeze and upsert" This reverts commit e699e36. Co-authored-by: compulim <[email protected]> * Store: validate JSON value by a single recursive check() Co-authored-by: compulim <[email protected]> * Plan: JSON literal matches JSON.parse(JSON.stringify()) Co-authored-by: compulim <[email protected]> * Store: JSON literal behaves as JSON.parse(JSON.stringify()) Co-authored-by: compulim <[email protected]> * Store: use propertyValueSchema output for rest properties of node Co-authored-by: compulim <[email protected]> * Store: consume readonly structures via `new ReadonlyMap()` etc. instead of `createReadonlyX()` Co-authored-by: compulim <[email protected]> * Store: rename ReadonlyX structures to FrozenX to avoid clashing with built-in types Co-authored-by: compulim <[email protected]> * Move DLJS to different entry, fix default entry * Plan: drop exported isFrozenMap Co-authored-by: compulim <[email protected]> * Structure entry * Store: add adopt, pending, and delete parent benchmarks Co-authored-by: compulim <[email protected]> * Store: keep node changes in ReadWriteNode drafts, build each node once per transaction Co-authored-by: compulim <[email protected]> * Store: createNode returns unfrozen node, freezeNode freezes in place Co-authored-by: compulim <[email protected]> * Store: benchmark unfrozen createNode Co-authored-by: compulim <[email protected]> * Store: move pending parts check to assertNoPendingParts Co-authored-by: compulim <[email protected]> * Store: type rest schema of Node via BaseSchema cast instead of transform Co-authored-by: compulim <[email protected]> * Store: validate identifier with a single custom() schema Co-authored-by: compulim <[email protected]> * Store: revert identifierSchema to union() and pipe() for readability Co-authored-by: compulim <[email protected]> * Store: brand InternalNode with a non-exported symbol Co-authored-by: compulim <[email protected]> * Store: copy skip list links by spread, slice() of frozen arrays is slow Co-authored-by: compulim <[email protected]> * Apply remaining changes Co-authored-by: compulim <[email protected]> * Store: reuse overlay changes as undo log in VersionedStorage commit Co-authored-by: compulim <[email protected]> * Add design choice * Add clarification --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: compulim <[email protected]> Co-authored-by: William Wong <[email protected]>
…#6038) * Plan: fix Node type accepting invalid property values Co-authored-by: compulim <[email protected]> * Store: reject non-array property values at compile-time in upsert() Co-authored-by: compulim <[email protected]> --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: compulim <[email protected]>
…r()` (#6041) * Store: add alternate `identifier` property and O(1) `queryByIdentifier()` Co-authored-by: compulim <[email protected]> * Add `identifier` property to custom parts --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: compulim <[email protected]> Co-authored-by: William Wong <[email protected]>
82ebfc8 to
1eb176a
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Node operations fail in supported browsers, async transactions can partially commit, and the new test suites fail the existing Jest job.
16 open issues
Run node:test suites explicitly in CI · New Reject async transactors before committing · New Avoid unsupported Set iterator helpers in target browsers · New Avoid unsupported iterator map helpers in target browsers · New Add workspace precommit and typecheck scripts · New Notify all subscribers when a listener throws · New Use intrinsic methods to unwrap boxed primitives · New Honor toJSON on callable objects · New Capture array length before serializing elements · New Reuse only validated node membership properties · New Use Object.is to preserve normalized zero values · New Implement Set algebra without unsupported native methods · New Use ID-based equality for node-set operations · New Preserve insertion order when reconstructing deleted keys · New Prevent snapshot iterators from observing later commits · New Move reinserted keys to the end of overlay collections · New
What changed in this PR
Adds a standalone graph-store package as groundwork for a new Web Chat state model. The store is not yet connected to the existing runtime; its Direct Line adapter is a placeholder.
Changes:
- Adds transactional graph edits, immutable snapshots, identifier lookup, sorted child traversal, and subscriptions.
- Introduces persistent collections, node validation, JSON normalization, tests, and benchmarks.
- Registers the workspace with separate store, collection, and Direct Line exports.
| File | Description |
|---|---|
packages/store/tsup.config.ts |
Configures ESM and CommonJS entry points. |
packages/store/src/tsconfig.json |
Enables TypeScript-extension imports. |
packages/store/src/structure/VersionedStorage.ts |
Implements versioned storage and writable overlays. |
packages/store/src/structure/VersionedStorage.test.ts |
Tests commits, historical reads, and branching. |
packages/store/src/structure/SkipListView.ts |
Provides skip-list traversal methods. |
packages/store/src/structure/SetView.ts |
Provides shared Set read operations. |
packages/store/src/structure/MapView.ts |
Provides shared Map read operations. |
packages/store/src/structure/FrozenSkipList.ts |
Implements persistent sorted collections. |
packages/store/src/structure/FrozenSkipList.test.ts |
Tests ordering, mutation, and historical traversal. |
packages/store/src/structure/FrozenSet.ts |
Implements immutable sets and writable overlays. |
packages/store/src/structure/FrozenSet.test.ts |
Tests set compatibility and version isolation. |
packages/store/src/structure/FrozenNodeSet.ts |
Represents node references using stored IDs. |
packages/store/src/structure/FrozenNodeSet.test.ts |
Tests reference membership and immutability. |
packages/store/src/structure/FrozenMap.ts |
Implements immutable maps and writable overlays. |
packages/store/src/structure/FrozenMap.test.ts |
Tests map compatibility and version isolation. |
packages/store/src/structure.ts |
Exports public map and set APIs. |
packages/store/src/Store.ts |
Implements graph transactions, queries, and subscriptions. |
packages/store/src/private/UnfrozenNode.ts |
Brands validated, writable node objects. |
packages/store/src/private/sortedChildIds.ts |
Maintains cached child ordering. |
packages/store/src/private/ReadWriteNode.ts |
Implements copy-on-write node drafts. |
packages/store/src/private/ReadWriteNode.test.ts |
Tests draft changes and reference removal. |
packages/store/src/private/InternalNode.ts |
Brands nodes stored in the graph. |
packages/store/src/private/identifierIndex.ts |
Maintains alternate-identifier lookup and uniqueness. |
packages/store/src/private/getPartIds.ts |
Extracts membership IDs efficiently. |
packages/store/src/private/freezeNode.ts |
Finalizes nodes with membership and ordering metadata. |
packages/store/src/private/freezeNode.test.ts |
Tests node finalization and hidden metadata. |
packages/store/src/private/deepFreeze.ts |
Copies and freezes validated values. |
packages/store/src/private/deepFreeze.test.ts |
Tests immutability, recursion, and dangerous-key removal. |
packages/store/src/private/dateCreatedTime.ts |
Caches parsed timestamps for sorting. |
packages/store/src/private/createNode.ts |
Validates and prepares graph nodes. |
packages/store/src/private/createNode.test.ts |
Tests validation, reuse, and JSON normalization. |
packages/store/src/OPINIONS.md |
Documents graph semantics and design choices. |
packages/store/src/node/PropertyValue.ts |
Defines supported property-value arrays. |
packages/store/src/node/NodeReference.ts |
Defines exact node-reference types and validation. |
packages/store/src/node/NodeReference.test.ts |
Tests reference validation and excess-property rejection. |
packages/store/src/node/NodeIdentifier.ts |
Defines supported node identifiers. |
packages/store/src/node/NodeIdentifier.test.ts |
Tests valid and invalid identifiers. |
packages/store/src/node/Node.ts |
Defines node schemas and read/write types. |
packages/store/src/node/JSONValueSchema.test.ts |
Tests JSON serialization-compatible normalization. |
packages/store/src/node/JSONValue.ts |
Normalizes values into JSON-compatible data. |
packages/store/src/node/JSONLiteral.ts |
Defines JSON-LD literal validation. |
packages/store/src/node/BlankNodeIdentifier.ts |
Defines blank-node identifier validation. |
packages/store/src/index.ts |
Exports the public store API. |
packages/store/src/index.test.ts |
Adds an entry-point import smoke test. |
packages/store/src/directLineJS/createDirectLineJSStore.ts |
Adds a placeholder adapter with sample data. |
packages/store/src/directLineJS/createDirectLineJSStore.test.ts |
Tests placeholder store initialization. |
packages/store/src/directLineJS.ts |
Exports the Direct Line adapter. |
packages/store/src/benchmark.ts |
Measures graph operations and memory usage. |
packages/store/src/assertion/assertUniqueIdentifier.ts |
Enforces identifier uniqueness. |
packages/store/src/assertion/assertReferencedNodesInGraph.ts |
Checks referenced-node existence. |
packages/store/src/assertion/assertNotSelfPart.ts |
Rejects self-membership relationships. |
packages/store/src/assertion/assertNoPendingParts.ts |
Rejects unresolved child memberships. |
packages/store/src/assertion/assertIdentifierNotId.ts |
Rejects primary IDs used as alternate identifiers. |
packages/store/package.json |
Declares exports, dependencies, and workspace scripts. |
packages/store/.gitignore |
Excludes generated and local workspace files. |
package.json |
Registers the store workspace. |
Review effort: Balanced
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
…de, FrozenNodeSet (#6059) * Add failing tests for outstanding review comments in #6053 Co-authored-by: copilot-swe-agent[bot] <[email protected]> * Fix review issues: createNode accessor, SetView polyfill-free Set methods, VersionedStorage ordering and snapshot-safe iteration Co-authored-by: copilot-swe-agent[bot] <[email protected]> * Simplify VersionedStorage: copy on move instead of tracking positions Co-authored-by: copilot-swe-agent[bot] <[email protected]> * Document why SetView does not use core-js-pure Co-authored-by: copilot-swe-agent[bot] <[email protected]> --------- Co-authored-by: compulim <[email protected]> Co-authored-by: copilot-swe-agent[bot] <[email protected]>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved issues affect collection semantics, JSON serialization, timestamp ordering, and property access, alongside unnecessary iterator allocation.
8 open issues
Detect boxed primitives by their internal slots · New Avoid rereading properties after validation · New Preserve sub-millisecond timestamp precision · New Copy null-prototype objects when normalizing · New Match live Map and Set iteration semantics · New Reject negative set-like sizes · New Make entries() iterate lazily · New Normalize set-like operands by node ID · New
16 resolved since last review
Avoid unsupported iterator map helpers in target browsers Avoid unsupported Set iterator helpers in target browsers Reject async transactors before committing Run node:test suites explicitly in CI Move reinserted keys to the end of overlay collections Prevent snapshot iterators from observing later commits Preserve insertion order when reconstructing deleted keys Use ID-based equality for node-set operations Implement Set algebra without unsupported native methods Use Object.is to preserve normalized zero values Reuse only validated node membership properties Capture array length before serializing elements Honor toJSON on callable objects Use intrinsic methods to unwrap boxed primitives Notify all subscribers when a listener throws Add workspace precommit and typecheck scripts
Review effort: Balanced
Implement generator function for entries method to yield key-value pairs. Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
* Add failing tests for unresolved review comments on #6053 Co-authored-by: copilot-swe-agent[bot] <[email protected]> * Show JSON.parse(JSON.stringify()) behavior side-by-side in failing tests Co-authored-by: copilot-swe-agent[bot] <[email protected]> * Show native Set behavior side-by-side in SetView invalid size tests Co-authored-by: copilot-swe-agent[bot] <[email protected]> * Fix review comments from #6053 so all failing tests pass Co-authored-by: copilot-swe-agent[bot] <[email protected]> * Drop sub-millisecond precision of dateCreated Co-authored-by: copilot-swe-agent[bot] <[email protected]> * Add TIL test * Remove read-once tests, add tests for non-enumerable/accessor/inherited properties in createNode Co-authored-by: copilot-swe-agent[bot] <[email protected]> * Add enum/non-enum and getter/value * createNode only reads own enumerable properties, like JSON.stringify() Co-authored-by: copilot-swe-agent[bot] <[email protected]> --------- Co-authored-by: compulim <[email protected]> Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: William Wong <[email protected]>
There was a problem hiding this comment.
🟡 Changes recommended
FrozenSet subclasses can let committed snapshots change outside transactions, and unresolved iterator, set-equality, and snapshot-mutation defects remain.
2 open issues
3 previously missed issues
In code that hasn't changed since last review

Copy FrozenSet subclasses when deep-freezing

packages/store/src/structure/FrozenSet.ts:122
isFrozenSet() also accepts subclasses whose read methods depend on mutable external state. Valibot's default-branch set parser accepts and iterates them, so deepFreeze() can reuse one for @type or identifier and let a committed snapshot change outside a transaction (this package pins 1.2.0). Only reuse instances with the exact FrozenSet.prototype; copy subclasses instead.

Freeze sorted entries before committing

packages/store/src/structure/FrozenSkipList.ts:110
commit() freezes each entry's arrays but leaves the entry object writable. Snapshot symbols expose the sorted list, and its TypeScript-protected read() is callable at runtime, so replacing entry.next or entry.previous corrupts later getSortedParts() results. Freeze each entry as well as its arrays before committing.

Use node-ID equality consistently in set operations

packages/store/src/structure/SetView.ts:131
parts.intersection(new Set(parts)) is empty for a nonempty FrozenNodeSet, because each iteration creates new reference objects and this branch compares them by identity. Adding an unrelated ID to parts switches branches and finds the matching IDs, so the result depends on relative set sizes. Apply node-ID equality consistently here and in the other.has() paths of difference(), isDisjointFrom(), isSubsetOf(), and symmetricDifference().
8 resolved since last review
Normalize set-like operands by node ID Make entries() iterate lazily Reject negative set-like sizes Match live Map and Set iteration semantics Copy null-prototype objects when normalizing Preserve sub-millisecond timestamp precision Avoid rereading properties after validation Detect boxed primitives by their internal slots
Review effort: Balanced
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
* Add failing tests for unresolved review comments on #6053 Co-authored-by: copilot-swe-agent[bot] <[email protected]> * Fix FrozenNodeSet equality key collision and overlay iteration across commit Co-authored-by: copilot-swe-agent[bot] <[email protected]> --------- Co-authored-by: compulim <[email protected]> Co-authored-by: copilot-swe-agent[bot] <[email protected]>
| * Returns the IDs in O(1) without copying if the property is a `FrozenNodeSet` (i.e. the node is created by | ||
| * `createNode()`), otherwise, builds them in O(k). | ||
| */ | ||
| function getPartIds(node: Pick<Node, PartProperty> | undefined, property: PartProperty): ReadonlySet<Identifier> { |
There was a problem hiding this comment.
Maybe we could consider attaching hidden symbol that is memoized string[], look at how we done it in dateCreatedTime.ts.
| export { default as FrozenMap, type ReadWriteMap } from './structure/FrozenMap.ts'; | ||
| export { default as FrozenSet, type ReadWriteSet } from './structure/FrozenSet.ts'; |
There was a problem hiding this comment.
NTM: Do we want to export them now or later?
Changelog Entry
Added
Description
Preparing a new graph store to ship in Web Chat. Currently not wired to Web Chat until it is ready.
Design
See OPINIONS.md for design.
Specific Changes
botframework-webchat-storepackage, which will be folded intocoreorapilater.changelog/<PR#>-<slug>.mdReview Checklist
Accessibility reviewed (tab order, content readability, alt text, color contrast)Browser and platform compatibilities reviewedCSS styles reviewed (minimal rules, noz-index)Internationalization reviewed (strings, unit formatting)package.jsonandpackage-lock.jsonreviewedSecurity reviewed (no data URIs, check for nonce leak)