Skip to content

New non-shipping graph store - #6053

Open
William Wong (compulim) wants to merge 31 commits into
mainfrom
feat-new-store
Open

William Wong (compulim) wants to merge 31 commits into
mainfrom
feat-new-store

Conversation

@compulim

@compulim William Wong (compulim) commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Changelog Entry

Added

  • Added new JSON-LD based graph backend store, in PR #6053, by @compulim

Description

Preparing a new graph store to ship in Web Chat. Currently not wired to Web Chat until it is ready.

Design

See OPINIONS.md for design.

Specific Changes

  • New botframework-webchat-store package, which will be folded into core or api later
  • I have added tests and executed them locally
  • I have added .changelog/<PR#>-<slug>.md
  • I have updated documentation

Review Checklist

This section is for contributors to review your work.

  • Accessibility reviewed (tab order, content readability, alt text, color contrast)
  • Browser and platform compatibilities reviewed
  • CSS styles reviewed (minimal rules, no z-index)
  • Documents reviewed (docs, samples, live demo)
  • Internationalization reviewed (strings, unit formatting)
  • package.json and package-lock.json reviewed
  • Security reviewed (no data URIs, check for nonce leak)
  • Tests reviewed (coverage, legitimacy)

…ing, and append performance work (#6034)

* Store: transactional edit, upsert(id, updater), delete with reference cleanup, tests

Co-authored-by: compulim <[email protected]>

* Store: track forward/backward node references, test node reference validation

Co-authored-by: compulim <[email protected]>

* Store: delete returns void, deep freeze nodes, verify referenced blank nodes on commit

Co-authored-by: compulim <[email protected]>

* Store: immutable Set via Proxy, require all referenced nodes to exist on commit

Co-authored-by: compulim <[email protected]>

* Store: auto-fix hasPart/isPartOf on commit, disallow self part, unique position per part

Co-authored-by: compulim <[email protected]>

* Store: make hasPart/isPartOf auto-fix opt-in via edit init, always validate inverse relationship

Co-authored-by: compulim <[email protected]>

* Store: split assertions/auto-relationship/deepFreeze into modules, notify listeners with affected node IDs

Co-authored-by: compulim <[email protected]>

* Rename to Source and Controller

* Store: cache part IDs in a hidden Symbol property on each node via createNode()

Co-authored-by: compulim <[email protected]>

* Store: lazily build part IDs cache on first getPartIds() call, validated by reference equality

Co-authored-by: compulim <[email protected]>

* Revert lazy part IDs cache, build cache up front in createNode()

Co-authored-by: compulim <[email protected]>

* Store: replace Proxy-based readonly Set with a class wrapping a private Set

Co-authored-by: compulim <[email protected]>

* Store: move deepFreeze into createNode, attach part IDs before the single top-level freeze

Co-authored-by: compulim <[email protected]>

* Store: batch part membership auto-fix right before commit, document and test inverse assertion

Co-authored-by: compulim <[email protected]>

* Store: replace autoRelationship init option with transaction.autoFixRelationship()

Co-authored-by: compulim <[email protected]>

* Store: check inverse part relationship on changed links only, add createTrustedNode for auto-fix

Co-authored-by: compulim <[email protected]>

* Store: combine createNode and createTrustedNode into editNode

Co-authored-by: compulim <[email protected]>

* Store: revert editNode and createTrustedNode, auto-fix uses createNode again

Co-authored-by: compulim <[email protected]>

* Store: use mutate/commit overlays instead of cloning state and reference maps

Co-authored-by: compulim <[email protected]>

* Store: derive hasPart from isPartOf, mirror parent hasPart to children, remove auto-fix

Co-authored-by: compulim <[email protected]>

* Store: add getSortedChildren sorted by dateCreated

Co-authored-by: compulim <[email protected]>

* Store: sorted children as versioned skip list, getSortedChildren returns Iterable

Co-authored-by: compulim <[email protected]>

* Store: getSortedChildren takes parent ID

Co-authored-by: compulim <[email protected]>

* Store: getSortedChildren requires init with sortBy

Co-authored-by: compulim <[email protected]>

* Store: allow duplicate position in the same part

Co-authored-by: compulim <[email protected]>

* Store: skip list follows mutate()/commit() pattern, remove position from Node

Co-authored-by: compulim <[email protected]>

* Store: store hasPart/isPartOf as ReadonlyNodeSet backed by IDs, getPartIds() is O(1)

Co-authored-by: compulim <[email protected]>

* Store: createNode() reuses ReadonlyNodeSet and copies IDs via iterator

Co-authored-by: compulim <[email protected]>

* Store: forbid nested edit(), move data structures to src/structure, rename getSortedChildren to getSortedParts

Co-authored-by: compulim <[email protected]>

* Store: JSON value follows JSON.stringify(), honors toJSON() and rejects recursive values

Co-authored-by: compulim <[email protected]>

* Store: JSON value allows plain objects/arrays only, no toJSON(), still rejects recursive

Co-authored-by: compulim <[email protected]>

* Store: dateCreated as ISO timestamp string, sorted as Date

Co-authored-by: compulim <[email protected]>

* Store: cache parsed dateCreated on node to avoid parsing on every comparison

Co-authored-by: compulim <[email protected]>

* Store: brand nodes from createNode() as InternalNode via valibot brand

Co-authored-by: compulim <[email protected]>

* Update

* Store: getSortedParts() supports order, rename sortBy to by

Co-authored-by: compulim <[email protected]>

* Update JSON literal type to @JSON

* Clarify IRI

* Add performance characteristics

* Typo

* Typo

* Store: seal NodeReference with Exact, reuse deeply frozen values in createNode()

Co-authored-by: compulim <[email protected]>

* Store: add npm run benchmark with aligned Markdown table of average per node

Co-authored-by: compulim <[email protected]>

* Store: move deepFreeze to own file, transpose benchmark table

Co-authored-by: compulim <[email protected]>

* Store: add createCachedIterable (not yet used, slower than array in deepFreeze)

Co-authored-by: compulim <[email protected]>

* Store: single-pass deepFreeze, remove createCachedIterable, merge ExactNodeReference into NodeReference

Co-authored-by: compulim <[email protected]>

* Store: skip dangerous keys (__proto__, constructor, prototype) when deep freezing

Co-authored-by: compulim <[email protected]>

* Store: VersionedStorage mutate()/commit() API, recursive deepFreeze

Co-authored-by: compulim <[email protected]>

* More

* Store: document why VersionedStorage uses undo log

Co-authored-by: compulim <[email protected]>

* More

* Plain object

* Store: add clone option to deepFreeze and upsert

Co-authored-by: compulim <[email protected]>

* Store: benchmark clone vs no clone, faster extra-key check

Co-authored-by: compulim <[email protected]>

* Revert "Store: benchmark clone vs no clone, faster extra-key check"

This reverts commit db766b8.

Co-authored-by: compulim <[email protected]>

* Revert "Store: add clone option to deepFreeze and upsert"

This reverts commit e699e36.

Co-authored-by: compulim <[email protected]>

* Store: validate JSON value by a single recursive check()

Co-authored-by: compulim <[email protected]>

* Plan: JSON literal matches JSON.parse(JSON.stringify())

Co-authored-by: compulim <[email protected]>

* Store: JSON literal behaves as JSON.parse(JSON.stringify())

Co-authored-by: compulim <[email protected]>

* Store: use propertyValueSchema output for rest properties of node

Co-authored-by: compulim <[email protected]>

* Store: consume readonly structures via `new ReadonlyMap()` etc. instead of `createReadonlyX()`

Co-authored-by: compulim <[email protected]>

* Store: rename ReadonlyX structures to FrozenX to avoid clashing with built-in types

Co-authored-by: compulim <[email protected]>

* Move DLJS to different entry, fix default entry

* Plan: drop exported isFrozenMap

Co-authored-by: compulim <[email protected]>

* Structure entry

* Store: add adopt, pending, and delete parent benchmarks

Co-authored-by: compulim <[email protected]>

* Store: keep node changes in ReadWriteNode drafts, build each node once per transaction

Co-authored-by: compulim <[email protected]>

* Store: createNode returns unfrozen node, freezeNode freezes in place

Co-authored-by: compulim <[email protected]>

* Store: benchmark unfrozen createNode

Co-authored-by: compulim <[email protected]>

* Store: move pending parts check to assertNoPendingParts

Co-authored-by: compulim <[email protected]>

* Store: type rest schema of Node via BaseSchema cast instead of transform

Co-authored-by: compulim <[email protected]>

* Store: validate identifier with a single custom() schema

Co-authored-by: compulim <[email protected]>

* Store: revert identifierSchema to union() and pipe() for readability

Co-authored-by: compulim <[email protected]>

* Store: brand InternalNode with a non-exported symbol

Co-authored-by: compulim <[email protected]>

* Store: copy skip list links by spread, slice() of frozen arrays is slow

Co-authored-by: compulim <[email protected]>

* Apply remaining changes

Co-authored-by: compulim <[email protected]>

* Store: reuse overlay changes as undo log in VersionedStorage commit

Co-authored-by: compulim <[email protected]>

* Add design choice

* Add clarification

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: compulim <[email protected]>
Co-authored-by: William Wong <[email protected]>
…#6038)

* Plan: fix Node type accepting invalid property values

Co-authored-by: compulim <[email protected]>

* Store: reject non-array property values at compile-time in upsert()

Co-authored-by: compulim <[email protected]>

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: compulim <[email protected]>
…r()` (#6041)

* Store: add alternate `identifier` property and O(1) `queryByIdentifier()`

Co-authored-by: compulim <[email protected]>

* Add `identifier` property to custom parts

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: compulim <[email protected]>
Co-authored-by: William Wong <[email protected]>
@compulim William Wong (compulim) changed the title New unwired graph store New unshipped graph store Nov 19, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Node operations fail in supported browsers, async transactions can partially commit, and the new test suites fail the existing Jest job.

16 open issues
What changed in this PR

Adds a standalone graph-store package as groundwork for a new Web Chat state model. The store is not yet connected to the existing runtime; its Direct Line adapter is a placeholder.

Changes:

  • Adds transactional graph edits, immutable snapshots, identifier lookup, sorted child traversal, and subscriptions.
  • Introduces persistent collections, node validation, JSON normalization, tests, and benchmarks.
  • Registers the workspace with separate store, collection, and Direct Line exports.
File Description
packages/​store/​tsup.config.ts Configures ESM and CommonJS entry points.
packages/​store/​src/​tsconfig.json Enables TypeScript-extension imports.
packages/​store/​src/​structure/​VersionedStorage.ts Implements versioned storage and writable overlays.
packages/​store/​src/​structure/​VersionedStorage.test.ts Tests commits, historical reads, and branching.
packages/​store/​src/​structure/​SkipListView.ts Provides skip-list traversal methods.
packages/​store/​src/​structure/​SetView.ts Provides shared Set read operations.
packages/​store/​src/​structure/​MapView.ts Provides shared Map read operations.
packages/​store/​src/​structure/​FrozenSkipList.ts Implements persistent sorted collections.
packages/​store/​src/​structure/​FrozenSkipList.test.ts Tests ordering, mutation, and historical traversal.
packages/​store/​src/​structure/​FrozenSet.ts Implements immutable sets and writable overlays.
packages/​store/​src/​structure/​FrozenSet.test.ts Tests set compatibility and version isolation.
packages/​store/​src/​structure/​FrozenNodeSet.ts Represents node references using stored IDs.
packages/​store/​src/​structure/​FrozenNodeSet.test.ts Tests reference membership and immutability.
packages/​store/​src/​structure/​FrozenMap.ts Implements immutable maps and writable overlays.
packages/​store/​src/​structure/​FrozenMap.test.ts Tests map compatibility and version isolation.
packages/​store/​src/​structure.ts Exports public map and set APIs.
packages/​store/​src/​Store.ts Implements graph transactions, queries, and subscriptions.
packages/​store/​src/​private/​UnfrozenNode.ts Brands validated, writable node objects.
packages/​store/​src/​private/​sortedChildIds.ts Maintains cached child ordering.
packages/​store/​src/​private/​ReadWriteNode.ts Implements copy-on-write node drafts.
packages/​store/​src/​private/​ReadWriteNode.test.ts Tests draft changes and reference removal.
packages/​store/​src/​private/​InternalNode.ts Brands nodes stored in the graph.
packages/​store/​src/​private/​identifierIndex.ts Maintains alternate-identifier lookup and uniqueness.
packages/​store/​src/​private/​getPartIds.ts Extracts membership IDs efficiently.
packages/​store/​src/​private/​freezeNode.ts Finalizes nodes with membership and ordering metadata.
packages/​store/​src/​private/​freezeNode.test.ts Tests node finalization and hidden metadata.
packages/​store/​src/​private/​deepFreeze.ts Copies and freezes validated values.
packages/​store/​src/​private/​deepFreeze.test.ts Tests immutability, recursion, and dangerous-key removal.
packages/​store/​src/​private/​dateCreatedTime.ts Caches parsed timestamps for sorting.
packages/​store/​src/​private/​createNode.ts Validates and prepares graph nodes.
packages/​store/​src/​private/​createNode.test.ts Tests validation, reuse, and JSON normalization.
packages/​store/​src/​OPINIONS.md Documents graph semantics and design choices.
packages/​store/​src/​node/​PropertyValue.ts Defines supported property-value arrays.
packages/​store/​src/​node/​NodeReference.ts Defines exact node-reference types and validation.
packages/​store/​src/​node/​NodeReference.test.ts Tests reference validation and excess-property rejection.
packages/​store/​src/​node/​NodeIdentifier.ts Defines supported node identifiers.
packages/​store/​src/​node/​NodeIdentifier.test.ts Tests valid and invalid identifiers.
packages/​store/​src/​node/​Node.ts Defines node schemas and read/write types.
packages/​store/​src/​node/​JSONValueSchema.test.ts Tests JSON serialization-compatible normalization.
packages/​store/​src/​node/​JSONValue.ts Normalizes values into JSON-compatible data.
packages/​store/​src/​node/​JSONLiteral.ts Defines JSON-LD literal validation.
packages/​store/​src/​node/​BlankNodeIdentifier.ts Defines blank-node identifier validation.
packages/​store/​src/​index.ts Exports the public store API.
packages/​store/​src/​index.test.ts Adds an entry-point import smoke test.
packages/​store/​src/​directLineJS/​createDirectLineJSStore.ts Adds a placeholder adapter with sample data.
packages/​store/​src/​directLineJS/​createDirectLineJSStore.test.ts Tests placeholder store initialization.
packages/​store/​src/​directLineJS.ts Exports the Direct Line adapter.
packages/​store/​src/​benchmark.ts Measures graph operations and memory usage.
packages/​store/​src/​assertion/​assertUniqueIdentifier.ts Enforces identifier uniqueness.
packages/​store/​src/​assertion/​assertReferencedNodesInGraph.ts Checks referenced-node existence.
packages/​store/​src/​assertion/​assertNotSelfPart.ts Rejects self-membership relationships.
packages/​store/​src/​assertion/​assertNoPendingParts.ts Rejects unresolved child memberships.
packages/​store/​src/​assertion/​assertIdentifierNotId.ts Rejects primary IDs used as alternate identifiers.
packages/​store/​package.json Declares exports, dependencies, and workspace scripts.
packages/​store/​.gitignore Excludes generated and local workspace files.
package.json Registers the store workspace.

Review effort: Balanced

Comment thread packages/store/package.json Resolved
Comment thread packages/store/src/Store.ts Resolved
Comment thread packages/store/package.json Resolved
Comment thread packages/store/src/structure/SetView.ts Resolved
Comment thread packages/store/src/structure/SetView.ts Resolved
William Wong (compulim) added a commit that referenced this pull request Nov 19, 2026
…de, FrozenNodeSet (#6059)

* Add failing tests for outstanding review comments in #6053

Co-authored-by: copilot-swe-agent[bot] <[email protected]>

* Fix review issues: createNode accessor, SetView polyfill-free Set methods, VersionedStorage ordering and snapshot-safe iteration

Co-authored-by: copilot-swe-agent[bot] <[email protected]>

* Simplify VersionedStorage: copy on move instead of tracking positions

Co-authored-by: copilot-swe-agent[bot] <[email protected]>

* Document why SetView does not use core-js-pure

Co-authored-by: copilot-swe-agent[bot] <[email protected]>

---------

Co-authored-by: compulim <[email protected]>
Co-authored-by: copilot-swe-agent[bot] <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread packages/store/src/node/JSONValue.ts Resolved
Comment thread packages/store/src/structure/SetView.ts Resolved
Comment thread packages/store/src/structure/SetView.ts Resolved
Comment thread packages/store/src/structure/SetView.ts Resolved
Implement generator function for entries method to yield key-value pairs.

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
William Wong (compulim) added a commit that referenced this pull request Nov 19, 2026
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
William Wong (compulim) added a commit that referenced this pull request Nov 19, 2026
William Wong (compulim) added a commit that referenced this pull request Nov 19, 2026
* Add failing tests for unresolved review comments on #6053

Co-authored-by: copilot-swe-agent[bot] <[email protected]>

* Show JSON.parse(JSON.stringify()) behavior side-by-side in failing tests

Co-authored-by: copilot-swe-agent[bot] <[email protected]>

* Show native Set behavior side-by-side in SetView invalid size tests

Co-authored-by: copilot-swe-agent[bot] <[email protected]>

* Fix review comments from #6053 so all failing tests pass

Co-authored-by: copilot-swe-agent[bot] <[email protected]>

* Drop sub-millisecond precision of dateCreated

Co-authored-by: copilot-swe-agent[bot] <[email protected]>

* Add TIL test

* Remove read-once tests, add tests for non-enumerable/accessor/inherited properties in createNode

Co-authored-by: copilot-swe-agent[bot] <[email protected]>

* Add enum/non-enum and getter/value

* createNode only reads own enumerable properties, like JSON.stringify()

Co-authored-by: copilot-swe-agent[bot] <[email protected]>

---------

Co-authored-by: compulim <[email protected]>
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: William Wong <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

FrozenSet subclasses can let committed snapshots change outside transactions, and unresolved iterator, set-equality, and snapshot-mutation defects remain.

2 open issues
3 previously missed issues

In code that hasn't changed since last review

High severity Copy FrozenSet subclasses when deep-freezing

packages/​store/​src/​structure/​FrozenSet.ts:122

isFrozenSet() also accepts subclasses whose read methods depend on mutable external state. Valibot's default-branch set parser accepts and iterates them, so deepFreeze() can reuse one for @type or identifier and let a committed snapshot change outside a transaction (this package pins 1.2.0). Only reuse instances with the exact FrozenSet.prototype; copy subclasses instead.

Medium severity Freeze sorted entries before committing

packages/​store/​src/​structure/​FrozenSkipList.ts:110

commit() freezes each entry's arrays but leaves the entry object writable. Snapshot symbols expose the sorted list, and its TypeScript-protected read() is callable at runtime, so replacing entry.next or entry.previous corrupts later getSortedParts() results. Freeze each entry as well as its arrays before committing.

Medium severity Use node-ID equality consistently in set operations

packages/​store/​src/​structure/​SetView.ts:131

parts.intersection(new Set(parts)) is empty for a nonempty FrozenNodeSet, because each iteration creates new reference objects and this branch compares them by identity. Adding an unrelated ID to parts switches branches and finds the matching IDs, so the result depends on relative set sizes. Apply node-ID equality consistently here and in the other.has() paths of difference(), isDisjointFrom(), isSubsetOf(), and symmetricDifference().

8 resolved since last review

Review effort: Balanced

William Wong (compulim) added a commit that referenced this pull request Nov 20, 2026
* Add failing tests for unresolved review comments on #6053

Co-authored-by: copilot-swe-agent[bot] <[email protected]>

* Fix FrozenNodeSet equality key collision and overlay iteration across commit

Co-authored-by: copilot-swe-agent[bot] <[email protected]>

---------

Co-authored-by: compulim <[email protected]>
Co-authored-by: copilot-swe-agent[bot] <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed to evaluate non-obvious risks to snapshot integrity and test discovery beyond the published review.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@compulim William Wong (compulim) changed the title New unshipped graph store New non-shipping graph store Nov 20, 2026
* Returns the IDs in O(1) without copying if the property is a `FrozenNodeSet` (i.e. the node is created by
* `createNode()`), otherwise, builds them in O(k).
*/
function getPartIds(node: Pick<Node, PartProperty> | undefined, property: PartProperty): ReadonlySet<Identifier> {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could consider attaching hidden symbol that is memoized string[], look at how we done it in dateCreatedTime.ts.

Comment on lines +1 to +2
export { default as FrozenMap, type ReadWriteMap } from './structure/FrozenMap.ts';
export { default as FrozenSet, type ReadWriteSet } from './structure/FrozenSet.ts';

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NTM: Do we want to export them now or later?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants