Skip to content

perf(core): cache NumPy matrices, add SQLite visibility revision, and dedupe batch embeddings - #337

Merged
Coding-Dev-Tools merged 2 commits into
mainfrom
perf/core-sqlite-vector
Nov 19, 2026
Merged

Coding-Dev-Tools merged 2 commits into
mainfrom
perf/core-sqlite-vector

Conversation

@Coding-Dev-Tools

Copy link
Copy Markdown
Owner

Summary

  • Add local content-free vector_visibility_state + nine trg_vector_vis_* triggers and idx_mem_ingested to track scope/visibility mutations separately from reinforcement updates (engraphis/core/schema.py, engraphis/core/store.py).
  • Add a 64 MiB bounded NumpyVectorIndex decoded matrix cache (ENGRAPHIS_VECTOR_CACHE_BYTES, default 67108864) with wall-clock transition expiry across world (valid_from, valid_to, memory_closures.valid_to) and known (ingested_at, expired_at, valid_to_recorded_at, memory_closures.recorded_at) axes, read-only arrays, over-budget eviction, connection-epoch invalidation, and streaming/legacy fallbacks (engraphis/backends/vector_numpy.py, engraphis/core/vector_search.py).
  • Deduplicate repeated texts in MemoryEngine.remember_many before embedding (engraphis/core/engine.py), configure ~16 MiB per-connection SQLite PRAGMA cache_size=-16384 and opt-in ENGRAPHIS_SQLITE_MMAP=1 (explicitly disabled on SQLCipher via supports_mmap_pragma = False in engraphis/backends/encrypted_db.py), and publish regenerated offline evidence offline-fixtures-perf-vector-20261118.json (91362de33eeb...) preserving #334 (offline-fixtures-connect-verify-20261117.json) as predecessor.
  • Non-registered direct search measurement (python -m eval.vector_scale --backend numpy --sizes 1000,10000 --dim 256 --queries 20 --iterations 3, warmups=1, Windows CPython 3.11.17 / NumPy 2.4.6): warm 64 MiB cache vs uncached ENGRAPHIS_VECTOR_CACHE_BYTES=0 p50 improves 16.234 ms -> 0.194 ms (1,000 vectors) and 80.929 ms -> 2.448 ms (10,000 vectors) with identical result_ids_sha256 (excludes cold fill and SQLite open setup).

Operator & Merge Coordination Notes

  • Additive v19 metadata (no SCHEMA_VERSION bump / no second backup): vector_visibility_state, its triggers, and idx_mem_ingested are installed additively onto existing v19 DBs without demanding a second .pre-migration-v19.bak; safe rollback drops all nine trg_vector_vis_* triggers before vector_visibility_state and idx_mem_ingested.
  • Evidence conflict coordination with feat/pro-team-ux-parity: Both PRs touch BENCHMARKS.md, README.md, docs/images/*.svg|png, and tests/test_benchmark_evidence.py. Whichever merges second must rebase and regenerate a unified offline-fixtures-*.json artifact and charts atop updated main.

Independent Verification (d425cadebef39d9c327f1ada94ac4e79faad55cc)

  • ruff check ., check_commercial_manifest.py, externalize_dashboard_assets.py: PASS
  • pyright --pythonpath .venv\Scripts\python.exe: PASS (0 errors, 0 warnings, 0 informations)
  • Full unit suite (pytest -o addopts= tests/ -q -rs -n 2 --dist loadfile): 10791 passed, 106 skipped (exit 0)
  • All 9 offline eval gates: PASS
  • Two fresh independent Command Code reviews (deepseek/deepseek-v4.1-flash --effort max): 2/2 APPROVE (0 Blockers, 0 Majors; independently verified valid_to_recorded_at system-time binding against _memory_temporal_visibility_sql and #334 preservation).

… dedupe batch embeddings

Maintain a local-only trigger-invalidated vector_visibility_state revision and
bounded 64 MiB NumPy matrix cache across temporal validity windows, install
idx_mem_ingested and bounded SQLite page caches (~16 MiB, with opt-in
ENGRAPHIS_SQLITE_MMAP=1) as content-free additive v19 metadata without a second
pre-migration backup, and deduplicate repeated texts in MemoryEngine.remember_many.
@chatgpt-codex-connector

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ❌ Failed 2026-11-19T03:12:04.139490Z
We couldn’t complete this review. Comment @codex review to try again.
d425cad PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@Coding-Dev-Tools
Coding-Dev-Tools merged commit 4201fab into main Nov 19, 2026
30 checks passed
@Coding-Dev-Tools
Coding-Dev-Tools deleted the perf/core-sqlite-vector branch November 19, 2026 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant