Repository navigation
perf(core): cache NumPy matrices, add SQLite visibility revision, and dedupe batch embeddings - #337
Merged
Merged
Conversation
… dedupe batch embeddings Maintain a local-only trigger-invalidated vector_visibility_state revision and bounded 64 MiB NumPy matrix cache across temporal validity windows, install idx_mem_ingested and bounded SQLite page caches (~16 MiB, with opt-in ENGRAPHIS_SQLITE_MMAP=1) as content-free additive v19 metadata without a second pre-migration backup, and deduplicate repeated texts in MemoryEngine.remember_many.
Codex Review Summary: ❌ FailedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
vector_visibility_state+ ninetrg_vector_vis_*triggers andidx_mem_ingestedto track scope/visibility mutations separately from reinforcement updates (engraphis/core/schema.py,engraphis/core/store.py).NumpyVectorIndexdecoded matrix cache (ENGRAPHIS_VECTOR_CACHE_BYTES, default67108864) with wall-clock transition expiry acrossworld(valid_from,valid_to,memory_closures.valid_to) andknown(ingested_at,expired_at,valid_to_recorded_at,memory_closures.recorded_at) axes, read-only arrays, over-budget eviction, connection-epoch invalidation, and streaming/legacy fallbacks (engraphis/backends/vector_numpy.py,engraphis/core/vector_search.py).MemoryEngine.remember_manybefore embedding (engraphis/core/engine.py), configure ~16 MiB per-connection SQLitePRAGMA cache_size=-16384and opt-inENGRAPHIS_SQLITE_MMAP=1(explicitly disabled on SQLCipher viasupports_mmap_pragma = Falseinengraphis/backends/encrypted_db.py), and publish regenerated offline evidenceoffline-fixtures-perf-vector-20261118.json(91362de33eeb...) preserving#334(offline-fixtures-connect-verify-20261117.json) as predecessor.python -m eval.vector_scale --backend numpy --sizes 1000,10000 --dim 256 --queries 20 --iterations 3,warmups=1, Windows CPython 3.11.17 / NumPy 2.4.6): warm 64 MiB cache vs uncachedENGRAPHIS_VECTOR_CACHE_BYTES=0p50 improves16.234 ms -> 0.194 ms(1,000 vectors) and80.929 ms -> 2.448 ms(10,000 vectors) with identicalresult_ids_sha256(excludes cold fill and SQLite open setup).Operator & Merge Coordination Notes
SCHEMA_VERSIONbump / no second backup):vector_visibility_state, its triggers, andidx_mem_ingestedare installed additively onto existing v19 DBs without demanding a second.pre-migration-v19.bak; safe rollback drops all ninetrg_vector_vis_*triggers beforevector_visibility_stateandidx_mem_ingested.feat/pro-team-ux-parity: Both PRs touchBENCHMARKS.md,README.md,docs/images/*.svg|png, andtests/test_benchmark_evidence.py. Whichever merges second must rebase and regenerate a unifiedoffline-fixtures-*.jsonartifact and charts atop updatedmain.Independent Verification (
d425cadebef39d9c327f1ada94ac4e79faad55cc)ruff check .,check_commercial_manifest.py,externalize_dashboard_assets.py: PASSpyright --pythonpath .venv\Scripts\python.exe: PASS (0 errors, 0 warnings, 0 informations)pytest -o addopts= tests/ -q -rs -n 2 --dist loadfile):10791 passed, 106 skipped(exit 0)deepseek/deepseek-v4.1-flash --effort max): 2/2 APPROVE (0Blockers,0Majors; independently verifiedvalid_to_recorded_atsystem-time binding against_memory_temporal_visibility_sqland#334preservation).