Repository navigation
perf!: share Python identity work per discovery pass and keep compile phases out of sinks - #1501
Merged
Merged
Conversation
- Compile: a sensor or calendar declared inside a `_`-prefixed folder under orchestration/sensors/ or orchestration/calendars/ now fails with an error naming the file, since such folders are private support. - SQBRPYTHON016-020: module paths, private support, module surface, import direction and no docstring on sensor and calendar functions. - Provider and orchestration rules share parsing and import resolution. - Evict nested namespace packages before their parents between projects in one interpreter, so a project importing orchestration.* no longer breaks the next compile. - Docs and DuckShop fixtures for each rule.
A provider whose required secret settings are unset no longer stops every command. Static commands (compile, dag, format, scope, lineage, rules run) still fail with D012 naming the variable, unless --allow-missing-secrets, SQLBUILD_ALLOW_MISSING_SECRETS or the local allow_missing_secrets setting allows it. The project file rejects that setting. build, load and check read secrets only for the providers the selected nodes and sinks use, and fail before anything runs when one is missing. Providers are instantiated lazily in the runtime session. sqb debug reports how many providers are ready and lists missing secrets (--verbose lists every one). Location health reports missing_secrets and exits 1, and the Providers page marks unset secret settings.
…PYTHON rules sqb test now runs the project's pytest modules under tests/python/ and tests/providers/ after its SQL tests, from the project directory with --import-mode=importlib. --sql and --python pick one kind. Selecting a Python node runs the tests in its mirrored folder; selecting only models or unit tests runs none. The command fails when either kind fails, and fails early when Python tests exist but pytest is not installed. sqb test --since, --inspect, review and ad-hoc runs stay SQL-only, and sqb build never runs Python tests. With --json, pytest writes to stderr. - SQBRPYTHON021: a test sits in the folder mirroring the code it tests, and test_<name>.py has a <name>.py or <name>/ beside that code. - SQBRPYTHON701-724: the Fensu test-shape rules (table-driven dataclass cases, naming, parametrize ids), applied only to Python tests. - Docs for sqb test, sqb build, testing, project layout, custom rule tests and the Python Rules reference, plus DuckShop fixtures.
sqb compile imports project Python with no warehouse connection, so the code that runs then must not reach outside the machine. SQBRPYTHON022 checks calls at module scope, in class bodies and directly in @factory bodies, follows the project functions they call through their imports, and reports a call that reaches a subprocess, an HTTP request, a socket, a database or warehouse driver, or a cloud or SSH client. It reads source only and follows only calls it can resolve, so a call through a variable or an object is never reported. - approved_loader_boundaries lists exact project functions the walk does not enter; a glob fails configuration and a module name fails the run. - Findings are reported at the root call and name the external call and where it is. Results per function are memoized, with a scaling test over deep shared call chains. - Docs for the rule, its option and its reference row, and a DuckShop violation behind a branch compile never takes.
…o perf/python-identity-sharing
… on dev start (work in progress)
… sources as staying Python
…sharing # Conflicts: # crates/sqlbuild-python/src/bindings/_helpers/debug_command/run_debug.rs
…globals An identity built after discovery looked up the node's module through sys.modules. When discovery had put back an unrelated top-level python package, that lookup found the unrelated module, so helpers were no longer dependencies and editing one did not change the node's version hash.
…the identity pass The shared Python identity pass and its native hooks add 169 runtime and 51 crate code lines over the caps measured before this change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Discovery re-read, re-parsed and re-hashed shared helpers per Python node.
sqb devdiscovered twice on start, and sinks slowedsqb compileby receiving every phase event. The dense delta loop failed on runner wall-time noise.Changes
sqb debugreads configuration (with its checks) and providers only, sosqb dev's connection check no longer discovers twice.sqb compileno longer sends phaseoperationevents to project lifecycle sinks. Invocation events still do.object_source.pyandidentity_pass.pystay Python (they read live user functions), as approved.sys.modules, which may hold an unrelatedpythonpackage after discovery.Verification
sqb devimports project Python once on start (was twice). Compile sinks get invocation events only.sqb debugstill rejects a forbidden local SQL analysis opt-out. An unrelatedpythonpackage no longer hides helpers (fails without the fix).make check-cipasses.DuckShop: adds
edge_cases/orchestration/python_identity_class_shapes.py.