Repository navigation
feat!: declare jobs, job groups and schedules in TOML - #1483
Merged
Merged
Conversation
Work in progress, not ready for review. - Publish-time job, sensor-condition and role selections resolve with the same unified SQL/Python resolver `sqb build --select` uses. - Python snapshot nodes publish deps (Python and SQL) and tags; an intermediate loader publishes the source node it writes, and loader chains list it in `loads`. - Server-side ad-hoc selection rebuilds the Python selection graph from the snapshot and resolves with the same resolver. - The worker selects Python nodes by typed key. Validated so far: test_orchestration_snapshot.py passes (99 tests, including the Python selector reproduction). The Rust unit tests for the ad-hoc planner and planner scope have not finished yet; no review and no full gate.
A Python node can now be selected without its Python upstream in every path that uses the shared resolver: sqb build --select, published jobs, sensors, roles and ad-hoc runs. It reads the upstream's latest persisted result, and fails at run time with an error naming the upstream and the command to build it when the upstream has never succeeded. An intermediate loader without a write_strategy that returns rows now writes them to __loader__<name> as a table, as the loaders docs describe. A self-managed intermediate that returns nothing keeps its behaviour. The published snapshot rebuilds the Python selection graph from node deps, and job groups order jobs across Python and Python-to-SQL edges. Validated locally: python_selection and adhoc planner Rust tests, clippy on both crates, fensu check, the worker claim tests, orchestration snapshot, Python node selection E2E, load E2E (sqb load and sqb build on the docs example) and the python_nodes and executor/load unit tests. Not yet run: make check-ci, the full suite, review.
- `job:` and bare job names resolve jobs through the Python-aware resolver publication uses, so a job that selects only Python nodes builds them. - Server ad-hoc runs accept `test:` selectors; a selected SQL test reports as a run node rather than as an attached check. - A provider function whose context is positional-only (`ctx, /`) receives it by position instead of failing at run time. - A role must grant the intermediate loader a job loads to run that job.
…odes Selections now publish a managed source instead of its loader, ad-hoc test, audit, seed and freshness runs resolve SQL only, a selected SQL test reports its own outcome, and a selected model folds in its failing test. A selector that matches a Python check must also select the nodes the check depends on, in jobs, roles and the CLI alike.
A build never loads such a source, so a matched check on it would be accepted and then never run. The resolver now names the check and the source and points to sqb check.
Loader and Python node contexts read upstream results through one mixin, and the selection entry points build their context in one place. Regenerates the packaged docs skill for the selector and check pages.
…dits JobGroup no longer takes after: a group's order comes only from the lineage between the nodes its jobs build, and declaring after fails with an error that names the group. build --no-audits now skips Python checks alongside audits, as documented.
…tions A Python task, asset, loader, check, hook, provider or UDF description that wraps past [format] max_description_lines is a P027 compile error at its declaration, and a job, job group, sensor or calendar schedule with one cannot publish.
…snapshot # Conflicts: # crates/sqlbuild-python/src/bindings/_helpers/compiled_project/snapshot_execution.rs
…th lineage-only groups Rust orchestration fixtures declare the node lineage their group waits come from. A Python description limit falls back to the defaults without a project directory, and a calendar schedule that never decorated a function no longer breaks publication for other targets; DuckShop's calendar edge case now declares one.
…p JobGroup after JobGroup no longer has an after parameter at all, so passing it fails at the call site. Calendar names and descriptions are read through the native host edge, the admission fixture builds stage lineage without branching, and the description-limit test keeps its cases beside the test.
…thout importing Python Discovery reads orchestration/jobs/ and orchestration/job_groups/ TOML with source positions, scans calendar and sensor modules statically for their literal names and job references, and reports layout, suffix and reference errors as D017. Not wired into compilation yet.
…heir names with the project namespace
…s by name Jobs and job groups publish from their TOML files. Selectors that name a job read the TOML without importing project Python. Publication imports only calendar and sensor modules. A calendar is @Calendar(name, description, horizon), and a job's schedule names it and sets the timezone. Sensor run= and run_event(job=) take names. The removed records stay private for the migration command.
…h it with the snapshot The renames file maps old job, group, sensor and trigger names to new ones. Compile refuses a new name that is not declared, an old name that still is, a chain, and two names mapped onto one, at the file position. Publication carries the map as the snapshot's renames document. Sensor targets written as literals are read statically for the job catalog.
The migration imports each orchestration module once with the removed records restored, so jobs built in loops or from constants migrate too. It appends missing name suffixes and prints each rename, writes one TOML file per job and group, writes renames.toml so stored history follows the new names, deletes modules that held only jobs, and prints the exact calendar and sensor edits.
…ts at it Wire the migrate command through the CLI table, dispatch and bindings. Loose Python files, unknown folders holding Python, and Python files in jobs/ or job_groups/ now fail with help to run the migration. A test keeps the removed job records reachable only from the migration.
Projects declare jobs, job groups and schedules in TOML directly, so the Python-to-TOML migrator, the renames document and its server-side rewrite are removed. Python modules that still declare jobs or schedules now point at the TOML layout and the orchestration docs.
sqb rename and sqb mv now update exact model names and whole-folder paths in orchestration/jobs and orchestration/job_groups selectors, keeping quote style. A folder that only some models leave is reported as a note, and multi-line selector strings are left for a manual edit.
Jobs, job groups and their schedules are now TOML files under orchestration/jobs/ and orchestration/job_groups/. Sensors and calendars stay Python. Python job, job_group, schedule and calendar_schedule usage fails compilation with D017 and points to the orchestration docs. Convert DuckShop, its orchestration edge cases, the deploy example and the tests to TOML, and add the orchestration concepts page.
# Conflicts: # crates/sqlbuild-python/src/bindings/_helpers/compiled_project/snapshot_declarations.rs # crates/sqlbuild-python/src/bindings/_helpers/location_command/snapshot.rs # crates/sqlbuild-python/src/bindings/_helpers/native_host/constants.rs # src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/selectors.md # tests/e2e/src/sqlbuild/cli/commands/main/build/helpers.py # tests/e2e/src/sqlbuild/cli/commands/main/build/test_python_node_selection.py # tests/integration/src/sqlbuild/compiler/compiled_project/helpers.py # tests/integration/src/sqlbuild/compiler/compiled_project/publication/test_project_roles.py # tests/integration/src/sqlbuild/compiler/compiled_project/publication/test_worker.py # tests/integration/src/sqlbuild/compiler/compiled_project/test_orchestration_snapshot.py # website/src/content/docs/docs/concepts/selectors.mdx
… selectors honest - Compile checks schedule cron, time zone and starts_at, build settings, worker resources and selector syntax, as D017 at the value's line and column, with the checks publication applies. - Only calls on the sqlbuild.orchestration module count as calendar, sensor or run_event declarations; a provider or stdlib method named calendar no longer reads as a second declaration. - rename and mv no longer rewrite a folder selector into a folder that keeps or receives other models; they report it instead. - Publication refuses a calendar or sensor that compile declared but import did not define, or the reverse, instead of dropping it. - Restore the check that model selectors never import orchestration Python, and match the nested-group rejection exactly.
# Conflicts: # benchmarks/duckshop/edge_cases/orchestration/calendar_orders.py # crates/sqlbuild-python/src/bindings/_helpers/compiled_project/snapshot_declarations.rs # crates/sqlbuild-python/src/bindings/_helpers/native_host/orchestration_declarations.rs # crates/sqlbuild-python/src/bindings/_helpers/native_host/orchestration_triggers.rs # src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/models/configuration.md # src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/project-configuration.md # src/sqlbuild/orchestration_declarations/models/authoring.py # tests/integration/src/sqlbuild/compiler/compiled_project/test_orchestration_snapshot.py # website/src/content/docs/docs/concepts/models/configuration.mdx # website/src/content/docs/docs/concepts/project-configuration.mdx
…zones at their line
… and orchestration
…h the project rules Move job and group file reading into the discovery helpers, split the inventory into named phases, keep Python object access at the native host edge, return schedules instead of filling caller collections, and restructure the new Rust and Python tests as branch-free case tables.
The failure corpus must reach every discovery error code. D017, which rejects Python job modules and invalid TOML orchestration declarations, had no case, so the corpus coverage E2E failed in the merge queue.
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Jobs, job groups and schedules were Python objects, so compile could only see them by importing project code. Mistakes surfaced at publication, and selectors that named a job had to import orchestration modules.
Changes
orchestration/jobs/andorchestration/job_groups/, with an optionalschedule. Calendars and sensors stay Python inorchestration/calendars/andorchestration/sensors/. Names end in_job,_job_group,_calendarand_sensorand share the project namespace.starts_at, settings, resources, selector syntax) and references.job,job_group,scheduleandcalendar_schedulefail with D017 and point to the new orchestration docs page. There is no migration tool.sqb rename/mvrewrite model names and wholly moved folders in job selectors; anything ambiguous becomes a note.Verification
Review findings, each with a test that fails before its fix and passes after. A follow-up review confirmed all five fixed.
sqb compileE2E cases..calendar()read as a declaration: pysource unit test; DuckShop calendar edge case.1023 targeted tests pass (publication, snapshot, selectors, changed builds, refactor, layout) (230 rerun after the rules cleanup); Rust tests for discovery, pysource and refactor pass;
make check-cipasses.DuckShop: its orchestration and edge cases are TOML, and the calendar fixture calls a provider method named
calendar. The production-only calendar schedule case was removed because calendars no longer have targets.