Repository navigation
feat: add the SQLClean engine, command and Python custom Rules - #1
Merged
Merged
Conversation
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
Built-in Rules use SQC codes. The engine exposes JSON lint, batch and format entries so host tools can embed it with their own codes. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
Configuration comes from the nearest sqlclean.toml. Results are cached per file, fixes run until stable and unsafe fixes need --unsafe-fixes. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
A finding whose only fix is unsafe was counted as fixable with plain sqlclean fix, which leaves it unchanged. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
A table fact for raw.payments covered only payments, so findings and fixes at the node left the schema out. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
sqlclean format formatted one file at a time while check already used rayon. On 10,282 files format --check drops from about 10 s to 2.7 s on 4 CPUs; output keeps file order. Co-authored-by: Kevin Longe <[email protected]>
- Formatted SQL is indented four spaces per step by default, matching SQLFluff and sqruff; [format] indent chooses 1 to 8. The engine keeps two spaces unless a request passes indent, so embedding tools keep their current layout. - The parse-tree layout broke every parenthesis level of a long expression. Groups that fit on their line are now rejoined, innermost first, so only the levels that overflow break. A group that fits once its leading operator starts the line breaks before that operator. - A leading comma after a line comment moves to the end of the commented line instead of standing on its own line. - A file that holds only comments keeps its final newline. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
- website/: an Astro Starlight site with a homepage and 18 docs pages covering every command, every Rule, configuration, custom Rules, integrations, embedding and benchmarks. Example output comes from the sqlclean command. A workflow builds the site and checks its links. - The homepage and README describe what SQLClean checks, without claiming it catches every query that returns wrong rows. - Examples and the README demo show the four-space indent, and the docs cover [format] indent. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
bench.sh runs each linter on the plain-SQL DuckShop 10K corpus from SQLBuild's generator, cold and, for SQLClean, warm, and summarize.py reports median wall time, CPU time, peak memory and finding counts. convert_duckshop.py turns SQLBuild's compiled models into plain Snowflake SQL. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
They move to private configuration. Co-authored-by: Kevin Longe <[email protected]>
`--config custom.toml` made the configuration root an empty path, so every command failed with ": no such file or directory". Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
Custom Rules rebuilt each file's facts serially and ran in a single Python process. Facts are now built in parallel, and statement and file Rules split the files across one host per CPU, from 250 files a host. Repository Rules still read every file in one host. On 10,282 files with three Python Rules, check drops from 5.4 s to 2.1 s on 4 CPUs. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
sqlclean_custom runs the default Rules plus three Python Rules that a data team might write: layering between marts, staging and raw sources, and a ban on nondeterministic functions. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
- The hero shows a real-time race on the 10,282-file benchmark: the sqlclean bar fills in 0.81 s while sqruff and SQLFluff advance at their measured rates. Reduced motion shows the moment sqlclean ends. - Three feature boxes replace six: speed, custom Rules in Python and safe fixes. - The Benchmarks page adds the custom Rules row, and the custom Rules docs say statement and file Rules can run in several processes. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
…he Rust engine - The race labels each linter with its total time instead of a percentage. - The lede describes the NULL example in words. - The custom Rules box says the Rust engine parses SQL and builds the facts, so Python only runs the Rule's logic. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
fensu requires dataclass cases with an expected_ field and no conditionals in tests. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
SQLFluff and sqruff users type lint. The alias is hidden from --help; check stays the documented name, and the check page mentions lint. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
Deploying sqlclean with
|
| Latest commit: |
261abfb
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://9bbc0f4c.sqlclean.pages.dev |
| Branch Preview URL: | https://feat-initial-sqlclean.sqlclean.pages.dev |
The executable embeds a SQLClean skill with a copy of every docs page. make skills regenerates the copies from the website docs and make check-docs-skill fails while they differ. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
…eases Verify classifies a pull request by changed paths. Website, docs and benchmark edits run the docs skill check and ruff; bundled skill edits add a Linux build and the skills tests; anything else runs everything. The Verify job aggregates the selected suites. Release Please counts every Conventional Commit regardless of hidden changelog sections, so exclude-paths keeps website, benchmark and workflow-only commits from cutting a release. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <[email protected]>
This was referenced Oct 30, 2026
kvlonge
added a commit
that referenced
this pull request
Oct 31, 2026
* build: add the Cargo workspace, Python packaging and fensu policy Co-authored-by: Kevin Longe <[email protected]> * feat: add the SQL lint, format and statement facts engine Built-in Rules use SQC codes. The engine exposes JSON lint, batch and format entries so host tools can embed it with their own codes. Co-authored-by: Kevin Longe <[email protected]> * feat: add the sqlclean check, fix, format and rules commands Configuration comes from the nearest sqlclean.toml. Results are cached per file, fixes run until stable and unsafe fixes need --unsafe-fixes. Co-authored-by: Kevin Longe <[email protected]> * feat: add Python custom Rules, their host and test harness Co-authored-by: Kevin Longe <[email protected]> * ci: add verify, release and publish workflows Co-authored-by: Kevin Longe <[email protected]> * docs: document commands, configuration and custom Rules Co-authored-by: Kevin Longe <[email protected]> * fix: satisfy clippy nonminimal_bool on the CI toolchain Co-authored-by: Kevin Longe <[email protected]> * docs: add the logo and a check and fix demo to the README Co-authored-by: Kevin Longe <[email protected]> * fix: count unsafe fixes apart in the check summary A finding whose only fix is unsafe was counted as fixable with plain sqlclean fix, which leaves it unchanged. Co-authored-by: Kevin Longe <[email protected]> * fix: locate qualified tables from their first name part A table fact for raw.payments covered only payments, so findings and fixes at the node left the schema out. Co-authored-by: Kevin Longe <[email protected]> * fix: separate the message and remediation in SARIF results Co-authored-by: Kevin Longe <[email protected]> * perf(cli): format files in parallel sqlclean format formatted one file at a time while check already used rayon. On 10,282 files format --check drops from about 10 s to 2.7 s on 4 CPUs; output keeps file order. Co-authored-by: Kevin Longe <[email protected]> * feat(format): indent four spaces and break only overflowing groups - Formatted SQL is indented four spaces per step by default, matching SQLFluff and sqruff; [format] indent chooses 1 to 8. The engine keeps two spaces unless a request passes indent, so embedding tools keep their current layout. - The parse-tree layout broke every parenthesis level of a long expression. Groups that fit on their line are now rejoined, innermost first, so only the levels that overflow break. A group that fits once its leading operator starts the line breaks before that operator. - A leading comma after a line comment moves to the end of the commented line instead of standing on its own line. - A file that holds only comments keeps its final newline. Co-authored-by: Kevin Longe <[email protected]> * docs: add the SQLClean website, docs and benchmark page - website/: an Astro Starlight site with a homepage and 18 docs pages covering every command, every Rule, configuration, custom Rules, integrations, embedding and benchmarks. Example output comes from the sqlclean command. A workflow builds the site and checks its links. - The homepage and README describe what SQLClean checks, without claiming it catches every query that returns wrong rows. - Examples and the README demo show the four-space indent, and the docs cover [format] indent. Co-authored-by: Kevin Longe <[email protected]> * chore(bench): compare SQLClean with sqruff and SQLFluff bench.sh runs each linter on the plain-SQL DuckShop 10K corpus from SQLBuild's generator, cold and, for SQLClean, warm, and summarize.py reports median wall time, CPU time, peak memory and finding counts. convert_duckshop.py turns SQLBuild's compiled models into plain Snowflake SQL. Co-authored-by: Kevin Longe <[email protected]> * chore: keep agent instructions and orb setup out of the repository They move to private configuration. Co-authored-by: Kevin Longe <[email protected]> * fix(cli): read a bare --config file name from the working directory `--config custom.toml` made the configuration root an empty path, so every command failed with ": no such file or directory". Co-authored-by: Kevin Longe <[email protected]> * perf(custom): evaluate custom Rules on one host per CPU Custom Rules rebuilt each file's facts serially and ran in a single Python process. Facts are now built in parallel, and statement and file Rules split the files across one host per CPU, from 250 files a host. Repository Rules still read every file in one host. On 10,282 files with three Python Rules, check drops from 5.4 s to 2.1 s on 4 CPUs. Co-authored-by: Kevin Longe <[email protected]> * chore(bench): add a custom Rules variant sqlclean_custom runs the default Rules plus three Python Rules that a data team might write: layering between marts, staging and raw sources, and a ban on nondeterministic functions. Co-authored-by: Kevin Longe <[email protected]> * docs(website): lead with the benchmark and custom Rules - The hero shows a real-time race on the 10,282-file benchmark: the sqlclean bar fills in 0.81 s while sqruff and SQLFluff advance at their measured rates. Reduced motion shows the moment sqlclean ends. - Three feature boxes replace six: speed, custom Rules in Python and safe fixes. - The Benchmarks page adds the custom Rules row, and the custom Rules docs say statement and file Rules can run in several processes. Co-authored-by: Kevin Longe <[email protected]> * docs(website): show race totals in time and say custom Rules run on the Rust engine - The race labels each linter with its total time instead of a percentage. - The lede describes the NULL example in words. - The custom Rules box says the Rust engine parses SQL and builds the facts, so Python only runs the Rule's logic. Co-authored-by: Kevin Longe <[email protected]> * test(e2e): parametrize the many-files custom Rules test fensu requires dataclass cases with an expected_ field and no conditionals in tests. Co-authored-by: Kevin Longe <[email protected]> * feat(cli): accept lint as an alias of check SQLFluff and sqruff users type lint. The alias is hidden from --help; check stays the documented name, and the check page mentions lint. Co-authored-by: Kevin Longe <[email protected]> * docs(website): shorten the hero lede Co-authored-by: Kevin Longe <[email protected]> * docs(website): serve the site from sqlclean.io Co-authored-by: Kevin Longe <[email protected]> * chore: trigger the first site preview build Co-authored-by: Kevin Longe <[email protected]> * feat(cli): add sqlclean skills to install a bundled agent skill The executable embeds a SQLClean skill with a copy of every docs page. make skills regenerates the copies from the website docs and make check-docs-skill fails while they differ. Co-authored-by: Kevin Longe <[email protected]> * ci: run only the suites a change needs and keep site edits out of releases Verify classifies a pull request by changed paths. Website, docs and benchmark edits run the docs skill check and ruff; bundled skill edits add a Linux build and the skills tests; anything else runs everything. The Verify job aggregates the selected suites. Release Please counts every Conventional Commit regardless of hidden changelog sections, so exclude-paths keeps website, benchmark and workflow-only commits from cutting a release. Co-authored-by: Kevin Longe <[email protected]> * test: write e2e project files with LF line endings on every platform Co-authored-by: Kevin Longe <[email protected]> ---------
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
SQLClean is SQLStack's standalone SQL linter and formatter, extracted from SQLBuild's engine so SQLBuild can depend on the crate and keep its own codes.
Changes
crates/sqlclean: built-in Rules (SQC###), the formatter and statement facts, with JSON entries for host tools.crates/sqlclean-cli:check,fix,format,rules,factsandskills, with config discovery, exceptions, a result cache and text, JSON or SARIF output.src/sqlclean: Python custom Rules (@rule,XSQCcodes, options, fixes), their host andsqlclean.testing.sqlclean skillsinstalls a bundled agent skill holding every docs page;make skillsregenerates it and CI fails while it is stale.website/(sqlclean.io, deployed by Cloudflare Pages) andbenchmarks/linters/(vs sqruff and SQLFluff).Verifyjob;exclude-pathskeeps website, benchmark and workflow-only commits out of releases. See.github/RELEASING.md.Verification
make verify: ruff, ty, fensu (0 faults), docs skill check, clippy, 52 Rust and 42 pytest tests.cargo +1.88 check, actionlint, website build with 0 broken links, hygiene scan clean.