Repository navigation
feat: named transforms, ODBC fetch buffer, peak memory, typed cursor flags, plan-time path checks and streamed --follow - #260
Merged
Merged
Conversation
…med @sl.transform function before they load
@sl.transform registers a function under its own name, or under the one
it is given (@sl.transform("kpis")). Discovery imports python_paths, and
a files or google_sheets table names the function with transform =
"name"; sl.http's paginate and fan_out take transform= with a name or
the function itself. Plan fails on a name no function has, listing the
registered ones, and on a name two functions have, naming both files.
Each unit gets one call, f(rows, ctx), with its rows as a pyarrow Table:
a file or ZIP member, a Sheets range, or an HTTP page after select,
fields and cells. ctx.unit is "file", "range" or "page", with that
unit's path, range or page number and fan-out key. The function returns
or yields, in any mix, what a Python table may yield, and yielded
batches load as they come, so memory stays near one unit and its
output. A unit whose function fails fails the table, uncommitted and,
for files, not moved by after_load. Plan takes the columns from what the
function returns for the newest file or the range. A table with a
transform is read by SQLLoad, never directly by the destination, and the
standalone executable refuses it.
Each commit records the transform's version (a hash of its file, or of
its code when the file can't be read) and the files it ran on, in the
table's run log, created by apply. Plan warns while the table holds
files an earlier version loaded, and the load report gives each file's
transform time.
--reload now works on files tables that merge into DuckDB: each file
merges on its own into a new table beside the old one, in the order the
run log merged them, then new files oldest first, and the new table
replaces the old one with its run log records in one commit. With
after_load = { move_to = ... }, --reload also reads the files under
move_to, from where they were moved, and plan shows both folders. After
the swap, files read from path move as usual and archived files stay.
…buffer, and show it in plan A text or binary column whose buffer would pass long_column_threshold (4 KiB by default), or that the driver gives no length for, makes its table read value by value with SQLGetData: one row at a time, each value at its own size, never cut. Block cursors are avoided because reading values from a block of rows breaks on MariaDB Connector/ODBC 3.1 and on Microsoft ODBC Driver 18. Tables without such a column still fetch in bulk. Each table has a fetch buffer, the most the rows of one fetch hold: 512 MiB, lowered to half the memory limit (the cgroup's memory.max in a container, else the machine's memory) shared between the tables loading at once, and never raised unless the source sets max_batch_memory. A bulk fetch reads as many rows as its buffers hold within it; a value-by-value fetch ends once its values reach it, and a single value larger than the buffer loads, ends its fetch and is logged. Commits are unchanged. The fetch buffer bounds the fetches only, not the load's whole memory. plan shows each ODBC table's fetch and fetch buffer, and warns when the fetch buffers of the tables loading at once together pass half the limit; load checks again.
…-memory-per-table apply --json and load report the most memory the whole process held at once: peak_memory_bytes in the JSON summary and the run_finished event, and a "peak memory" branch under load's summary. It is ru_maxrss on Linux and macOS and the peak working set on Windows. load --memory-per-table (Linux only) loads one table at a time, resets the high-water mark before each table and reads VmHWM after it, and reports each table's peak in the JSON report, its table_done or table_failed event and its tally line. Elsewhere plan refuses the flag. The library denies unsafe code instead of forbidding it, so the one Windows call, GetProcessMemoryInfo, can allow it in its own module.
…lags as in SQLBuild, and an integer lookback on integer cursors plan, apply and load take --start-cursor-ts/--end-cursor-ts for date and timestamp cursors and --start-cursor-int/--end-cursor-int for integer cursors. As in sqb, the -ts and -int flags can be given together, each table taking its cursor's family, and the untyped flags can't be combined with them. A table whose cursor gets only the other family's flags, a text cursor, or a date cursor given a time of day fails plan. lookback takes a number of cursor units on an integer cursor, so a fiscal_year table can reload last year after the year-end close. plan refuses a lookback that doesn't fit the cursor.
… a bad path fails plan instead of load Discovery reads each table function's source with ast, without calling it, and collects the literal paths of its paginate and fan_out jobs: fields= values, the paths in sl.field and sl.parent, and key_path=. Calls count only when their names resolve to sqlload, however it was imported. plan checks those paths with load's own checker, so a bad one fails plan with load's message, naming the table, field and path. A computed path, or a function whose source can't be read, stays checked at load, and plan says so in one line on that table.
…live block as load load --follow now goes through the same streamed output as load: the one-line plan, a "written" line for each table in each batch with how far it is behind the server, the loaded lines and the closing summary. On a terminal it draws the live block, marked FOLLOWING, with log lines printed above it. Piped output has no escape codes, and --json is unchanged. A plan with errors still refuses to start, now under the new plan view.
…erminal or NO_COLOR is set Log lines went to stderr in colour even when it was piped or redirected, so a CI log or a captured run held ANSI escape codes between every field. Colour is now used only on a terminal, and never under NO_COLOR, as the rest of the output already does.
Deploying sqlload with
|
| Latest commit: |
943f18c
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://9df3a80a.sqlload.pages.dev |
| Branch Preview URL: | https://wip-features-cli-4.sqlload.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Six features, one commit each.
Transform files, Sheets ranges and HTTP pages with a named
@sl.transformfunction (feat(python))@sl.transformregisters a function under its own name, or under the name given to@sl.transform("kpis"). The TOML picks one withtransform = "kpi_sheet"; Python can also pass the function itself.sl.httppage, throughpaginate(..., transform=f)orfan_out(..., transform=f).f(rows, ctx), withctx.unitset to"file","range"or"page". The function may return or yield dicts, Arrow or DataFrames, and a generator's batches stream to the load.transform changed: N files loaded under an earlier version (--reload transforms every file again)._sqlload_runs, so a load role needs no CREATE.--reloadnow works on merge andmove_tofiles tables:move_toreload also reads the archived files, through SQLLoad rather than the warehouse's direct route.after_loadis still refused.unnestload into Postgres already reaches about 1.77 MiB throughPublisher::commit→ChildTables::stage→ the Postgresexecute. With the transform's extra frames, three Postgresunnesttests overflowed 2 MiB. Boxing a future can't fix depth, so the runtime setting is the fix. The measurements are in theruntime()doc comment.ODBC: long columns read value by value under a per-table fetch buffer (
feat(odbc))long_column_threshold(4 KB), or an unbounded one, reads every column value by value at one row per fetch. Values are never truncated.max_batch_memory.fetchandbufferlines, and warns when the tables' fetch buffers together near the memory limit. The warning says it counts only those buffers.Peak memory (
feat(load))peak_memory_bytesappears in the--jsonsummary and in therun_finishedevent, and the tally shows└── peak memory 85.4 MiB.--memory-per-table(Linux only) runs one table at a time and reports each table's peak. Windows reads the peak through a small module allowedunsafe.Typed cursor flags and integer lookback (
feat(cursor))--start-cursor-ts/--end-cursor-tsand--start-cursor-int/--end-cursor-int, as in SQLBuild.-tsand-intcan be combined; untyped and typed flags can't, and are refused with SQLBuild's wording.lookback = 1on an integer cursor givesreads fiscal_year >= 2025 (warehouse maximum 2026 minus lookback 1). The start is clamped to the column type's minimum.Plan checks
sl.httpfield paths written as literals (feat(python))load --followuses the new load output (feat(load))load.--followstill refuses to start on plan errors, and stops with the real error when a re-plan fails.Log colour (
fix(cli))Log lines were written in colour to a piped or redirected stderr, so CI logs and captured runs held ANSI codes between fields (the first CI run of this PR failed on exactly that: a test parsing
rows=from the log found nothing). Colour is now used only on a terminal withoutNO_COLOR, like the rest of the output. The ODBC memory test failed withNO_COLORunset on 361379e and passes on 943f18c.Checks
make verifyon this head (361379e), withRUST_MIN_STACKunset: exit 0. That covers fmt, clippy (all variants), 316 lib tests, 15 CLI tests and 110 container loader tests.sl.httpjob keeping its transform across two loads in one process; direct Parquet loads into ClickHouse writing real values, not NULLs; an archived reload reading the archived copy; merge--reloadwithmove_tokeeping the newest file.unsafe.BEGIN_COMMIT_OVERRIDE
feat(python): transform files, Sheets ranges and HTTP pages with a named @sl.transform function before they load, and --reload merge and move_to files tables
feat(odbc): read long columns value by value under a per-table fetch buffer, and show it in plan
feat(load): report peak memory per run, and per table on Linux with --memory-per-table
feat(cursor): typed --start-cursor-ts/-int and --end-cursor-ts/-int flags as in SQLBuild, and an integer lookback on integer cursors
feat(python): plan checks sl.http field paths written as literals, so a bad path fails plan instead of load
feat(load): load --follow prints the same plan line, table lines and live block as load
fix(cli): write log lines without colour codes when stderr is not a terminal or NO_COLOR is set
END_COMMIT_OVERRIDE