Repository navigation
fix(cache): release memory a liquid array holds beyond its charge - #61
Merged
Merged
Conversation
The cache charges an in-memory liquid entry by its Vortex nbytes(), the sum of its buffers' lengths. Two kinds of compressor output keep more memory alive than that: - A constant string column stores a scalar that is a zero-copy slice of the whole source views or data buffer (vortex-data/vortex#10599, fixed upstream in #11382 after our pinned revision). - An FSST codes buffer grows by doubling and is frozen with its spare capacity (vortex-data/vortex#11420). Both are freed only with the entry, so the cache can hold close to twice what it counts. Round-trip each compressed array through the IPC serialization the disk tier already uses, which detaches it from both allocations.
anoop-narang
requested review from
eddietejeda
and removed request for
a team
November 12, 2026 20:33
anoop-narang
changed the title
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
📊 Benchmark ComparisonCurrent:
Compared Liquid vs DataFusionDefault on the same runner |
Collaborator
Author
|
The TPC-DS failure is unrelated to this change, and it reproduced on a rerun. It fails in "Setup TPC-DS data" before any Rust runs. The runner now has Python 3.14, whose argparse rejects |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Each array
LiquidArray::from_arrow_arraycompresses is round-tripped throughto_bytes/from_bytes, the IPC serialization the disk tier already uses. That detaches the entry from its source column's buffers and from the compressor's spare capacity.Why
The cache charges an in-memory liquid entry by Vortex
nbytes(), which sums buffer lengths. Two kinds of compressor output hold more than that:ConstantArraywhose scalar is a zero-copy slice of the source column's whole views buffer (strings of 12 bytes or fewer) or data buffer. An 8192-row column is charged a few bytes but holds about 128 KiB. Reported upstream as Writer holds every uncompressed chunk of a constant string column until the file ends (ConstantArray scalar slices its chunk; BufferedStrategy counts nbytes) vortex-data/vortex#10599 and fixed in fix(compressor): copy constant string and binary values out of the source buffer vortex-data/vortex#11382, which is newer than the Vortex revision this repo pins.Both are freed only when the entry is, so a full cache can hold much more than its budget.
Effect
Measured over 200 batches × 25 columns of a real 540 MB Parquet file (log-style data with many string columns):
The round trip took 135 ms against 3,820 ms of compression for the same batches (about 3.5%). Most of the remaining gap is per-node overhead (symbol tables, tree nodes) that
nbytes()also does not count.Tests
constant_strings_do_not_pin_the_source_buffers: after compressing a constant column and dropping the source, the source's views and data buffers are uniquely owned again. Fails without the fix.fsst_codes_growth_allocation_is_released: keeps a handle to the compressor's FSST codes buffer (len 544,535, capacity 1,048,577 in this fixture), compacts, and requires that handle to be uniquely owned again. Fails without the fix.Both pass with the fix, along with
cargo test --workspace --exclude dev-tools.The charge stays approximate:
nbytes()still leaves out per-node overhead such as tree nodes and decoded FSST symbol tables.Once the Vortex pin includes #11382 and a fix for #11420, this round trip can be reconsidered.
This change was investigated and written with AI assistance (Claude Code).