Repository navigation
GH-34319: [R] Fast path for plain R metadata sanitization - #55673
Draft
Aurantium-47 wants to merge 1 commit into
Draft
Aurantium-47 wants to merge 1 commit into
Aurantium-47 wants to merge 1 commit into
Conversation
Check the whole metadata tree before returning plain safe metadata unchanged; retain the existing filtering and error path for all other shapes. AI-assisted implementation and review. The supplied benchmark documents both the narrow-projection gain and the late-fallback regression; no human review is claimed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Related to #34319; separate from attribute-restoration PR #55668.
@thisisnic's analysis in #34319 already identified deserializing and sanitizing the full R metadata as the main cost when selecting only a few columns from a wide file. That comment explicitly discloses AI-assisted analysis. This draft investigates that first work item; it does not claim to discover the bottleneck or bypass metadata checks for unselected columns.
The trade-off is material: remeasurement of the final head improves the synthetic narrow-projection benchmarks by 3.12× (20 columns) and 2.93× (200 columns), while metadata with a complex attribute in the final column makes the sanitizer-only fallback 20.7% slower. This draft is for review of that trade-off and the safety boundary, not a claim that every metadata workload improves.
What changes are included in this PR?
on_saveargument evaluation and avoid dispatching list class methods during preflight.on_saveevaluation.The original protection in #41969 and list-subclass handling in #43895 remain in place. There is no API or file-format change.
Performance, including the regression
Final-head remeasurement
The R code at final head
b5773cd280674c0ce0816c8e2e601c0d3ba554a3was remeasured with R 4.5.3 and the Arrow native library held fixed at 25.0.1. Each workload below used eight alternating original/candidate trials, with matching outputs. The file-read workloads use a synthetic 16,000-column Parquet file; attribute restoration is held fixed, so these results do not combine this patch with #55668.The last row measures the sanitizer alone: a late fallback pays for both the preflight and the original walk. The positive and negative results are both part of the trade-off. This rerun is not a full current-main native/source package build or a real production workload, and it does not resolve the other unvalidated items listed below.
Historical measurements (retained for traceability)
Linux x86_64, R 4.5.3, conda-forge Arrow 25.0.1 native library. Synthetic 16,000-column × 10-row Parquet file, with
textandlevelsattributes on each column. Original attribute restoration is held fixed; this is not a measurement of #55668 combined with this patch. Six alternating trials per end-to-end case, garbage collection before each trial, output equality checked.The last two rows use three trials. A late fallback pays for both the preflight and the original walk. Batched, warmed small-input tests likewise show fallback regressions of about 9%, 15% and 21% at 1, 20 and 200 columns respectively. These are synthetic examples, not a general Arrow speedup claim.
ASCII metadata decode plus sanitization measured 1.835 → 0.514 s; gzip-wrapped decode plus sanitization measured 1.727 → 0.452 s (three trials each, identical results). These are metadata-only measurements, not full compressed-Parquet reads.
Recorded allocations for the wide sanitizer decreased from 640,288 to 384,192 bytes on the plain path but increased to 1,024,480 bytes on the fallback path. Whole-process peak RSS did not improve: approximately 167 → 168 MiB on the plain path and 167 → 170 MiB on fallback, one isolated process per condition. No memory improvement is claimed.
The timing series preceded the final fixes for rejecting custom classes without calling
names(x)and preservingon_saveevaluation; the extended sanitizer/compression series includes the former fix. Correctness and the peak-RSS measurements were rerun on the final code. These historical timings should be reconfirmed by CI/review rather than treated as final-code release benchmarks.Are these changes tested?
Passed locally on the final candidate:
stop()fixture was not executed.r/R/metadata.R.Review caught and corrected three initial problems: replacing recursive
purrr::map()changed indexed error context; the first guard could invokenames()methods on a classed list; and the initial early return skippedon_saveevaluation. The first design was discarded; the latter two have explicit regressions.Not yet validated:
R CMD check.Was AI used for this PR?
AI was used for investigation, implementation, differential review, regression tests, benchmark execution and this description. No human review is claimed.
PR code and description written by:
Reviewed before submission by: