Repository navigation
Conversation
…sting 3 object-store requests where 1 listing is enough - FileInputFormat.listStatus globs every input path before listing it; on S3A the glob's getFileStatus is a HEAD and a LIST probe, so a partition directory cost three requests where its LIST alone carries the files - TPC-DS 10 TB q14 (12 map vertices listing date partitions at once): a 16-20 s split-generation window per execution, every retry-backoff sample (SDK RetryableStage2, S3A Invoker) under the glob; 10 listing threads instead of 64 leave the window unchanged, so the request rate is the limit - MapredParquetInputFormat.listStatus lists each input path with one listLocatedStatus on hive.compute.splits.num.threads threads, keeping FileInputFormat's result: hidden-file and job filters, the recursive walk, subdirectories as entries when not recursive, shrinkStatus (ORC's split generation never globs; Trino lists a partition with one ListObjectsV2) - TestMapredParquetInputFormat: the same statuses as FileInputFormat, recursive and not; no status call and one listing per listed path (the glob made 6 status calls for 3 paths); dropping the filter, the recursion or the fix each fails - Impact (cluster, TPC-DS 10TB Parquet, 10 x r5d): the window with no running task 83.2 -> 44.1 s per 99-query pass; q75 -8.9 s, q14 -7.7 s
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



What changes were proposed in this pull request?
MapredParquetInputFormat.listStatuslists each input path with onelistLocatedStatuscall, onhive.compute.splits.num.threadsthreads. It keeps the result ofFileInputFormat.listStatus: hidden-file and jobfilters, the recursive walk, subdirectories as entries when not recursive, and
shrinkStatus.Why are the changes needed?
FileInputFormat.listStatusglobs every input path before listing it. On S3A the glob'sgetFileStatusis a HEAD plusa LIST probe, so a partition directory costs three requests where its LIST alone carries the files. ORC's split
generation never globs; Trino lists a partition with one ListObjectsV2.
TPC-DS 10 TB q14 (12 map vertices listing date partitions at once) had a 16-20 s split-generation window per
execution, with every retry-backoff sample (SDK
RetryableStage2, S3AInvoker) under the glob. Cutting the listingthreads from 64 to 10 left the window unchanged, so the request rate is the limit.
On a 10 x r5d cluster (TPC-DS 10 TB Parquet), the time with no running task went from 83.2 to 44.1 s per 99-query
pass; q75 -8.9 s, q14 -7.7 s.
Does this PR introduce any user-facing change?
No
How was this patch tested?
TestMapredParquetInputFormat:FileInputFormat, recursive and not