Skip to content

HIVE-30200: Parquet: split generation globs every input directory, costing 3 object-store requests where 1 listing is enough - #6978

Open
deniskuzZ wants to merge 1 commit into
apache:masterfrom
deniskuzZ:HIVE-30200
Open

deniskuzZ wants to merge 1 commit into
apache:masterfrom
deniskuzZ:HIVE-30200

Conversation

@deniskuzZ

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

MapredParquetInputFormat.listStatus lists each input path with one listLocatedStatus call, on
hive.compute.splits.num.threads threads. It keeps the result of FileInputFormat.listStatus: hidden-file and job
filters, the recursive walk, subdirectories as entries when not recursive, and shrinkStatus.

Why are the changes needed?

FileInputFormat.listStatus globs every input path before listing it. On S3A the glob's getFileStatus is a HEAD plus
a LIST probe, so a partition directory costs three requests where its LIST alone carries the files. ORC's split
generation never globs; Trino lists a partition with one ListObjectsV2.

TPC-DS 10 TB q14 (12 map vertices listing date partitions at once) had a 16-20 s split-generation window per
execution, with every retry-backoff sample (SDK RetryableStage2, S3A Invoker) under the glob. Cutting the listing
threads from 64 to 10 left the window unchanged, so the request rate is the limit.

On a 10 x r5d cluster (TPC-DS 10 TB Parquet), the time with no running task went from 83.2 to 44.1 s per 99-query
pass; q75 -8.9 s, q14 -7.7 s.

Does this PR introduce any user-facing change?

No

How was this patch tested?

TestMapredParquetInputFormat:

  • returns the same statuses as FileInputFormat, recursive and not
  • no status call and one listing per listed path (the glob made 6 status calls for 3 paths)
  • removing the filter, the recursion or the fix each makes it fail

…sting 3 object-store requests where 1 listing is enough

- FileInputFormat.listStatus globs every input path before listing it; on S3A the glob's getFileStatus is a HEAD and
  a LIST probe, so a partition directory cost three requests where its LIST alone carries the files
- TPC-DS 10 TB q14 (12 map vertices listing date partitions at once): a 16-20 s split-generation window per
  execution, every retry-backoff sample (SDK RetryableStage2, S3A Invoker) under the glob; 10 listing threads instead
  of 64 leave the window unchanged, so the request rate is the limit
- MapredParquetInputFormat.listStatus lists each input path with one listLocatedStatus on
  hive.compute.splits.num.threads threads, keeping FileInputFormat's result: hidden-file and job filters, the
  recursive walk, subdirectories as entries when not recursive, shrinkStatus (ORC's split generation never globs;
  Trino lists a partition with one ListObjectsV2)
- TestMapredParquetInputFormat: the same statuses as FileInputFormat, recursive and not; no status call and one
  listing per listed path (the glob made 6 status calls for 3 paths); dropping the filter, the recursion or the fix
  each fails
- Impact (cluster, TPC-DS 10TB Parquet, 10 x r5d): the window with no running task 83.2 -> 44.1 s per 99-query pass;
  q75 -8.9 s, q14 -7.7 s
@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants