Repository navigation
HIVE-30248: LLAP IO native Parquet cache should not fall back to the normal reader on an interrupt - #6955
HIVE-30248: LLAP IO native Parquet cache should not fall back to the normal reader on an interrupt#6955abstractdog wants to merge 4 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Inconsistent interruption handling can silently convert non-limit failures into empty splits while still failing reader creation for some limit-reached interruptions.
2 open issues
What changed in this PR
Updates LLAP’s native Parquet cache path from HIVE-30059 to stop falling back to the normal reader when a footer read is interrupted.
Changes:
- Detects interruptions through the thread flag or exception causes, while excluding socket timeouts alone.
- Rethrows the original exception, restores the interrupt flag, and logs at INFO; ordinary failures retain fallback behavior.
- Adds producer-level tests for interruption propagation and normal fallback.
| File | Description |
|---|---|
llap-server/src/test/org/apache/hadoop/hive/llap/io/encoded/TestParquetEncodedDataReader.java |
Adds footer-failure tests using a filesystem that throws controlled exceptions. |
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ParquetColumnVectorProducer.java |
Classifies footer interruptions and propagates them instead of requesting fallback. |
Review effort: Balanced
| } | ||
| for (Throwable t = e; t != null; t = t.getCause() == t ? null : t.getCause()) { | ||
| if (t instanceof InterruptedException || t instanceof ClosedByInterruptException | ||
| || (t instanceof InterruptedIOException && !(t instanceof SocketTimeoutException))) { |
| LlapIoImpl.LOG.info("Parquet native cache: interrupted while reading the footer of {}, " | ||
| + "not falling back to normal reader", split.getPath()); | ||
| Thread.currentThread().interrupt(); | ||
| throw e; |
258c50d to
f706beb
Compare
f706beb to
4cf63df
Compare
| RecordReader innerReader = null; | ||
| try { | ||
| innerReader = inputFormat.getRecordReader(split.getInputSplit(), jobConf, reporter); | ||
| } catch (InterruptedIOException iioe) { | ||
| // If reading from the underlying record reader is interrupted, return a no-op record reader | ||
| LOG.info("Interrupted while getting the input reader for {}", split.getInputSplit()); | ||
| return new ZeroRowsInputFormat().getRecordReader(split.getInputSplit(), jobConf, reporter); | ||
| } | ||
| // An interrupt propagates: HiveInputFormat turns it into an empty split only once the limit is | ||
| // reached, and otherwise the task must fail rather than finish with this split's rows missing. | ||
| RecordReader innerReader = inputFormat.getRecordReader(split.getInputSplit(), jobConf, reporter); |
There was a problem hiding this comment.
removing this is safe: HiveInputFormat.getRecordReader is the only caller of RecordReaderWrapper.create, so every InterruptedIOException now reaches the HIVE-25751 handler, which uses JavaUtils.isInterrupt, so what can happen is:
- Limit reached and the error is an interrupt (including InterruptedIOException): NullRowsRecordReader, the case the qtests needed. Same outcome as before.
- Limit not reached: creating the reader fails. For a task that's being killed this changes nothing about results, only that it fails with an exception instead of finishing with zero rows.
- SocketTimeoutException: always fails the task, as an I/O error should.
0fd2f00 to
7dbb85a
Compare
…normal reader on an interrupt ParquetColumnVectorProducer.createReadPipeline declined the split on any IOException from the footer read, so LlapInputFormat handed it to the stock VectorizedParquetRecordReader. When that IOException is an interrupt (the fragment being killed, the daemon shutting down), the fallback just restarts the same read on the stock reader, on a thread that should be unwinding. An interrupt is now rethrown unchanged instead of declining the split: InterruptedIOException (but not SocketTimeoutException, which is a timeout), ClosedByInterruptException, an InterruptedException anywhere in the cause chain, or any IOException on a thread whose interrupt flag is set. The exception is not rewrapped, so HiveInputFormat's limit-reached bailout still finds the interrupt as the root cause, and the interrupt flag is restored in case a wrapper consumed it. Other footer failures still fall back as before. Follow-up to HIVE-30059.
- Move the interrupt classification to JavaUtils.isInterrupt and use it in the limit-reached checks of HiveInputFormat and LlapInputFormat too, so an interrupt the producer now rethrows (a wrapped InterruptedIOException, or any IOException on an interrupted thread) still becomes an empty split once the limit is reached. - RecordReaderWrapper no longer turns an InterruptedIOException into an empty split regardless of the limit; HiveInputFormat decides, and before the limit is reached the interrupt fails the reader. - Walk the cause chain with plain getCause(), which already returns null for a self-cause. - Add TestHiveInputFormatInterruptedReader. Co-Authored-By: Claude Code <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
7dbb85a to
98138e5
Compare
| // prevent instantiation | ||
| } | ||
|
|
||
| public static Throwable findRootCause(Throwable throwable) { |
There was a problem hiding this comment.
findRootCause has no callers left after this PR
| if (Thread.currentThread().isInterrupted()) { | ||
| return true; | ||
| } | ||
| for (Throwable t = throwable; t != null; t = t.getCause()) { |
There was a problem hiding this comment.
Nit: commons-lang3's ExceptionUtils.stream already walks the cause chain, and unlike this loop it stops on a cause cycle:
/**
* Whether a failure is the current thread being interrupted rather than the operation failing on its own.
* The whole cause chain is checked, not just the root cause, because Hadoop's IPC and filesystem clients
* wrap an interrupt in an InterruptedIOException of their own.
*/
public static boolean isInterrupt(Throwable throwable) {
return Thread.currentThread().isInterrupted()
|| ExceptionUtils.stream(throwable).anyMatch(JavaUtils::isInterruptType);
}
// A SocketTimeoutException is an InterruptedIOException too, but it is a timeout, not an interrupt.
private static boolean isInterruptType(Throwable t) {
return t instanceof InterruptedException
|| t instanceof ClosedByInterruptException
|| (t instanceof InterruptedIOException && !(t instanceof SocketTimeoutException));
}
The javadoc could drop the list of exception types, since the code shows them. The two sentences on why it walks the whole chain and why it excludes SocketTimeoutException are worth keeping.
|





What changes were proposed in this pull request?
This is a follow-up to HIVE-30059 (#6793). When the native LLAP Parquet pipeline is interrupted while reading the footer, it now fails the split instead of falling back to the stock reader. It also makes Hive's limit-reached handling recognise all the forms that interrupt can take.
ParquetColumnVectorProducer.createReadPipelinerethrows the footer-readIOExceptionunwrapped when it is an interrupt, and sets the thread's interrupt flag again. Any other footer failure falls back as before.JavaUtils.isInterrupt(Throwable). It returns true when the current thread's interrupt flag is set, or when the cause chain contains anInterruptedException, aClosedByInterruptExceptionor anInterruptedIOException.SocketTimeoutExceptiondoesn't count: it extendsInterruptedIOExceptionbut is a timeout.HiveInputFormat.getRecordReaderandLlapInputFormat.getRecordReaderuseJavaUtils.isInterruptinstead of matching only the root cause againstInterruptedExceptionorClosedByInterruptException. That root-cause match missed anIOExceptionwrapping anInterruptedIOException, which is how Hadoop clients report an interrupt, and a plainIOExceptionthrown on an interrupted thread.RecordReaderWrapper.createno longer turns everyInterruptedIOExceptioninto an empty split (catch added in HIVE-24484). The exception now propagates toHiveInputFormat: once the limit is reached the split is served by aNullRowsRecordReader, and before that creating the reader fails.Why are the changes needed?
When the native pipeline declines a split,
LlapInputFormat.fallback()serves it with the stockVectorizedParquetRecordReader. HIVE-30059 declined the split on every footer-readIOException, interrupts included. An interrupt means the fragment is being killed or the daemon is shutting down, so falling back just starts the same read again on a thread that should be stopping. It also logs a WARN saying the split lost its vectored reads, which hides what actually happened.Rethrowing the interrupt exposed two problems in the code above the producer:
RecordReaderWrapperreturned an empty split for anyInterruptedIOExceptionwithout checking the limit. ASocketTimeoutExceptioncounted as one too. A task could therefore finish with a split's rows missing.Does this PR introduce any user-facing change?
No change to results. Before the limit is reached, an
InterruptedIOExceptionorSocketTimeoutExceptionwhile creating a reader now fails the task instead of quietly returning an empty split. This applies to every input format.How was this patch tested?
New unit tests:
TestParquetEncodedDataReader. These use a filesystem whosegetFileStatusthrows a chosen exception:testInterruptedFooterReadIsNotAFallback: anInterruptedIOException, aClosedByInterruptExceptionand anIOExceptionwrapping anInterruptedExceptionare each rethrown as the same object, with the interrupt flag set.testFailedFooterReadOnAnInterruptedThreadIsNotAFallback: a plainIOExceptionon an interrupted thread is rethrown.testFailedFooterReadFallsBack(control): a plainIOExceptionand aSocketTimeoutExceptionstill fall back.TestHiveInputFormatInterruptedReader. These go throughHiveInputFormat.getRecordReader, with the limit flag set in the Tez object cache:IOExceptionon an interrupted thread;IOExceptionand aSocketTimeoutExceptionstill fail.Without the production changes, the first two producer tests and all four
HiveInputFormattests fail.Existing tests still pass: