Skip to content

HIVE-30248: LLAP IO native Parquet cache should not fall back to the normal reader on an interrupt - #6955

Open
abstractdog wants to merge 4 commits into
apache:masterfrom
abstractdog:parquet-native-interrupt-no-fallback
Open

abstractdog wants to merge 4 commits into
apache:masterfrom
abstractdog:parquet-native-interrupt-no-fallback

Conversation

@abstractdog

@abstractdog abstractdog commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This is a follow-up to HIVE-30059 (#6793). When the native LLAP Parquet pipeline is interrupted while reading the footer, it now fails the split instead of falling back to the stock reader. It also makes Hive's limit-reached handling recognise all the forms that interrupt can take.

  • ParquetColumnVectorProducer.createReadPipeline rethrows the footer-read IOException unwrapped when it is an interrupt, and sets the thread's interrupt flag again. Any other footer failure falls back as before.
  • New JavaUtils.isInterrupt(Throwable). It returns true when the current thread's interrupt flag is set, or when the cause chain contains an InterruptedException, a ClosedByInterruptException or an InterruptedIOException. SocketTimeoutException doesn't count: it extends InterruptedIOException but is a timeout.
  • The limit-reached checks in HiveInputFormat.getRecordReader and LlapInputFormat.getRecordReader use JavaUtils.isInterrupt instead of matching only the root cause against InterruptedException or ClosedByInterruptException. That root-cause match missed an IOException wrapping an InterruptedIOException, which is how Hadoop clients report an interrupt, and a plain IOException thrown on an interrupted thread.
  • RecordReaderWrapper.create no longer turns every InterruptedIOException into an empty split (catch added in HIVE-24484). The exception now propagates to HiveInputFormat: once the limit is reached the split is served by a NullRowsRecordReader, and before that creating the reader fails.

Why are the changes needed?

When the native pipeline declines a split, LlapInputFormat.fallback() serves it with the stock VectorizedParquetRecordReader. HIVE-30059 declined the split on every footer-read IOException, interrupts included. An interrupt means the fragment is being killed or the daemon is shutting down, so falling back just starts the same read again on a thread that should be stopping. It also logs a WARN saying the split lost its vectored reads, which hides what actually happened.

Rethrowing the interrupt exposed two problems in the code above the producer:

  • RecordReaderWrapper returned an empty split for any InterruptedIOException without checking the limit. A SocketTimeoutException counted as one too. A task could therefore finish with a split's rows missing.
  • After the limit was reached, some interrupt forms still failed reader creation instead of producing an empty split.

Does this PR introduce any user-facing change?

No change to results. Before the limit is reached, an InterruptedIOException or SocketTimeoutException while creating a reader now fails the task instead of quietly returning an empty split. This applies to every input format.

How was this patch tested?

New unit tests:

  • TestParquetEncodedDataReader. These use a filesystem whose getFileStatus throws a chosen exception:
    • testInterruptedFooterReadIsNotAFallback: an InterruptedIOException, a ClosedByInterruptException and an IOException wrapping an InterruptedException are each rethrown as the same object, with the interrupt flag set.
    • testFailedFooterReadOnAnInterruptedThreadIsNotAFallback: a plain IOException on an interrupted thread is rethrown.
    • testFailedFooterReadFallsBack (control): a plain IOException and a SocketTimeoutException still fall back.
  • TestHiveInputFormatInterruptedReader. These go through HiveInputFormat.getRecordReader, with the limit flag set in the Tez object cache:
    • once the limit is reached, every interrupt form produces an empty split, including a plain IOException on an interrupted thread;
    • before the limit is reached, an interrupt fails reader creation;
    • after the limit is reached, a plain IOException and a SocketTimeoutException still fail.

Without the production changes, the first two producer tests and all four HiveInputFormat tests fail.

Existing tests still pass:

mvn -pl ql test -Dtest=TestHiveInputFormatInterruptedReader,TestCombineHiveInputFormat,TestLimitOperator,TestVectorLimitOperator
mvn -pl llap-server test -Dtest=TestParquetEncodedDataReader,TestParquetCachedPageReadStore,TestLowLevelCacheImpl,TestLlapCacheAwareFsInterop,TestLlapRecordReader,TestVectoredReadBuffers,TestParquetNativeIOCacheDisabled,TestOrcEncodedDataReaderIOCacheDisabled

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Inconsistent interruption handling can silently convert non-limit failures into empty splits while still failing reader creation for some limit-reached interruptions.

2 open issues
What changed in this PR

Updates LLAP’s native Parquet cache path from HIVE-30059 to stop falling back to the normal reader when a footer read is interrupted.

Changes:

  • Detects interruptions through the thread flag or exception causes, while excluding socket timeouts alone.
  • Rethrows the original exception, restores the interrupt flag, and logs at INFO; ordinary failures retain fallback behavior.
  • Adds producer-level tests for interruption propagation and normal fallback.
File Description
llap-server/​src/​test/​org/​apache/​hadoop/​hive/​llap/​io/​encoded/​TestParquetEncodedDataReader.java Adds footer-failure tests using a filesystem that throws controlled exceptions.
llap-server/​src/​java/​org/​apache/​hadoop/​hive/​llap/​io/​decode/​ParquetColumnVectorProducer.java Classifies footer interruptions and propagates them instead of requesting fallback.

Review effort: Balanced

}
for (Throwable t = e; t != null; t = t.getCause() == t ? null : t.getCause()) {
if (t instanceof InterruptedException || t instanceof ClosedByInterruptException
|| (t instanceof InterruptedIOException && !(t instanceof SocketTimeoutException))) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checking this

LlapIoImpl.LOG.info("Parquet native cache: interrupted while reading the footer of {}, "
+ "not falling back to normal reader", split.getPath());
Thread.currentThread().interrupt();
throw e;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checking this

Comment on lines -75 to +76
RecordReader innerReader = null;
try {
innerReader = inputFormat.getRecordReader(split.getInputSplit(), jobConf, reporter);
} catch (InterruptedIOException iioe) {
// If reading from the underlying record reader is interrupted, return a no-op record reader
LOG.info("Interrupted while getting the input reader for {}", split.getInputSplit());
return new ZeroRowsInputFormat().getRecordReader(split.getInputSplit(), jobConf, reporter);
}
// An interrupt propagates: HiveInputFormat turns it into an empty split only once the limit is
// reached, and otherwise the task must fail rather than finish with this split's rows missing.
RecordReader innerReader = inputFormat.getRecordReader(split.getInputSplit(), jobConf, reporter);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removing this is safe: HiveInputFormat.getRecordReader is the only caller of RecordReaderWrapper.create, so every InterruptedIOException now reaches the HIVE-25751 handler, which uses JavaUtils.isInterrupt, so what can happen is:

  • Limit reached and the error is an interrupt (including InterruptedIOException): NullRowsRecordReader, the case the qtests needed. Same outcome as before.
  • Limit not reached: creating the reader fails. For a task that's being killed this changes nothing about results, only that it fails with an exception instead of finishing with zero rows.
  • SocketTimeoutException: always fails the task, as an I/O error should.

abstractdog and others added 2 commits November 20, 2026 11:55
…normal reader on an interrupt

ParquetColumnVectorProducer.createReadPipeline declined the split on any
IOException from the footer read, so LlapInputFormat handed it to the stock
VectorizedParquetRecordReader. When that IOException is an interrupt (the
fragment being killed, the daemon shutting down), the fallback just restarts
the same read on the stock reader, on a thread that should be unwinding.

An interrupt is now rethrown unchanged instead of declining the split:
InterruptedIOException (but not SocketTimeoutException, which is a timeout),
ClosedByInterruptException, an InterruptedException anywhere in the cause
chain, or any IOException on a thread whose interrupt flag is set. The
exception is not rewrapped, so HiveInputFormat's limit-reached bailout still
finds the interrupt as the root cause, and the interrupt flag is restored in
case a wrapper consumed it. Other footer failures still fall back as before.

Follow-up to HIVE-30059.
- Move the interrupt classification to JavaUtils.isInterrupt and use it in
  the limit-reached checks of HiveInputFormat and LlapInputFormat too, so an
  interrupt the producer now rethrows (a wrapped InterruptedIOException, or
  any IOException on an interrupted thread) still becomes an empty split
  once the limit is reached.
- RecordReaderWrapper no longer turns an InterruptedIOException into an
  empty split regardless of the limit; HiveInputFormat decides, and before
  the limit is reached the interrupt fails the reader.
- Walk the cause chain with plain getCause(), which already returns null for
  a self-cause.
- Add TestHiveInputFormatInterruptedReader.

Co-Authored-By: Claude Code <[email protected]>
@abstractdog
abstractdog force-pushed the parquet-native-interrupt-no-fallback branch from 7dbb85a to 98138e5 Compare November 20, 2026 10:55

@deniskuzZ deniskuzZ left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

// prevent instantiation
}

public static Throwable findRootCause(Throwable throwable) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

findRootCause has no callers left after this PR

if (Thread.currentThread().isInterrupted()) {
return true;
}
for (Throwable t = throwable; t != null; t = t.getCause()) {

@deniskuzZ deniskuzZ Nov 20, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: commons-lang3's ExceptionUtils.stream already walks the cause chain, and unlike this loop it stops on a cause cycle:

 /**                                                                                                                                                                                                          
     * Whether a failure is the current thread being interrupted rather than the operation failing on its own.                                                                                                   
     * The whole cause chain is checked, not just the root cause, because Hadoop's IPC and filesystem clients                                                                                                    
     * wrap an interrupt in an InterruptedIOException of their own.                                                                                                                                              
     */                                                                                                                                                                                                          
    public static boolean isInterrupt(Throwable throwable) {                                                                                                                                                     
      return Thread.currentThread().isInterrupted()                                                                                                                                                              
          || ExceptionUtils.stream(throwable).anyMatch(JavaUtils::isInterruptType);                                                                                                                              
    }                                                                                                                                                                                                            
                                                                                                                                                                                                                 
    // A SocketTimeoutException is an InterruptedIOException too, but it is a timeout, not an interrupt.                                                                                                         
    private static boolean isInterruptType(Throwable t) {                                                                                                                                                        
      return t instanceof InterruptedException                                                                                                                                                                   
          || t instanceof ClosedByInterruptException                                                                                                                                                             
          || (t instanceof InterruptedIOException && !(t instanceof SocketTimeoutException));                                                                                                                    
    }           

The javadoc could drop the list of exception types, since the code shows them. The two sentences on why it walks the whole chain and why it excludes SocketTimeoutException are worth keeping.

@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants