Repository navigation
GH-55669: [C++][IPC] Validate buffer bounds and replace raw pointer arithmetic in MessageDecoder - #55671
GH-55669: [C++][IPC] Validate buffer bounds and replace raw pointer arithmetic in MessageDecoder#55671Awesome06 wants to merge 5 commits into
Conversation
…in MessageDecoder * Replace raw pointer arithmetic and unchecked memcpy offsets in MessageDecoderImpl with std::span and AddWithOverflow * Enforce validation against negative data size in MessageDecoder::Consume * Add opt-in max_metadata_length, max_body_length, and max_decompression_ratio ceilings to IpcReadOptions (defaulting to unlimited: -1) * Use CHECK_FLATBUFFERS_NOT_NULL guards in Tensor metadata decoders * Add deterministic unit tests covering negative sizes, opt-in limits, and decompression bounds in read_write_test
There was a problem hiding this comment.
🟡 Changes recommended
The change still has an unresolved critical gap where configured IPC size limits are ignored on file-reader paths, plus a decompression-ratio boundary bug that allows unlimited expansion when set to zero.
2 open issues
What changed in this PR
This PR hardens C++ IPC decoding against malformed or oversized input by validating message lengths, protecting chunk reassembly, checking FlatBuffers fields, and adding configurable decompression and allocation limits.
Changes:
- Adds bounds validation for decoder input, metadata, body lengths, and chunk copying.
- Adds null checks for tensor and custom metadata FlatBuffers fields.
- Adds configurable metadata, body, and decompression-ratio limits to
IpcReadOptions, with associated tests.
| File | Description |
|---|---|
| cpp/src/arrow/ipc/reader.cc | Adds decompression-ratio validation and propagates read options to streaming decoding. |
| cpp/src/arrow/ipc/reader_internal.h | Exposes the internal decompression helper declaration. |
| cpp/src/arrow/ipc/read_write_test.cc | Adds tests for decoder bounds, metadata validation, body limits, and decompression limits. |
| cpp/src/arrow/ipc/options.h | Adds configurable IPC metadata, body, and decompression limits. |
| cpp/src/arrow/ipc/metadata_internal.cc | Validates required FlatBuffers tensor and custom metadata fields. |
| cpp/src/arrow/ipc/message.h | Adds MessageDecoder constructors accepting IpcReadOptions. |
| cpp/src/arrow/ipc/message.cc | Validates input sizes and message lengths, uses spans for chunk copying, and enforces decoder limits. |
Review effort: Lite
…r and enforce reader limits * Avoid use-after-free in MessageDecoderImpl when listener OnEOS resets the decoder * Early exit when data span is fully consumed before trailing member dereferences * Wire IpcReadOptions through ReadMessage, MessageReader, and RecordBatchFileReader * Add block bounds validation and test coverage for IPC reader limits
There was a problem hiding this comment.
🟡 Changes recommended
Approval is blocked by two active moderate issues: invalid negative length limits can disable IPC protections, and file-footer metadata is not covered by the configured metadata bound.
3 open issues
1 resolved since last review
Review effort: Lite
…force file footer bound * Add IpcReadOptions::Validate() to reject negative values below -1 * Wire options validation to stream and file reader entry points * Enforce max_metadata_length against footer_length in ReadFooterAsync * Add tests for footer bounds, negative options validation, and decoder rejection
… MessageReader streams * Validate metadata and body lengths in StreamDecoderInternal::OnMessageDecoded * Protect custom MessageReader pathways against unbounded allocations * Add unit test coverage with MockCustomMessageReader
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed because the change is unusually complex and a critical hidden issue indicates untrusted input can trigger oversized prebuffer reads before configured limits are enforced.
2 open issues
Review effort: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed because moderate and critical findings indicate the hardening may still permit allocation denial-of-service and allow the advertised body-length limit to be bypassed on a public reader path.
No open issues 🎉
2 resolved since last review
Review effort: Lite


Rationale for this change
In
arrow::ipc::MessageDecoderImpl, chunk reassembly relied on unchecked pointer arithmetic (memcpy(out + offset, ...)) during streaming message reconstruction. Furthermore,MessageDecoder::Consume(data, size)did not validate thatsize >= 0, which could construct invalid negative-sizedBufferinstances and corrupt internal chunk accounting. In addition, FlatBuffers tensor metadata decoders called.size()directly ontensor->shape()without verifying that the optional vector was non-null, and IPC readers lacked opt-in limits to constrain untrusted metadata sizes, body lengths, or decompression expansion ratios.What changes are included in this PR?
MessageDecoderImpl::ConsumeDataChunkswithstd::spanand checked addition via::arrow::internal::AddWithOverflow.DCHECK_GEinvariants on chunk sizing inConsumeDataChunks.MessageDecoder::Consumerejectingsize < 0at the public API entry point.CHECK_FLATBUFFERS_NOT_NULLguards fortensor->shape(), dimension elements, and custom metadata pairs inmetadata_internal.cc.max_metadata_length,max_body_length, andmax_decompression_ratiotoIpcReadOptions(defaulting to-1for unlimited / full backwards compatibility).max_decompression_ratioinDecompressBufferusingMultiplyWithOverflowand slices compressed payloads withstd::span.IpcReadOptionsthroughMessageDecoderconstructors andStreamDecoderImpl.Are these changes tested?
Yes. Existing IPC streaming and file tests pass. New deterministic unit tests were added in
cpp/src/arrow/ipc/read_write_test.cc:TestMessageDecoderBounds.NegativeConsumeSizeTestMessageDecoderBounds.NegativeContinuationTokenTestMessageDecoderBounds.NegativeMetadataLengthTestMessageDecoderBounds.MaxMetadataLengthExceededTestStreamDecoderLimits.MaxBodyLengthExceededTestDecompressBufferLimits.NegativeUncompressedSizeTestDecompressBufferLimits.ZeroCompressedPayloadWithNonZeroUncompressedTestDecompressBufferLimits.MaxDecompressionRatioExceededTestMetadataBounds.EmptyOrInvalidTensorMetadataAre there any user-facing changes?
No breaking changes. Users can optionally configure
max_metadata_length,max_body_length, andmax_decompression_ratioonIpcReadOptionswhen reading IPC streams.Was AI used for this PR?
PR code and description written by:
Reviewed before submission by: