Skip to content

GH-50822 [C++][Parquet] Make DELTA_BYTE_ARRAY value reconstruction storage-agnostic - #50823

Open
marcin-krystianc wants to merge 2 commits into
apache:mainfrom
marcin-krystianc:dev-20260702-flba-decoder-follow-up
Open

GH-50822 [C++][Parquet] Make DELTA_BYTE_ARRAY value reconstruction storage-agnostic#50823
marcin-krystianc wants to merge 2 commits into
apache:mainfrom
marcin-krystianc:dev-20260702-flba-decoder-follow-up

Conversation

@marcin-krystianc

@marcin-krystianc marcin-krystianc commented Aug 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Rationale for this change

Follow-up to #50335.

DeltaByteArrayFLBADecoder::Decode(uint8_t*, int) decodes into a
temporary std::vector<ByteArray> and then copies the bytes into the caller's buffer,
because DeltaByteArrayDecoderImpl::GetInternal only supports ByteArray output. The
vector is a per-call allocation, and the pointer it holds for every value is discarded
immediately after the copy.

What changes are included in this PR?

Adds two output policy structs, ByteArrayOutput and DenseOutput for DeltaByteArrayFLBADecoder:

  • ByteArrayOutput reproduces the previous behavior: it decodes values into a ByteArray* buffer
  • DenseOutput writes each reconstructed fixed-length value directly into a caller-provided uint8_t* buffer

Adds benchmarks to test the DeltaByteArrayFLBADecoder performance.

  • Before the change
Running /workspace/arrow/cpp/build/release/parquet-encoding-benchmark
Run on (20 X 2112 MHz CPU s)
CPU Caches:
  L1 Data 48 KiB (x10)
  L1 Instruction 32 KiB (x10)
  L2 Unified 1280 KiB (x10)
  L3 Unified 25600 KiB (x1)
Load Average: 2.22, 3.84, 3.64
---------------------------------------------------------------------------------------------------------------------------------------
Benchmark                                                                             Time             CPU   Iterations UserCounters...
---------------------------------------------------------------------------------------------------------------------------------------
BM_DeltaDecodingFLBA/type-length:2/batch-size:512/prefixed-percent:10              3217 ns         3215 ns       221856 bytes_per_second=303.706Mi/s compression_ratio=0.760208 items_per_second=159.229M/s
BM_DeltaDecodingFLBA/type-length:2/batch-size:512/prefixed-percent:90              3923 ns         3921 ns       185631 bytes_per_second=249.056Mi/s compression_ratio=1.38753 items_per_second=130.577M/s
BM_DeltaDecodingFLBA/type-length:2/batch-size:512/prefixed-percent:99              3929 ns         3927 ns       182369 bytes_per_second=248.649Mi/s compression_ratio=1.79649 items_per_second=130.363M/s
BM_DeltaDecodingFLBA/type-length:2/batch-size:2048/prefixed-percent:10            11335 ns        11329 ns        62995 bytes_per_second=344.803Mi/s compression_ratio=0.76148 items_per_second=180.776M/s
BM_DeltaDecodingFLBA/type-length:2/batch-size:2048/prefixed-percent:90            15514 ns        15506 ns        43892 bytes_per_second=251.918Mi/s compression_ratio=1.40466 items_per_second=132.078M/s
BM_DeltaDecodingFLBA/type-length:2/batch-size:2048/prefixed-percent:99            16525 ns        16517 ns        42482 bytes_per_second=236.502Mi/s compression_ratio=1.7747 items_per_second=123.995M/s
BM_DeltaDecodingFLBA/type-length:16/batch-size:512/prefixed-percent:10             2796 ns         2794 ns       249283 bytes_per_second=2.73034Gi/s compression_ratio=0.966836 items_per_second=183.23M/s
BM_DeltaDecodingFLBA/type-length:16/batch-size:512/prefixed-percent:90             4512 ns         4510 ns       151783 bytes_per_second=1.69171Gi/s compression_ratio=1.70489 items_per_second=113.529M/s
BM_DeltaDecodingFLBA/type-length:16/batch-size:512/prefixed-percent:99             4729 ns         4727 ns       139167 bytes_per_second=1.61411Gi/s compression_ratio=1.78203 items_per_second=108.321M/s
BM_DeltaDecodingFLBA/type-length:16/batch-size:2048/prefixed-percent:10           11151 ns        11144 ns        62274 bytes_per_second=2.73856Gi/s compression_ratio=0.966836 items_per_second=183.781M/s
BM_DeltaDecodingFLBA/type-length:16/batch-size:2048/prefixed-percent:90           17927 ns        17918 ns        39408 bytes_per_second=1.70318Gi/s compression_ratio=1.6732 items_per_second=114.298M/s
BM_DeltaDecodingFLBA/type-length:16/batch-size:2048/prefixed-percent:99           19689 ns        19678 ns        34933 bytes_per_second=1.55083Gi/s compression_ratio=1.79168 items_per_second=104.075M/s
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:512/prefixed-percent:10         3309 ns         3308 ns       211062 bytes_per_second=295.22Mi/s compression_ratio=0.760208 items_per_second=154.78M/s
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:512/prefixed-percent:90         4273 ns         4271 ns       162512 bytes_per_second=228.642Mi/s compression_ratio=1.38753 items_per_second=119.874M/s
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:512/prefixed-percent:99         4404 ns         4402 ns       157929 bytes_per_second=221.87Mi/s compression_ratio=1.79649 items_per_second=116.324M/s
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:2048/prefixed-percent:10       12811 ns        12804 ns        54299 bytes_per_second=305.071Mi/s compression_ratio=0.76148 items_per_second=159.945M/s
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:2048/prefixed-percent:90       17186 ns        17178 ns        39521 bytes_per_second=227.399Mi/s compression_ratio=1.40466 items_per_second=119.223M/s
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:2048/prefixed-percent:99       18260 ns        18250 ns        38287 bytes_per_second=214.042Mi/s compression_ratio=1.7747 items_per_second=112.219M/s
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:512/prefixed-percent:10        3218 ns         3217 ns       214108 bytes_per_second=2.37183Gi/s compression_ratio=0.966836 items_per_second=159.171M/s
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:512/prefixed-percent:90        4807 ns         4805 ns       126853 bytes_per_second=1.58781Gi/s compression_ratio=1.70489 items_per_second=106.556M/s
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:512/prefixed-percent:99        4941 ns         4939 ns       137130 bytes_per_second=1.54488Gi/s compression_ratio=1.78203 items_per_second=103.675M/s
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:2048/prefixed-percent:10      12255 ns        12249 ns        55533 bytes_per_second=2.4915Gi/s compression_ratio=0.966836 items_per_second=167.202M/s
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:2048/prefixed-percent:90      19460 ns        19338 ns        35959 bytes_per_second=1.57813Gi/s compression_ratio=1.6732 items_per_second=105.906M/s
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:2048/prefixed-percent:99      21114 ns        21103 ns        32878 bytes_per_second=1.4461Gi/s compression_ratio=1.79168 items_per_second=97.0458M/s
  • After the change
Running /workspace/arrow/cpp/build/release/parquet-encoding-benchmark
Run on (20 X 2112 MHz CPU s)
CPU Caches:
  L1 Data 48 KiB (x10)
  L1 Instruction 32 KiB (x10)
  L2 Unified 1280 KiB (x10)
  L3 Unified 25600 KiB (x1)
Load Average: 1.24, 6.66, 4.25
---------------------------------------------------------------------------------------------------------------------------------------
Benchmark                                                                             Time             CPU   Iterations UserCounters...
---------------------------------------------------------------------------------------------------------------------------------------
BM_DeltaDecodingFLBA/type-length:2/batch-size:512/prefixed-percent:10              3419 ns         3418 ns       188811 bytes_per_second=285.752Mi/s compression_ratio=0.760208 items_per_second=149.816M/s
BM_DeltaDecodingFLBA/type-length:2/batch-size:512/prefixed-percent:90              3843 ns         3841 ns       174937 bytes_per_second=254.249Mi/s compression_ratio=1.38753 items_per_second=133.3M/s
BM_DeltaDecodingFLBA/type-length:2/batch-size:512/prefixed-percent:99              3730 ns         3728 ns       186091 bytes_per_second=261.966Mi/s compression_ratio=1.79649 items_per_second=137.345M/s
BM_DeltaDecodingFLBA/type-length:2/batch-size:2048/prefixed-percent:10            13447 ns        13439 ns        51135 bytes_per_second=290.661Mi/s compression_ratio=0.76148 items_per_second=152.39M/s
BM_DeltaDecodingFLBA/type-length:2/batch-size:2048/prefixed-percent:90            17300 ns        17292 ns        39339 bytes_per_second=225.901Mi/s compression_ratio=1.40466 items_per_second=118.437M/s
BM_DeltaDecodingFLBA/type-length:2/batch-size:2048/prefixed-percent:99            16789 ns        16781 ns        40393 bytes_per_second=232.784Mi/s compression_ratio=1.7747 items_per_second=122.046M/s
BM_DeltaDecodingFLBA/type-length:16/batch-size:512/prefixed-percent:10             3138 ns         3137 ns       226716 bytes_per_second=2.43218Gi/s compression_ratio=0.966836 items_per_second=163.221M/s
BM_DeltaDecodingFLBA/type-length:16/batch-size:512/prefixed-percent:90             3569 ns         3568 ns       192341 bytes_per_second=2.13857Gi/s compression_ratio=1.70489 items_per_second=143.517M/s
BM_DeltaDecodingFLBA/type-length:16/batch-size:512/prefixed-percent:99             3437 ns         3435 ns       198130 bytes_per_second=2.22082Gi/s compression_ratio=1.78203 items_per_second=149.036M/s
BM_DeltaDecodingFLBA/type-length:16/batch-size:2048/prefixed-percent:10           12079 ns        12072 ns        56551 bytes_per_second=2.52788Gi/s compression_ratio=0.966836 items_per_second=169.643M/s
BM_DeltaDecodingFLBA/type-length:16/batch-size:2048/prefixed-percent:90           14949 ns        14941 ns        43250 bytes_per_second=2.04249Gi/s compression_ratio=1.6732 items_per_second=137.069M/s
BM_DeltaDecodingFLBA/type-length:16/batch-size:2048/prefixed-percent:99           15358 ns        15350 ns        45877 bytes_per_second=1.98814Gi/s compression_ratio=1.79168 items_per_second=133.422M/s
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:512/prefixed-percent:10         3262 ns         3261 ns       212131 bytes_per_second=299.505Mi/s compression_ratio=0.760208 items_per_second=157.027M/s
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:512/prefixed-percent:90         3518 ns         3516 ns       196251 bytes_per_second=277.715Mi/s compression_ratio=1.38753 items_per_second=145.603M/s
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:512/prefixed-percent:99         3536 ns         3534 ns       199117 bytes_per_second=276.305Mi/s compression_ratio=1.79649 items_per_second=144.863M/s
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:2048/prefixed-percent:10       12662 ns        12656 ns        55254 bytes_per_second=308.658Mi/s compression_ratio=0.76148 items_per_second=161.826M/s
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:2048/prefixed-percent:90       16081 ns        16073 ns        40982 bytes_per_second=243.035Mi/s compression_ratio=1.40466 items_per_second=127.421M/s
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:2048/prefixed-percent:99       16415 ns        16407 ns        42847 bytes_per_second=238.086Mi/s compression_ratio=1.7747 items_per_second=124.825M/s
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:512/prefixed-percent:10        2969 ns         2968 ns       238205 bytes_per_second=2.57067Gi/s compression_ratio=0.966836 items_per_second=172.515M/s
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:512/prefixed-percent:90        3462 ns         3460 ns       203926 bytes_per_second=2.20507Gi/s compression_ratio=1.70489 items_per_second=147.98M/s
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:512/prefixed-percent:99        3392 ns         3390 ns       200885 bytes_per_second=2.25044Gi/s compression_ratio=1.78203 items_per_second=151.025M/s
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:2048/prefixed-percent:10      11560 ns        11554 ns        60264 bytes_per_second=2.64135Gi/s compression_ratio=0.966836 items_per_second=177.258M/s
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:2048/prefixed-percent:90      14602 ns        14512 ns        49410 bytes_per_second=2.10297Gi/s compression_ratio=1.6732 items_per_second=141.128M/s
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:2048/prefixed-percent:99      14782 ns        14775 ns        47312 bytes_per_second=2.06547Gi/s compression_ratio=1.79168 items_per_second=138.612M/s

Are these changes tested?

Yes

Are there any user-facing changes?

No

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown

⚠️ GitHub issue #50822 has been automatically assigned in GitHub to PR creator.

@marcin-krystianc
marcin-krystianc force-pushed the dev-20260702-flba-decoder-follow-up branch from 7d36f89 to 550ba67 Compare August 6, 2026 10:02
@marcin-krystianc
marcin-krystianc force-pushed the dev-20260702-flba-decoder-follow-up branch 3 times, most recently from 8d863c7 to 1433e54 Compare August 18, 2026 10:53
@marcin-krystianc marcin-krystianc mentioned this pull request Aug 18, 2026
Merged
@marcin-krystianc
marcin-krystianc force-pushed the dev-20260702-flba-decoder-follow-up branch from bfac4bf to dd5c36a Compare September 14, 2026 10:40
@marcin-krystianc
marcin-krystianc marked this pull request as ready for review September 14, 2026 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wgtmac wgtmac Awaiting requested review from wgtmac wgtmac is a code owner
@pitrou pitrou Awaiting requested review from pitrou pitrou is a code owner

Assignees

No one assigned

Labels

awaiting review Awaiting review Component: C++ Component: Parquet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@marcin-krystianc