GH-50944: [C++] Replace RapidJSON with simdjson in JSON chunker - #50945

While working on this migration, @rok's earlier implementation rok#47 of the simdjson-based JSON chunker. It was very helpful. Thanks :)

I don't understand why this is parsing JSON by hand?
It seems that we might be able to use simdjson::ondemand::parser::iterate_many.

I initially tried using simdjson::ondemand::parser::iterate_many, but after a few attempts I ran into boundary-handling issues with the chunker semantics. I then referred to an earlier implementation by @rok as a reference and followed that approach.

That said, I agree that parsing JSON manually here is not ideal. I'll revisit this using iterate_many and address the other review comments as well.

@pitrou I tried using simdjson::ondemand::parser::iterate_many(). Most tests pass, but PropagateErrorsNonLinewiseChunker behaves differently: malformed input is split into a separate document and the error is then reported by the JSON parser, whereas the current implementation reports a chunker error.

Manual structural parsing approach preserved the existing test behavior. @rok, since your earlier implementation was helpful here, do you have any suggestions on how to preserve the current error behavior with iterate_many()?

Most tests pass, but PropagateErrorsNonLinewiseChunker behaves differently: malformed input is split into a separate document and the error is then reported by the JSON parser, whereas the current implementation reports a chunker error.

As long as an error is reported while reading the JSON stream, I don't think we care if it's reported by the chunker or the parser.

@pitrou I gave iterate_many() more that a few attempts, but I am running into semantic differences with the previous RapidJSON implementation, especially around stopping after a complete value when it is followed by a partial or malformed value. At this point, I think manually finding the boundary of the first complete object/array and then validating that slice with simdjson may be simpler and closer to the existing behavior. Do you think that would be a reasonable approach?

@pitrou I gave iterate_many() more that a few attempts, but I am running into semantic differences with the previous RapidJSON implementation, especially around stopping after a complete value when it is followed by a partial or malformed value.

Hmm, I see. Thanks for trying anyway :-)

At this point, I think manually finding the boundary of the first complete object/array and then validating that slice with simdjson may be simpler and closer to the existing behavior. Do you think that would be a reasonable approach?

Well, as a last resort, yes. The problem:

1. We're writing our own JSON parser, which means we must careful test it.
2. We're losing performance unless we implement our own SIMD optimizations.

Thoughts @HuaHuaY @cyb70289 ?

Try to understand the issue. Is it that json strings legal for rapidjson may fail on simdjson, makes future Arrow release potentially incompatible to old version?

Writing our own optimized version looks not ideal. Can we just use simdjson? It's state-of-the-art, and even with self written object delimiter, there's still incompatibility risk I'm afraid.

@cyb70289 I don't think the issue is JSON compatibility between RapidJSON and simdjson. The main issue I ran into is the boundary/streaming semantics.

The previous RapidJSON implementation uses kParseStopWhenDoneFlag, so it stops as soon as the first complete top-level value is parsed, even if the remaining input contains a partial or malformed value.

With iterate_many(), I wasn't able to reproduce that behavior: errors in subsequent/incomplete data can affect whether we successfully identify the preceding complete value.

I agree that writing our own optimized JSON parser would not be ideal. The manual approach I used, and which @rok also implemented in rok#47, only scans for the boundary of the first complete object or array while respecting strings and escapes, and then lets simdjson perform the actual JSON validation. But I agree this still introduces complexity and needs careful testing.

If there is a way to use simdjson directly while preserving the old stop-after-one-value semantics, that would definitely be preferable.

With iterate_many(), I wasn't able to reproduce that behavior: errors in subsequent/incomplete data can affect whether we successfully identify the preceding complete value.

Is that a problem? We want to keep compatibility when parsing valid JSON streams. The failure mode for an invalid JSON stream can change.

A discussion about ignoring trailing garbage in simdjson. Looks there're real use cases lenient parsing can be useful.
simdjson/simdjson#2502

Trailing garbage is not the problem here. We are parsing a stream of valid JSON documents. We are happy to error out on trailing garbage.

@taepper I've addressed your review comments, do you want to take another look? Otherwise I think I'll merge.

Thanks a lot @pitrou for all the help on this PR and the additional changes. I learned a lot from your reviews. Really appreciate it!

Minio-related CI failures need #51326 for resolving.

@Reranko05 You're welcome! Your contributions are always welcome :)