feat(page size reduction): rewrite the page size inside a RequestPath page token URL - #1169

Stacked on #1149 — review that one first; this branch targets tolik0/cdk/reduce-page-size and its diff is the last commit only.

What

Lets page_size_reduction be used on a paginator whose page_token_option is a RequestPath, by rewriting the page size inside the URL the API returned for the next page instead of sending the reduced page size next to the one that URL already carries. Opted into with a new rewrite_page_size_in_page_token_url flag on PageSizeReduction.

Why

#1149 rejects a RequestPath page token at config time. The reason is real, and reproducible on main today:

HttpClient(name="t", logger=logger)._create_prepared_request(
    http_method="GET",
    url="https://airbyte.io/events?per_page=100&start_time=1234",
    params={"per_page": 50},
    dedupe_query_params=True,
)
# https://airbyte.io/events?per_page=100&start_time=1234&per_page=50

HttpClient._dedupe_query_params drops an injected parameter only when the URL already carries the same value — which is exactly the case where there is nothing to reduce. So with a RequestPath token the request goes out asking for both page sizes and the API picks; every page after the first keeps asking for the size that just failed.

27 connectors in the monorepo pair a RequestPath page token with a page_size_option. The near-term adopter is source-zendesk-support ticket_comments (manifest.yaml:984-995): CursorPagination on next_page, page_token_option: RequestPath, page_size_option: per_page, and a 504 → RETRY filter that retries at an unchanged per_page so the retries cannot succeed.

One correction to the #1149 description, which said 3 of those 27 (mendeley, sendgrid, zendesk-support) already handle 502/504: grepping the current manifests, only zendesk-support has any 5xx response filter — mendeley and sendgrid have none. Structurally this unlocks 27 connectors; today exactly one of them has an error path to adopt it for, and that path is a gateway timeout rather than an explicit "page too large" signal. Worth weighing before merging.

How

• DefaultPaginator.path takes page_size_override and rewrites the parameter named by page_size_option inside the token URL. Only that pair is edited; the rest of the query is kept byte for byte rather than parsed and re-encoded, so a URL the API built is handed back as it was written (percent-encoding, blank values and bare flags survive).
• A URL that does not carry the parameter is left alone — get_request_params then adds the reduced page size the usual way, and appending it to the path would send it twice.
• The reduced page size reaches path through SimpleRetriever._paginator_path, passed with page_size_override_kwargs so a paginator defined outside the CDK that does not accept the argument keeps working. PaginatorTestReadDecorator forwards it the same way.
• After the rewrite the URL and the injected parameter agree, so the existing _dedupe_query_params drops the injected one and a single page size goes out. There is a test asserting exactly that, next to one asserting the duplicated URL without the rewrite.

Why it is opt-in

Whether a page token URL can be re-requested at a smaller page size is a property of the API, and a RequestPath token is opaque to the CDK. It holds for a URL addressing records by cursor or timestamp; it does not hold for one carrying a page number, where a smaller page size moves every following page boundary and skips records — the same hazard that makes #1149 reject PageIncrement. So the connector asserts it rather than the CDK assuming it.

Rejected at config time:

• rewrite_page_size_in_page_token_url without a RequestPath page token — there is no URL to rewrite in.
• rewrite_page_size_in_page_token_url with a page_size_option that does not inject into request_parameter — only a query parameter of that URL can be rewritten.
• A RequestPath page token without the flag — the existing feat: add REDUCE_PAGE_SIZE response action for dynamic page-size reduction #1149 error, now naming the flag and what has to be true about the API to set it.

Scope

Nothing changes for a stream without page_size_reduction: page_size_override is None for every other stream, and path returns the token unchanged as before. The URL rewrite is reachable only from a reduced page.

Testing

• DefaultPaginator.path: rewrite, absent parameter, no query, no override, non-request_parameter injection, repeated parameter, a parameter whose name is a prefix of the page size field, an interpolated field_name, a fragment, other parameters returned byte for byte.
• End-to-end through HttpClient._create_prepared_request: one page size out after the rewrite, two without it.
• SimpleRetriever: the override reaches path, is omitted when there is none, and _fetch_next_page sends the rewritten URL.
• Factory: the three rejections above and a manifest that parses with the flag set.
• Full unit_tests/sources/declarative and unit_tests/sources/streams/http green (2468 passed); the two memory-usage tests were skipped locally, they SIGILL on this machine independently of this branch.

No connector adopts this yet; source-zendesk-support would need its 504 filter switched to REDUCE_PAGE_SIZE and an SDM prerelease image, since it is manifest-only.