Fix DOCX text_as_html duplicating merged-cell text instead of colspan…

…/rowspan (#4469)

## Summary

DOCX table extraction produced two different representations of a merged
cell: `.text` included its content once, but `.metadata.text_as_html`
repeated the content into every `<td>` the merge visually covered, with
no `colspan`/`rowspan` attribute marking that a merge had happened at
all.

Root cause: `_convert_table_to_html` built its HTML from `python-docx`'s
`row.cells`, which resolves both horizontal (`gridSpan`) and vertical
(`vMerge="continue"`) merges by yielding the *same* underlying cell
content at every grid position the merge spans. That already-expanded
grid was serialized straight into HTML with one `<td>` per matrix
position and no span-collapsing step.

Fix: track the underlying `tc` XML element's identity (not cell text)
when building the matrix — the same `tc` object appears at every grid
position a merge covers — then collapse runs of identical identity into
`(colspan, rowspan)` before emitting `<td>`s. DOCX only permits
rectangular merges, so there's no irregular-region case to handle.

A new shared helper (`collapse_matrix_of_keyed_cells_to_spans`) does the
collapsing; the existing `htmlify_matrix_of_cell_texts` (still used
unchanged by the pptx and HTML-parser partitioners) now shares its
cell-escaping logic with the new span-aware path via an extracted
`_format_td` helper.

## Test plan

- [x] `partition_docx` on `example-docs/docx-tables.docx` (the
merged-cell fixture) now produces `text_as_html` with correct
`colspan`/`rowspan` and no duplicated cell text — added a behavioral
regression test through the public `partition_docx` API.
- [x] Existing `test_docx.py` fixtures pinning the old
duplicated-text/no-span output updated to the new expected output.
- [x] Checked downstream consumers that assumed DOCX tables never carry
spans:
- `unstructured/metrics/table/table_extraction.py`'s span-aware grid
reconstruction already handles `colspan`/`rowspan` correctly (written
for other span-producing sources) — traced by hand against the
merged-cell fixture.
- Chunking's table splitter handles real spanned DOCX tables without
crashing at multiple `max_characters` values. Found one pre-existing
(not introduced here) limitation: splitting between rows that share a
`rowspan` can drop that cell's data in the later chunk — this was always
latent for any spanned HTML source, just never exercised for DOCX before
since DOCX never emitted spans until now. Not fixed in this PR; flagging
as a possible follow-up.
- [x] Full relevant test suites green (partition/docx,
common/html_table, chunking, metrics/table), lint clean.

<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4469?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->

chore: make a release (#4461)

<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4461?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->

fix: support core metadata 2.5 publishing (#4452)

## Summary

- refresh Python package publishing tooling
- validate distribution artifacts before upload
- support retrying publication from an existing published release tag
- advance the package version and changelog

## Validation

- workflow linting
- version and lockfile checks
- package build and metadata validation

(authored by codex)

feat: add partition runtime telemetry (#4442)

## Summary

Adds one privacy-bounded, best-effort runtime telemetry attempt for each
outermost public partition invocation. A shared outer decorator covers
all 26 public entrypoints and uses context-local invocation state to
suppress automatic dispatch, format conversion, body/attachment, and
other nested calls.

Successful events report only fixed-enum processing characteristics and
aggregate counts over the final returned elements. Error events contain
only the required environment/package fields and a safely resolved
normalized document type when available. Processing results, ordering,
signatures, and original exceptions are preserved.

## Privacy and controls

Startup and runtime telemetry share `GET
https://packages.unstructured.io/python-telemetry`; the endpoint
supports both the existing per-application-startup query schema and the
new per-document runtime query schema. Runtime telemetry is default-on,
matching startup telemetry, and uses URL query parameters with no
request body. Either `DO_NOT_TRACK` or `SCARF_NO_ANALYTICS`, when
nonempty after trimming, disables collection before runtime argument
inspection or scheduling.

The event excludes content, filenames, paths, input URLs, raw MIME
values, exception details, credentials, proxy configuration, and
persistent identifiers. Delivery uses a fresh session that disables
environment/netrc trust, redirects, retries, and response-body
downloads.

## Reliability

Each eligible outer invocation makes one local delivery attempt. A
process-wide nonblocking single slot allows at most one daemon worker
and no queue. Partition processing never waits for the network; if the
worker is occupied or thread creation fails, the event is dropped. A
stalled DNS/proxy/socket operation can strand at most one daemon thread,
later events drop, and interpreter/process exit does not wait for it.
The `(0.5, 0.5)` connect/read timeout is an inactivity bound, not a
total network wall-clock deadline.

## Validation

- `uv run pytest -q test_unstructured/test_runtime_telemetry.py
test_unstructured/test_telemetry.py
test_unstructured/partition/test_text.py
test_unstructured/partition/test_api.py` — 133 passed, 6 skipped
- `uv run ruff check unstructured/telemetry.py
unstructured/partition/pdf.py
test_unstructured/test_runtime_telemetry.py`
- `uv run ruff format --check unstructured/telemetry.py
unstructured/partition/pdf.py
test_unstructured/test_runtime_telemetry.py`
- `uv run mypy unstructured/telemetry.py`
- Exact-head GitHub checks — 53 passed
- GPT-5.6 Sol Pro exact-head production review — `SAFE TO MERGE`

(authored by codex)

feat: add lazy chunking entry points (#4423)

## Summary

Adds `iter_chunk_elements()` and `iter_chunks_by_title()`, generator
counterparts to the existing `chunk_elements()` and `chunk_by_title()`.
Same options, same chunks, same order — the only difference is that
chunks are yielded as they are formed rather than accumulated into a
list.

## Why

Chunking has always been lazy internally. `PreChunker.iter_pre_chunks()`
and `PreChunk.iter_chunks()` are generators, and the public functions
only wrapped them in a list comprehension. There was no public way to
reach that pipeline, so a caller that wanted to stream chunks had to
either buffer the whole document or import private symbols
(`_ByTitleChunkingOptions`, `_BasicChunkingOptions`) and reassemble the
pipeline by hand.

Combined with a lazy `elements` source, the new entry points keep peak
memory proportional to the largest pre-chunk instead of the whole
document. That matters when elements carry large `metadata.image_base64`
payloads, where a document's element list can be far larger than the
source file.

## Implementation

This exposes the existing pipeline rather than adding a second one:

- `_iter_chunk_elements()` / `_iter_chunks_by_title()` hold the
generator logic.
- `_chunk_elements()` / `_chunk_by_title()` become `list()` over those
generators, so there is a single implementation to keep correct and the
list and generator forms cannot drift.
- Both public entry points keep their existing `(elements, opts)` call
into the private implementation, so the option-plumbing unit tests are
unchanged.

Options are validated **eagerly**, when the function is called, not when
the returned iterator is first advanced. `iter_chunk_elements` and
`iter_chunks_by_title` are therefore plain functions returning an
iterator, not generator functions — a generator function would have
deferred the whole body, so an invalid option combination would surface
at an unrelated point in the caller, or not at all for a document that
yields no chunks.

## Tests

`test_unstructured/chunking/test_lazy.py`:

- **Equivalence** with the list form for both strategies, with
`include_orig_elements` both on and off, comparing serialized chunks
field-by-field (`orig_elements` included; `element_id` is a fresh UUID
per chunk and is excluded).
- **Laziness** — yielding the first chunk does not drain the source
iterator.
- **Eager option validation** — an invalid option raises at the call,
not at first advance.
- **Signature parity** — guards against the generator and list
signatures drifting as options are added.

The existing chunking suite (343 tests) passes unchanged.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4423?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

feat: partition arbitrary valid JSON and NDJSON files (#4391)

## Summary

`partition_json()` and `partition_ndjson()` currently accept **only
serialized Unstructured element output** - any other valid JSON is
rejected with `"Schema does not match the Unstructured schema"` (and an
array of scalars crashes with a raw `AttributeError`). This PR makes
both partitioners **two-mode**:

- **Rehydration (unchanged):** a payload of serialized Unstructured
elements is rehydrated back into its elements, exactly as before.
- **Arbitrary JSON (new):** any other valid JSON/NDJSON is converted to
`Text` elements containing the pretty-printed JSON, instead of raising.

It also fixes a file-type detection bug this feature exposed: a
**compact single-line JSON object** was misrouted to `FileType.NDJSON`
and could never reach `partition_json` at all.

## Behavior change

| Input | Before | After |
|---|---|---|
| Object `{"customer": "Acme", ...}` | `ValueError: JSON cannot be
partitioned. Schema does not match…` | 1 `Text` element, pretty-printed
|
| Array of objects `[{"id":"one"},{"id":"two"}]` | same `ValueError` |
one `Text` per object, array order |
| Array of scalars `[1,2,3]` | crash: `AttributeError: 'int' object has
no attribute 'get'` | 1 `Text` with the whole array |
| Mixed array / top-level scalar | `ValueError` / crash | 1 `Text` |
| Mixed element-shaped + arbitrary array | partial rehydrate, arbitrary
items silently dropped | whole array as arbitrary JSON - nothing dropped
|
| Compact single-line object, `.json` file | misrouted to NDJSON →
rejected | detected as `FileType.JSON` → 1 `Text` |
| Arbitrary NDJSON (one record per line) | `ValueError` | one `Text` per
line, line order |
| Serialized element JSON / NDJSON (arrays / line-per-element) |
rehydrates | rehydrates (unchanged, full regression suite green) |
| Element-shaped payload with corrupt contents (bad
`metadata.coordinates`, non-gzip `orig_elements`) | raw
`ValueError`/`binascii.Error`/`zlib.error` leaks | chained
`ValueError("Payload resembles serialized Unstructured elements but
could not be reconstructed: …")` |
| Serialized `TableChunk` elements (chunked output with split tables) |
silently dropped by `elements_from_dicts()` | rehydrate as `TableChunk`;
whole chunked payloads round-trip and can feed
`reconstruct_table_from_chunks()` (#4291) |
| `{}` (JSON route) | error | one `Text` containing `{}` (per the output
contract) |
| `[]` / empty string (JSON route) | `[]` / `[]` | `[]` |
| `{}` / `[]` as an NDJSON line | error | `Text("{}")` / `Text("[]")` (a
line is a record) |
| Deeply nested payload (any depth) | `RecursionError` escapes |
`ValueError` (`"Not a valid json"` / `"…nested too deeply…"`) |
| Malformed JSON `[{"hi":"there"}]]` | `ValueError("Not a valid json")`
| unchanged |
| Malformed NDJSON line | `ValueError("Not a valid ndjson")` | unchanged
|

## Design notes

- **Mode selection:** an explicit shape predicate
(`is_element_shaped_dict` in `partition/common/json_partitioning.py`) -
a list rehydrates only when every item is a dict with a recognized `str`
`type`, the type's required field (`str` `text` / `bool` `checked` for
CheckBox), and dict-or-absent `metadata`. Branches are exclusive; no
exception-based control flow. Prefix/schema pre-gates are removed from
`partition_json`, `partition_ndjson`, and `auto.py`;
`is_json_processable`/`is_ndjson_processable` are **deprecated**
(DeprecationWarning naming the replacement) but keep working for
downstream importers. `unstructured.file_utils.ndjson.loads/load` are
intentionally retained undeprecated as generic utilities.
- **Documented limitation** (pinned by tests): an array whose items
*all* look like serialized elements rehydrates rather than being treated
as arbitrary JSON. An element-shaped payload whose field contents fail
rehydration raises a chained `ValueError` - loud, never a leaked
low-level error.
- **`staging/base.py` deliberately not modified:** the shape predicate
rejects payloads like `{"type": "Title"}` (no `str` text) before
`elements_from_dicts` is ever called, so they partition as arbitrary
JSON; hardening `item["text"]` → `.get()` in staging instead would
silently rehydrate customer dicts as empty elements.
- **Filetype disambiguation:** whole-payload `json.loads` success →
`FileType.JSON`; else ≥2 newline-delimited JSON values → `NDJSON`. The
probe is **bounded to 1 MiB**, distinguishes an exact-bound-size payload
from a truncated one, restores the file position (a
`detect_filetype(file=f)` → `partition_json(file=f)` sequence works on
the same handle), and treats `RecursionError` as a parse failure. For
payloads exceeding the bound, one or more complete parsing lines
classify as NDJSON (first-line semantics for oversized records); the
residual degradation is an NDJSON file whose *first* record has no
newline inside the bound, which classifies as JSON. One intentional
flip: a *one-line serialized-element object* previously rehydrated via
the NDJSON route; it now partitions as arbitrary JSON (rehydration
applies only to arrays).
- **Output contract:** `pretty_json_text()` - `json.dumps(value,
indent=2, sort_keys=True)`; stable, diffable output; `sort_keys`
alphabetizes source field order (commented at the definition as the knob
to revisit). NDJSON is strictly one `Text` per line; the only
empty-container divergence is `[]` (an NDJSON line yields `Text("[]")`,
a JSON-mode `[]` document yields no elements).
- **Deferred by design:** per-field metadata / JSONPath addressing and
structure-aware tree walking. `elements_from_arbitrary_value()`
(`partition/common/json_partitioning.py`) is the single swap-point for a
future walker.

## TableChunk rehydration - intent

Review of this branch surfaced that serialized `TableChunk` dicts could
not be deserialized at all: `TYPE_TO_TEXT_ELEMENT_MAP` has no
`TableChunk` entry, so `elements_from_dicts()` silently dropped them on
`main`, and with this branch's shape predicate one TableChunk flipped an
entire serialized payload to arbitrary-JSON `Text`. The intent of the
fix commit is narrow and explicit:

- **Complete #4291's design, using only existing mechanisms.**
`reconstruct_table_from_chunks()` (added in #4291) filters
`isinstance(e, TableChunk)` and its docstring states reconstruction "can
be called on user-provided/deserialized chunks" - but no deserializer
could produce a `TableChunk` until now. The reconstruction metadata
(`table_id`, `chunk_index`, `is_continuation`,
`num_carried_over_header_rows`) already round-trips via
`ElementMetadata`; only the element-class dispatch was missing.
- **Special-case, not a map entry - deliberately.**
`TYPE_TO_TEXT_ELEMENT_MAP` also feeds the COCO category vocabulary
(`convert_to_coco` derives positional category ids from its keys), so
adding TableChunk there would renumber existing category ids on every
export, even for data with no TableChunks. Instead
`elements_from_dicts()` special-cases `"TableChunk"` exactly like the
existing `CheckBox` special case. Verified strictly additive: a 40-dict
all-type sweep shows byte-identical output vs the pre-fix function for
every non-TableChunk payload; `documents/elements.py` is untouched; COCO
ids unchanged.
- **No new downstream exposure.** `type: "TableChunk"` records already
flow to destinations today whenever live chunking splits a table
(library output, ingest, hosted API responses); this changes only the
read-back side.

## Testing

- Combined suites (partition json/ndjson/shared-predicate, filetype,
staging serde, chunking dispatch/reconstruct): **691 passed, 11 skipped
(pre-existing optional-dep skips), 1 xfailed**; `test_auto.py -k "json
or ndjson"` **7 passed, 1 xfailed** (the #3365 strict-xfail - unaffected
and kept).
- Full rehydration regression set green (round-trips, chunking,
`last_modified`, metadata stamping), including end-to-end NDJSON
rehydration through `partition()`.
- Five pre-existing tests intentionally flip expected behavior (`{}` no
longer raises, empty NDJSON container lines emit `Text`, one-record
payloads route JSON) - each renamed to describe the new behavior.
- Exact pretty-printed literals are confined to one canonical test per
output shape; other tests assert structurally (element type + content
containment) so a future formatting change doesn't invalidate dozens of
assertions.
- TableChunk: end-to-end chunked-table round-trip (chunk -> serialize ->
`partition_json` -> byte-identical re-serialization ->
`reconstruct_table_from_chunks` returns the table), NDJSON TableChunk
lines, predicate accept/reject, and staging serde tests. The
exact-bound-size detection probe is pinned by a test proven to fail if
the probe is reverted.
- Coverage includes `text=`/`file=`/`filename=` routes, corrupt-payload
errors via `file=`, deep-nesting through both partitioners,
boundary-size (1 MiB ± 1) disambiguation, detect-then-partition on one
file handle, and chunking over arbitrary JSON **and** NDJSON output.
- New business-neutral fixtures: `example-docs/arbitrary-records.json`,
`single-line-object.json`, `arbitrary-records.ndjson`.
- Hardened by two independent review passes (adversarial correctness +
convention/nit pass), each finding verified by reproduction before being
fixed.

fix: sanitize v2 HTML output to prevent stored XSS (GHSA-v5mq-3xhg-98m9…

…) (#4394)

## Summary

The v2 (ontology) HTML path emitted untrusted document markup with **no
output encoding**, so attacker-controlled content in a parsed document
survived into `elements_to_html()` / `metadata.text_as_html` and
executed when the output was viewed in a browser (stored XSS,
[GHSA-v5mq-3xhg-98m9](GHSA-v5mq-3xhg-98m9)).
All four reported vectors — `<img onerror>`, `<svg onload>`
(attribute-value breakout), `<a href="javascript:">`, and `on*` handlers
— are now neutralized.

The fix layers output-encoding at the emitter, filtering at ingest, and
a sanitizer sweep at the assembly boundary.

## Changes

- **New `unstructured/documents/html_sanitization.py`** — single source
of truth for the policy: tag allowlist, attribute allowlist (drops all
`on*` handlers), URL-scheme filter (`is_safe_url`) rejecting
`javascript:`/`vbscript:`/non-image `data:` while preserving
`http`/`https`/`mailto`/`tel`/relative and `data:image/*`, plus an
`nh3`-backed `sanitize_html_fragment`.
- **`ontology.py` (`OntologyElement.to_html`)** — root-cause fix:
HTML-escape element text and attribute values (`quote=True` closes the
attribute-value breakout), drop unsafe attributes, and validate the tag
name against the allowlist (non-allowlisted tags like `<script>` fall
back to inert `<span>`). This makes `text_as_html` safe on its own.
`to_text` now strips markup from the raw text rather than the
newly-escaped HTML, preserving text extraction.
- **`transformations.py`** — attribute handling at ingest now *filters*
(drops `on*`/unsafe schemes) instead of escaping, so escaping happens
exactly once at emit (no double-encoding). Backwards-compatible alias
retained.
- **`convert.py` (`elements_to_html`)** — runs assembled output through
`nh3` as defense-in-depth, covering attributes injected outside the
emitter (e.g. `href` from `metadata.url`). Also fixes a node-skipping
bug when reinserting sanitized content.
- Adds `nh3` dependency; version bump to `0.24.1` with CHANGELOG entry.

## Tests

- New `test_html_sanitization.py` (unit) and `test_xss_sanitization.py`
(end-to-end PoC from the advisory), asserting all four vectors render
inert in both `elements_to_html` output and `text_as_html`, plus
preservation of tables, headings, safe links, and base64 images.
- Updated two existing ontology tests whose expectations encoded the
pre-fix behavior — notably `test_malformed_html`, which previously
asserted a **live `<script>` tag** in the output.

## Acceptance criteria

- [x] PoC renders inertly — none of the four vectors execute
- [x] `on*` attributes stripped/neutralized
- [x] `javascript:`/`data:`/`vbscript:` schemes dropped;
`http`/`https`/`mailto`/relative + `data:image/*` preserved
- [x] Element text and attribute values HTML-escaped
- [x] Attribute-value breakout impossible
- [x] Regression tests for all four vectors + legitimate-formatting
preservation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4394?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

refactor: centralize URL fetching with host validation and default ti…

…meouts (#4388)

## Summary

Introduces `unstructured/safe_http.py` as a single, shared entry point
for
outbound URL fetches, and routes the `url=` code paths in `partition`,
`partition_html`, and `partition_md` through it. Previously each of
these
called `requests.get` directly with inconsistent (and in some cases
absent)
timeout and validation behavior; this centralizes that logic in one
place so
it stays consistent and testable.

## What changed

- **New `unstructured/safe_http.py`** — a `safe_get()` helper that all
three
  partitioners now use instead of calling `requests.get` directly.
- The helper applies, consistently:
  - an `http`/`https` scheme allowlist
  - a hostname denylist with IDNA normalization
  - address validation performed at TCP connect time, so the address
    validated is the address actually connected to
  - manual redirect following with per-hop re-validation, dropping
    credential material (`Authorization`/`Cookie`/proxy auth, plus
    `auth=`/`cookies=`) on cross-origin hops via requests' own
    `should_strip_auth`
  - default `(connect, read)` timeouts, and refusal of proxied requests
- an opt-out via `allow_private=` / the `UNSTRUCTURED_ALLOW_PRIVATE_URL`
    environment variable for controlled local usage

## Behavior changes (why this is a minor release)

Fetches that resolve to non-routable, loopback, or link-local addresses
are
now rejected by default. Outbound fetches also now carry default
timeouts
where some previously had none. Callers that legitimately need to reach
a
private/internal host can opt out with
`UNSTRUCTURED_ALLOW_PRIVATE_URL=1` (or
`allow_private=True`).

## Tests

Adds `test_unstructured/test_safe_http.py` covering IP/hostname
classification, connect-time validation, redirect re-validation,
cross-origin
credential stripping, and the opt-out.

## Version

Bumps to `0.24.0` (see CHANGELOG) — minor, reflecting the behavior
changes
above.

feat: extract filled AcroForm field text in PDF partitioning (#4372)

## Summary

Values typed into fillable PDF form fields live in **widget
annotations** (`/Annots`), not the page content stream. pdfminer's
layout pass only reads the content stream, so these values were dropped
entirely from partition output.

This recovers filled form-field values and emits them as elements
alongside the content-stream text, for both the `fast` and `hi_res`
strategies.

## Changes

- **`pdfminer_processing.py`**
- New `get_widget_text_from_annots(annots, height)` — resolves
`page.annots`, keeps `/Widget` annotations, walks the `/Parent` chain
for inherited `FT`/`V`, handles text (`/Tx`) and choice (`/Ch`) fields,
and returns `{text, bbox}` per filled field. Empty/`/Off` fields are
skipped; `/Btn` checkboxes/radios are intentionally excluded.
- `_decode_field_value` / `_decode_scalar_field_value` decode PDF string
(UTF-16/PDFDocEncoded via pdfminer's `decode_text`) and name values,
including multi-value choice fields.
- **hi_res:** `process_page_layout_from_pdfminer` accepts a
`widget_list` and appends each widget as an extracted text region
(`is_extracted=True`, `source=PDFMINER`); `process_data_with_pdfminer`
computes it next to the existing `get_uris` call. Values then ride the
normal merge/dedup/coordinate machinery.
- **`pdf.py` (fast):** `_process_pdfminer_pages` emits a `Text` element
per widget after the content-stream loop.

## Tests

`test_pdfminer_processing.py` builds a synthetic AcroForm PDF in-test
(pypdf, empty content stream, generic fields `name`/`date of
birth`/`address` + one empty field) and asserts:
- the helper recovers filled values and skips empty ones,
- the hi_res extracted layer includes them as `IsExtracted.TRUE`
regions,
- `partition_pdf(strategy="fast")` recovers them end-to-end.

All 38 tests in the file pass; lint clean.

## Notes
- Behavior change: every fillable PDF now yields previously-missing
field text.
- Pairs with the `init_forms()` render fix in `unstructured-inference`
(so the flattened image and the extracted layer agree). The shared bbox
means the existing IoU dedup collapses any OCR overlap.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat: add enrichment origins metadata field (#4370)

### Summary
Adds a new `enrichment_origins` field to `ElementMetadata` that tracks
which model wrote (or contributed to) each enriched attribute.

### What's new
- **`ElementMetadata.enrichment_origins`** — a serialized `dict[str,
list[dict[str, str]]]` mapping a written attribute name (e.g. `text`,
`text_as_html`, `embeddings`) to a list of `{"type", "provider",
"model"}` records, in application order. Authoring enrichments overwrite
the list; additive enrichments append, preserving the prior author.
- **`ConsolidationStrategy.DICT_LIST_UNIQUE`** — a new chunking
consolidation strategy for `dict[str, list]` fields. It unions keys
across elements and, per key, concatenates then dedupes records while
preserving first-seen order. Wired up as the strategy for
`enrichment_origins`.

### Testing
- Unit tests for the new metadata field (`test_elements.py`).
- Unit tests for the `DICT_LIST_UNIQUE` merge behavior during chunking
(`test_base.py`).

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds per-attribute enrichment provenance via
`ElementMetadata.enrichment_origins` and a dict-list merge strategy, and
isolates per-chunk metadata so provenance updates can’t leak across
split chunks.

- **New Features**
- `ElementMetadata.enrichment_origins`: maps an attribute to a list of
`{"type","provider","model"}` in application order. Authoring
overwrites; additive appends.
- `ConsolidationStrategy.DICT_LIST_UNIQUE`: unions keys and per-key
concat+dedupes while preserving order; used for `enrichment_origins` in
chunking.

- **Bug Fixes**
- First and continuation split chunks each get a fresh metadata object.
- Deep-copies `enrichment_origins` per chunk and sets `is_continuation`
on later chunks, preventing any in-place updates from leaking to
siblings or lazy continuations.

<sup>Written for commit 7a03b8c.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4370?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>