Skip to content

fix: Allow dill 0.4 so UDF serialization works on Python 3.14 - #6858

Open
YHC66 wants to merge 2 commits into
feast-dev:masterfrom
YHC66:fix-dill-python-3-14
Open

YHC66 wants to merge 2 commits into
feast-dev:masterfrom
YHC66:fix-dill-python-3-14

Conversation

@YHC66

@YHC66 YHC66 commented Sep 22, 2026

Copy link
Copy Markdown

What this PR does / why we need it:

dill~=0.3.0 keeps Python 3.14 users on dill 0.3.9, which doesn't support 3.14's changes to pickle. Serializing a UDF with dill.dumps(udf, recurse=True), as Transformation.to_proto() and StreamFeatureView.to_proto() do, fails as soon as the UDF refers to a module-level global, such as pd in a pandas UDF:

TypeError: _Pickler._batch_setitems() missing 1 required positional argument: 'obj'

So an on-demand feature view written like the docs example can't be applied on 3.14:

@on_demand_feature_view(
    sources=[driver_stats, request],
    schema=[Field(name="conv_rate_plus_val", dtype=Float64)],
    mode="pandas",
)
def transformed_conv_rate(inputs: pd.DataFrame) -> pd.DataFrame:
    df = pd.DataFrame()
    df["conv_rate_plus_val"] = inputs["conv_rate"] + inputs["val_to_add"]
    return df

transformed_conv_rate.to_proto()  # TypeError on Python 3.14 with dill 0.3.9

dill added Python 3.14 support in 0.4.0 and made it formal in 0.4.1. This widens the requirement to dill>=0.3.0,<0.5 and records it in pixi.lock (regenerated with pixi lock; the locked dill stays at 0.3.9). The sdk/python/requirements lock files are unchanged and still resolve dill 0.3.9 for 3.10–3.12, so CI keeps testing the same versions; installs on 3.14 now get a dill that works.

A UDF serialized with dill 0.3.9 loads with 0.4.1 and the other way round (checked on 3.12), so existing registries and mixed client/server versions keep working.

Which issue(s) this PR fixes:

No existing issue.

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests
  • Testing is not required for this change

Misc

CI runs Python 3.10–3.12, which is why this doesn't show up there, and I didn't add a test for the same reason: the existing on-demand feature view tests cover it on 3.14. Locally, on macOS:

  • Python 3.14.7 with -e ".[test,k8s,aws,gcp]": the ODFV, transformation and stream feature view unit tests (103 tests in 11 files) have 15 failures with the TypeError above on dill 0.3.9, and none on dill 0.4.1. Five further tests need pyspark, which I didn't install, and fail either way.
  • Python 3.12 with sdk/python/requirements/py3.12-ci-requirements.txt: the full sdk/python/tests/unit run gives 2773 passed, 44 skipped with dill 0.3.9, and the same with dill 0.4.1 swapped in.

@ntkathole

Copy link
Copy Markdown
Member

@YHC66 Please run make lock-python-dependencies-all to update requirements file

dill~=0.3.0 keeps Python 3.14 users on dill 0.3.9, which does not
support 3.14's changes to pickle. Serializing a UDF with
dill.dumps(udf, recurse=True), as Transformation.to_proto() and
StreamFeatureView.to_proto() do, fails with "TypeError:
_Pickler._batch_setitems() missing 1 required positional argument:
'obj'" whenever the UDF refers to a module-level global such as pd, so
on-demand feature views cannot be applied on 3.14.

dill added Python 3.14 support in 0.4.0 and made it formal in 0.4.1.
Allow it. The lock files still resolve dill 0.3.9 for 3.10-3.12, and
pixi.lock only records the new requirement (from `pixi lock`).

Signed-off-by: Yihang Chen <yhc0720@berkeley.edu>
Signed-off-by: Yihang Chen <yhc0720@berkeley.edu>
@YHC66
YHC66 force-pushed the fix-dill-python-3-14 branch from 44b56ef to ab57414 Compare September 25, 2026 15:47
@YHC66
YHC66 requested a review from a team as a code owner September 25, 2026 15:47
@YHC66

YHC66 commented Sep 25, 2026

Copy link
Copy Markdown
Author

Done: rebased on current master and regenerated all the lock files with the lock-python-dependencies-all recipe (ab57414); every requirements file now pins dill==0.4.1. One thing I hit while running it: pybuild-deps==0.5.0 fails at the output step with the newly released pip-tools 7.6.1 (OutputWriter.__init__() got an unexpected keyword argument 'generate_hashes'), so I ran the build-deps step with pip-tools 7.5.3. You may want to pin that in the Makefile at some point.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants