Skip to content

Commit c89c7ad

Browse files
gaogaotiantianHyukjinKwon
authored andcommitted
[SPARK-56463][PYTHON] Disallow unpickling UDT
### What changes were proposed in this pull request? Disallow unpickling UDT. ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? Yes, unpickling UDT is not supported. ### How was this patch tested? CI. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #55330 from gaogaotiantian/disallow-unpickling. Authored-by: Tian Gao <gaogaotiantian@hotmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
1 parent 26ae8c1 commit c89c7ad

2 files changed

Lines changed: 5 additions & 3 deletions

File tree

python/pyspark/sql/types.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1725,8 +1725,10 @@ def fromJson(cls, json: Dict[str, Any]) -> "UserDefinedType":
17251725
pyClass = pyUDT[split + 1 :]
17261726
m = __import__(pyModule, globals(), locals(), [pyClass])
17271727
if not hasattr(m, pyClass):
1728-
s = base64.b64decode(json["serializedClass"].encode("utf-8"))
1729-
UDT = CloudPickleSerializer().loads(s)
1728+
raise PySparkValueError(
1729+
errorClass="UNSUPPORTED_OPERATION",
1730+
messageParameters={"operation": "unpickling user defined types"},
1731+
)
17301732
else:
17311733
UDT = getattr(m, pyClass)
17321734
return UDT()

python/pyspark/testing/objects.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ def sqlType(cls):
9090

9191
@classmethod
9292
def module(cls):
93-
return "__main__"
93+
return "pyspark.testing.objects"
9494

9595
def serialize(self, obj):
9696
return [obj.x, obj.y]

0 commit comments

Comments
 (0)