Repository navigation
Closed
Labels
Milestone
Description
What happened?
When Beam derives a schema (RowTypeConstraint / RowCoder) for user types (such as dataclasses, NamedTuples, or dynamic keys generated by transforms like beam.GroupBy), fields of type tuple (or Tuple[...], tuple[T, ...]) exhibit inconsistent and breaking behavior depending on whether type hints are inferred:
- Untyped /
Any: Falls back toFastPrimitivesCoderand preservestupleidentity. - Typed as
tuple: Converted toArrayTypein Schema and deserialized byIterableCoderas alist. - Impact: Breaks downstream code expecting hashable/immutable objects (e.g.,
TypeError: cannot use 'list' as a dict key). - Related bug (heterogeneous tuples):
schemas.pyassumes allSequencetypes are homogeneous by taking onlyarg_types[0]. Fortuple[str, int], it treats the entire tuple asstrand crashes during encoding withAttributeError: 'int' object has no attribute 'encode'.
Minimal Reproducer
import apache_beam as beam
from apache_beam.testing.test_pipeline import TestPipeline
# Case 1: tuple deserializes as unhashable list
with TestPipeline() as p:
_ = (
p
| beam.Create([('a', 'b')])
| beam.Map(lambda x: beam.Row(spec=x))
| beam.Reshuffle() # forces RowCoder serialization
| beam.Map(lambda row: {row.spec: 1}) # TypeError: unhashable type: 'list'
| beam.Map(print)
)
# Case 1b: pipeline succeeded with tuple preserved if typehint get lost
def no_hint(x):
return x if isinstance(x, tuple) else str(x)
with TestPipeline() as p:
_ = (
p
| beam.Create([('a', 'b')])
| beam.Map(no_hint)
| beam.Map(lambda x: beam.Row(spec=x))
| beam.Reshuffle()
| beam.Map(lambda row: {row.spec: 1})
| beam.Map(print)
)
# Case 2: Heterogeneous tuple crashes on encode
with TestPipeline() as p:
_ = (
p
| beam.Create([('count', 42)])
| beam.Map(lambda x: beam.Row(spec=x))
| beam.Reshuffle() # AttributeError: 'int' object has no attribute 'encode'
| beam.Map(print)
)
# Case 2b: success if typehint get lost
with TestPipeline() as p:
_ = (
p
| beam.Create([('count', 42)])
| beam.Map(no_hint)
| beam.Map(lambda x: beam.Row(spec=x))
| beam.Reshuffle()
| beam.Map(print)
)Root Cause
apache_beam/typehints/schemas.py:377: Maps allSequencetypes (includingtuple) toArrayType, taking onlyarg_types[0].apache_beam/coders/row_coder.py:165&coder_impl.py:1466: UsesIterableCoder, which constructs a Pythonlistupon decodingArrayType.
Suggested Fix
- Reconstruct
tuple(or useTupleSequenceCoderImpl) inRowCoderwhen the original type constraint is aTupleConstraint. - Do not treat fixed-length / heterogeneous
Tuple[T1, T2]as homogeneousArrayType(element_type=T1).
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Infrastructure
- Component: Spark Runner
- Component: Flink Runner
- Component: Prism Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner
Activity
Abacn commentedon Sep 9, 2026
I've been thinking about this issue, but didn't find a good solution within current Beam portable schema framework.
A quick solution is to exclude tuple from resolving to ArrayType, but fallback to Any. This would solve both scenario but currently working scenarios e.g.
tuple[str, ...]ortuple[int, int]now all fallbacks to FastPrimitiveTypeA second solution would be to introduce logical types to represent a Python tuple
(internal bug context: b/518498022)