Skip to content

[Python] Cache opcode dispatch tables in trivial_inference and atomic types in SchemaTranslation - #41061

Draft
damccorm wants to merge 1 commit into
apache:masterfrom
damccorm:perf/py-trivial-inference-opcodes-cache
Draft

damccorm wants to merge 1 commit into
apache:masterfrom
damccorm:perf/py-trivial-inference-opcodes-cache

Conversation

@damccorm

@damccorm damccorm commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Description

Optimizes bytecode return-type inference and schema type translation during pipeline graph construction:

  1. Cache opcode dispatch tables in trivial_inference.infer_return_type_func (sdks/python/apache_beam/typehints/trivial_inference.py):
    • Previously, infer_return_type_func imported opcodes and intrinsic_one_ops and rebuilt the simple_ops dictionary ({k.upper(): v for k, v in opcodes.__dict__.items()}) on every call.
    • Hoists _SIMPLE_OPS and _INT_ONE_OPS to module-level cached tables initialized once on first use.
  2. Fast-path primitive atomic types in SchemaTranslation.typing_to_runner_api (sdks/python/apache_beam/typehints/schemas.py):
    • Checks PRIMITIVE_TO_ATOMIC_TYPE at the top of typing_to_runner_api before inspecting generic origin/dataclass/NamedTuple metadata (1.26x faster schema field translation).

Testing

  • Verified with pytest sdks/python/apache_beam/typehints/trivial_inference_test.py sdks/python/apache_beam/typehints/schemas_test.py.

@github-actions github-actions Bot added python core size/s S: 10-49 lines changed labels Nov 6, 2026
@damccorm

damccorm commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

/gemini review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR introduces two clean, well-targeted performance optimizations for pipeline construction: caching the opcode dispatch tables (_SIMPLE_OPS and _INT_ONE_OPS) at the module level in trivial_inference.py, and fast-pathing primitive atomic types in SchemaTranslation.typing_to_runner_api. Both changes preserve existing semantics while avoiding redundant dictionary reconstructions and costly type introspection.

🔍 General Feedback

  • In trivial_inference.py, hoisting _SIMPLE_OPS and _INT_ONE_OPS out of infer_return_type_func eliminates repeated module imports and opcodes.__dict__ iterations across recursive inference calls.
  • In schemas.py, guarding the early PRIMITIVE_TO_ATOMIC_TYPE check with try ... except TypeError safely handles unhashable type representations (such as schema_pb2.Schema instances) while returning a fresh schema_pb2.FieldType proto for each call.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant