Repository navigation
[ZeroOps bug bash] Use fixed-point decimals in the schema-drift example - #174
Closed
lennartkats-db wants to merge 2 commits into
Closed
lennartkats-db wants to merge 2 commits into
lennartkats-db wants to merge 2 commits into
Conversation
Root cause: job 760861345567039, run 25772954634192 (task validate_transaction_schema) failed deterministically with TypeError: Schema contract violation in simulated source schema — the simulated source schema in knowledge_base/failing_job_examples/src/schema_drift.py declared the amount field as StringType, so the isinstance(amount_type, NumericType) guard raised instead of passing.
Change: replace StringType with DecimalType(12, 2) for the amount field (importing DecimalType instead), per the user's revised fix. DecimalType is a subclass of NumericType, so the existing NumericType guard is kept intact and still passes.
Validation: file content in the incident checkout verified after edit — import line reads "from pyspark.sql.types import DecimalType, NumericType, StringType, StructField, StructType" and the field is StructField("amount", DecimalType(12, 2), False). No runtime execution or deployment was performed.
Follow-up refactor on top of the DecimalType(12, 2) fix (job 760861345567039, run 25772954634192): extract the inline NumericType contract check into a small validate_amount_schema(schema) helper with a short docstring, keep the error text byte-for-byte identical for invalid schemas, and add a one-line comment explaining that the simulated monetary amount uses a fixed-point decimal to avoid binary floating-point rounding drift. The helper call validate_amount_schema(observed_source_schema) preserves the original run-time behavior: DecimalType is a subclass of NumericType, so the guard still passes for DecimalType(12, 2). Validation: tool-returned Git diff reviewed in the incident checkout before committing — DecimalType(12, 2) untouched, identical error string literals, guard intact. No runtime execution, deployment, or merge was performed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ZeroOps bug-bash experiment. Do not merge or deploy this change.
The failing-job example intentionally declares a string amount in its simulated source schema. ZeroOps initially suggested DoubleType. In conversation, we requested DecimalType(12, 2) for monetary values while retaining the NumericType guard; ZeroOps pushed commit 0dea675. We then requested a small schema-validation helper and brief decimal rationale; ZeroOps pushed 9d32e88 to this same branch, updating this PR.
Verified: the PR diff contains only the Python notebook source. A local control-flow check against the remote helper passed for numeric acceptance and string rejection; Apache Spark types define DecimalType as a NumericType subclass. PySpark was not installed locally, and no Databricks rerun of the proposed fix was performed. The deployed demonstration job intentionally remains failing.
ZeroOps generated the fix and pushed the scoped branch with runGit after a dedicated ZeroOps push tool was refused by a tool-safety check. Codex opened this draft PR manually because the debug browser was signed out of GitHub. Base is the published failing-examples branch, keeping the diff scoped to this case.