perf: incremental parent-map maintenance - #13
Closed
itamarga wants to merge 5 commits into
Closed
Conversation
…KT-16478] Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
prepare-pr-next polish: condense the repeated coarse-budget clauses in the deobfuscate/deobfuscate_file docstrings. No behavior change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
esprima2 5.0.2 changed optional-chaining (a?.b) parsing to the ESTree ChainExpression wrapper node. The code generator has no ChainExpression handler (emits "/* unknown: ChainExpression */") and the deobfuscation transforms don't see through the wrapper, so on a fresh install (which floated the unpinned dependency up to 5.0.2/6.0.0) the suite fails on optional-chaining round-trips, invalid ?? / || output, and leftover _0x identifiers. The last green build (March) resolved to 5.0.1. Cap the dependency at the tested range to restore a green build; full esprima2 >=5.0.2 support (generator + transforms + snapshot) is a separate follow-up. Ref: TKT-16478 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tighten the docstrings added for the incremental parent-map maintenance and the optional deobfuscate() time budget, removing redundancy while keeping the behavioural caveats. Replace the wall-clock parent-map regression test with deterministic build_parent_map call-count assertions (immune to machine variance) and drop a budget test that duplicated existing hex-escape coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
Superseded by #14 — reopened from a renamed branch ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Fixes a quadratic parent-map rebuild that made
pyjsclear.deobfuscate()run for 15+ minutes on large obfuscator.io-style samples. On two representative samples, 0.1.5 takes 1011.9s / 1134.9s; this branch takes 10.4s / 27.5s under the same profiler (3.1s / 8.3s without).Changes
base.py: newrecord_replacement(replacement, parent, key, index)— an O(1) patch of the cached parent map instead of a full invalidation.ObjectSimplifier,ClassStaticResolver,StringRevealer: node replacement now callsrecord_replacement(...)instead ofinvalidate_parent_map(), eliminating the full O(N) parent-map rebuild on every subsequentfind_parent.esprima2>=…,<5.0.2: 5.0.2 switched optional-chaining (a?.b) parsing to the ESTreeChainExpressionwrapper, which the generator and transforms don't yet support (full support tracked separately).0.1.6.Full description
Root cause. Every node replacement in
ObjectSimplifier,ClassStaticResolver, andStringRevealercalledinvalidate_parent_map(), so the nextfind_parentrebuilt the entire parent map with a full-AST walk. With thousands of proxy-object references that is O(n²). Profiling 0.1.5 on the 606KB sample showsbuild_parent_mapcalled 12,023 times, consuming 980.8s of the 1011.9s total (97%) — 1.76 billiondict.getcalls — all triggered by node replacement invalidating the parent map on every swap.Evidence (same machine, both versions under cProfile):
Fix.
record_replacement(replacement, parent, key, index)patches the cached parent map in O(1). This is safe because the replacement swaps a node in place: in-place swaps don't shift list indices, and replaced subtrees are detached and never looked up again, so only the replacement's own entry changes. The incremental map is behaviourally identical to a full rebuild.esprima2 pin. esprima2 5.0.2 changed optional-chaining parsing from a bare
MemberExpression(withoptional: true) to the ESTree-specChainExpressionwrapper. The code generator has noChainExpressionhandler and the transforms don't see through the wrapper, so a fresh install that floats the dependency up to 5.0.2+ breaks optional-chaining round-trips and leaves obfuscated identifiers unresolved. Capping below 5.0.2 restores a green build; fullChainExpressionsupport (generator + transforms) is tracked as a separate follow-up.Tests. Full suite green: 1754 passed, 3 skipped (unit + regression, fuzz excluded). The parent-map test asserts
build_parent_mapis called exactly once across many in-place replacements — a deterministic check of the fix rather than a timing threshold.🤖 Generated with Claude Code