perf(crawler): parse HTML once per page in FessXpathTransformer#3183
Merged
Conversation
The web-crawl transformer parsed each page's DOM twice - once in storeData (field/anchor extraction) and again in the inherited storeChildUrls (child-URL discovery) - reading and re-parsing the full response body a second time. Reuse the Document parsed in storeData for child-URL extraction: stash it in a ThreadLocal (the transformer is a shared singleton across crawler threads), override the two-argument storeChildUrls to delegate to the new base storeChildUrls(ResponseData, ResultData, Document) overload with that Document, and clear the ThreadLocal in a transform() finally block so a pooled crawler thread never leaks or reuses a stale DOM. Because the Document is now shared with child-URL/anchor extraction, the field-rule loop's default branch prunes a deep clone instead of the live node, so links nested under pruned subtrees (e.g. a <nav> inside a pruned //H1) are no longer silently dropped from child-URL discovery. As an intended consequence, child-URL/anchor extraction now runs on the storeData parse (Fess-detected charset via InputSource.setEncoding + UTF-8 BOM strip) instead of a separate neko auto-detected parse, and the indexed anchor field can now also retain links under pruned-tag subtrees nested in headings. Requires fess-crawler providing storeChildUrls(ResponseData, ResultData, Document). Add tests: a link nested under a pruned <nav> inside a pruned heading is still discovered (verified to fail without the clone-before-prune fix), the response body is parsed only once, and normal pages extract identical fields and child URLs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Parse each crawled HTML page's DOM once instead of twice.
FessXpathTransformerpreviously parsed the response body into a DOM instoreData(field/anchor extraction) and then the inheritedHtmlTransformer.storeChildUrlsparsed the same body a second time for child-URL discovery.Dependency
Requires codelibs/fess-crawler#177 (the
storeChildUrls(ResponseData, ResultData, Document)overload). Merge that first; this PR builds against fess-crawler15.8.0-SNAPSHOTincluding it.Changes
Documentparsed instoreDatainto aThreadLocal(the transformer is a shared singleton across crawler threads, so an instance field would not be thread-safe).storeChildUrlsto reuse thatDocumentvia the new base three-argument overload instead of re-parsing. Its exception handling mirrors the base method exactly — and sinceChildUrlsException extends CrawlerSystemException, the canonical/noindex redirect mechanism is preserved with its original type.transform()in atry/finallythat clears theThreadLocal, so a pooled crawler thread never leaks or reuses a stale DOM. A defensivenullcheck falls back to the base (re-parsing) method.defaultbranch prunes a deep clone instead of the live node.Behavior notes
Because the DOM is now shared between field extraction and child-URL/anchor discovery:
<nav>inside a pruned//H1///H2///H3) from being silently dropped from child-URL discovery. The indexed field value (getTextContent()of the pruned clone) is byte-identical to before.storeDataparse (Fess-detected charset viaInputSource.setEncoding+ UTF-8 BOM strip) instead of a separate neko auto-detected parse — a consistency improvement, converging on the charset already used for content/anchor extraction.anchorfield can now also retain links under pruned-tag subtrees nested in headings.Testing
test_transform_linkUnderPrunedHeading_isDiscovered— a link nested under a pruned<nav>inside a pruned heading is still discovered. Verified to FAIL if the clone-before-prune fix is reverted to in-place pruning (the link is dropped), i.e. it genuinely guards the regression.test_transform_parsesBodyOnce— asserts the response body is read exactly 3 times pertransform()(charset sniff +storeDataparse + document-cache read); there is no 4th read, confirming child-URL discovery no longer re-parses.test_transform_normalPage_fieldsAndChildUrlsUnaffected— ordinary pages extract identical fields and child URLs.FessXpathTransformerTest: 65 tests pass;mvn formatter:format/license:formatclean.