Skip to content

perf(crawler): parse HTML once per page in FessXpathTransformer#3183

Merged
marevol merged 1 commit into
masterfrom
perf/fessxpath-parse-once
Jul 10, 2026
Merged

perf(crawler): parse HTML once per page in FessXpathTransformer#3183
marevol merged 1 commit into
masterfrom
perf/fessxpath-parse-once

Conversation

@marevol

@marevol marevol commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Parse each crawled HTML page's DOM once instead of twice. FessXpathTransformer previously parsed the response body into a DOM in storeData (field/anchor extraction) and then the inherited HtmlTransformer.storeChildUrls parsed the same body a second time for child-URL discovery.

Dependency

Requires codelibs/fess-crawler#177 (the storeChildUrls(ResponseData, ResultData, Document) overload). Merge that first; this PR builds against fess-crawler 15.8.0-SNAPSHOT including it.

Changes

  • Stash the Document parsed in storeData into a ThreadLocal (the transformer is a shared singleton across crawler threads, so an instance field would not be thread-safe).
  • Override the two-argument storeChildUrls to reuse that Document via the new base three-argument overload instead of re-parsing. Its exception handling mirrors the base method exactly — and since ChildUrlsException extends CrawlerSystemException, the canonical/noindex redirect mechanism is preserved with its original type.
  • Wrap transform() in a try/finally that clears the ThreadLocal, so a pooled crawler thread never leaks or reuses a stale DOM. A defensive null check falls back to the base (re-parsing) method.
  • Clone-before-prune fix: the field-rule loop's default branch prunes a deep clone instead of the live node.

Behavior notes

Because the DOM is now shared between field extraction and child-URL/anchor discovery:

  • Clone-before-prune prevents links nested under pruned subtrees (e.g. a <nav> inside a pruned //H1///H2///H3) from being silently dropped from child-URL discovery. The indexed field value (getTextContent() of the pruned clone) is byte-identical to before.
  • Child-URL/anchor extraction now runs on the storeData parse (Fess-detected charset via InputSource.setEncoding + UTF-8 BOM strip) instead of a separate neko auto-detected parse — a consistency improvement, converging on the charset already used for content/anchor extraction.
  • The indexed anchor field can now also retain links under pruned-tag subtrees nested in headings.

Testing

  • test_transform_linkUnderPrunedHeading_isDiscovered — a link nested under a pruned <nav> inside a pruned heading is still discovered. Verified to FAIL if the clone-before-prune fix is reverted to in-place pruning (the link is dropped), i.e. it genuinely guards the regression.
  • test_transform_parsesBodyOnce — asserts the response body is read exactly 3 times per transform() (charset sniff + storeData parse + document-cache read); there is no 4th read, confirming child-URL discovery no longer re-parses.
  • test_transform_normalPage_fieldsAndChildUrlsUnaffected — ordinary pages extract identical fields and child URLs.
  • FessXpathTransformerTest: 65 tests pass; mvn formatter:format / license:format clean.

The web-crawl transformer parsed each page's DOM twice - once in storeData
(field/anchor extraction) and again in the inherited storeChildUrls
(child-URL discovery) - reading and re-parsing the full response body a
second time.

Reuse the Document parsed in storeData for child-URL extraction: stash it
in a ThreadLocal (the transformer is a shared singleton across crawler
threads), override the two-argument storeChildUrls to delegate to the new
base storeChildUrls(ResponseData, ResultData, Document) overload with that
Document, and clear the ThreadLocal in a transform() finally block so a
pooled crawler thread never leaks or reuses a stale DOM.

Because the Document is now shared with child-URL/anchor extraction, the
field-rule loop's default branch prunes a deep clone instead of the live
node, so links nested under pruned subtrees (e.g. a <nav> inside a pruned
//H1) are no longer silently dropped from child-URL discovery. As an
intended consequence, child-URL/anchor extraction now runs on the storeData
parse (Fess-detected charset via InputSource.setEncoding + UTF-8 BOM strip)
instead of a separate neko auto-detected parse, and the indexed anchor
field can now also retain links under pruned-tag subtrees nested in
headings.

Requires fess-crawler providing storeChildUrls(ResponseData, ResultData,
Document).

Add tests: a link nested under a pruned <nav> inside a pruned heading is
still discovered (verified to fail without the clone-before-prune fix), the
response body is parsed only once, and normal pages extract identical
fields and child URLs.
@marevol marevol added this to the 15.8.0 milestone Jul 10, 2026
@marevol marevol self-assigned this Jul 10, 2026
@marevol
marevol merged commit 7611132 into master Jul 10, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant