Repository navigation
Conversation
A start URL ending in #httrack-bundle=<name> crawls with the user agent, headers and cookies of <name>.hts. The file is read from the project folder, or from the folder given to --bundle-dir (-@b), and the engine never writes it. The suffix is read on argv, -%L and doit.log, and cut before the URL reaches any buffer. The bundle's UA and headers replace -F and -%X only for hosts in its scope: the registrable domains of its URLs and cookie domains, or its scope lines. A cookie outside the scope is dropped with a warning. Overlapping bundles must agree on UA, headers and cookie values. Bundle cookies load first and cookies.txt or -%K loads over them. The per-bundle table is a registry in htsbundle.c keyed by opt, so no installed struct changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Keep the bundle state in two httrackp tail fields (bundle_dir, bundle_state) instead of a global registry and its mutex, and bump the VERSION_INFO revision. Load bundle cookies straight into the crawl's jar through a scope filter in the existing Netscape loader, which -%K and cookies.txt still call without one. Of the bundle and the jar, the newer file wins, and the bundle wins a tie. Three security bugs are fixed: - a refused header's message printed its value, which can be a secret, so it now prints the header name only - a start URL on a public suffix or a single label scoped its whole subtree, so it now scopes to that host alone - a CR or other control byte inside a bundle line reached the wire, so the line is now refused --bundle-dir is long-only and doit.log keeps it absolute. Cookie domains no longer widen the default scope, and cookies outside it are dropped with one summary warning. Header lines go over https alone when every URL naming the bundle is https. A repeated cookie or unknown comment warns instead of stopping the run. Windows device names are refused as bundle names, and scope and url lines are capped at 256. The new prototypes moved out of the installed htsbauth.h, which is back to master's bytes. The cookies.txt load order for runs without a bundle is master's again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
A bundle cookie line with a control byte other than TAB now stops the run. Only the bundle path changed, so -%K and cookies.txt read as on master. No bundle message prints a byte of a bundle line any more. It gives the file, the line number and the reason, and names an unknown key only when the key is all lowercase letters. The https rule follows the design. When a URL naming a bundle or one of its url lines is https, its headers go over https alone. Bundles that merge share the rule, whichever one is listed first. The cookie filter is now an admit callback, and the scope and clash checks live in htsbundle.c. A full comparison jar stops the run, and so does a header block over 8192 bytes. One helper cuts the suffix for the command line and for -%L. The dump function is gone, and the self-test reads the identity each host gets instead. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A bundle cookie line that is too long, or that holds a control byte anywhere, now stops the run. Before, a long line with a control byte only warned. Under -b0 the cookie section is still read into a scratch jar, so a bundle is valid or not whatever -b0 says. -%K and cookies.txt load as before. The https rule now merges bundles with a union-find in the existing pair loop, rather than a fixpoint that was cubic on scope chains. A header line limit counts the line without its CR, so LF and CRLF files agree. A cookie over 4096 bytes names that as the reason. htscore.c gets two helpers for the bundle start, COOKIE_WARN becomes a function, bundle_cut is static, and the tests drop GNU touch -d. Test 656 adds a redirect out of the bundle's site and back, and a userinfo URL that spells the site's name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Floor the queued rows in 655, accept a 4096-byte cookie pair, time 2000 chained bundles, print a skip line for the sub-second order row, and shorten three comments. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The 2000-chain row now compares its load time with a 250-chain load in the same run, so slow legs do not trip an absolute bound. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <roche@httrack.com> # Conflicts: # src/htsbauth.c
Expand a possibly empty array the bash 3.2 way under set -u, skip the 2000-URL timing row where the Windows command line is too short, and write the bad -%L list through the shell path rather than the native one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A start URL ending in
#httrack-bundle=<name>crawls the hosts in its scope with the user agent, headers and cookies of<name>.hts. A missing, broken or conflicting bundle stops the run before any request, and no message shows a value from a bundle line. The headers stay off plain http when any bundle URL is https, and a public suffix such ass3.amazonaws.comscopes to that host alone.