Repository navigation
Conversation
hts-ioinfo.txt printed every request and response header in clear, so a user sharing it for a bug report shared their Authorization, Proxy-Authorization and Cookie values, and the Set-Cookie values the server gave them. The FTP control log printed PASS in clear too. fprintfio(), the only writer of header blocks to that log, now replaces the value of those four fields with "<redacted, N bytes>". The name is matched as header_name_len() does: case-insensitive, with blanks before the colon. A folded continuation of a hidden field is hidden too. The FTP PASS argument uses the same marker. The bytes sent on the wire do not change. Test 657 crawls with a cookies.txt cookie, user:pass in the URL and two -%X login headers, plus an FTP login, and checks the log against the wire. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Two cookie-file warnings in hts-log.txt printed up to 64 bytes of the refused line, value included. A pair with no name is now logged without its text. An unknown comment line is logged up to its first TAB, which holds the domain rather than a value. The other warnings already print the name only. Test 657 plants a value in each refused line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
hts-log.txt printed URL passwords in several places: the "launched at" line, the command-line echo, and warnings that name a link. The echo also printed -%X login headers in clear. hts_log_vprint() now formats each line and passes it through hts_mask_passwords(), which writes user:***@ in place of user:pass@. The echo goes through the same helper, and then through fprintfio() for the header redaction. doit.log, new.txt and new.zip keep the URL, because --update needs it. A cookie-file comment line with no TAB was still logged whole. The warning now gives the line number and no line text. Test 657 now checks that look-alike headers and the FTP USER stay in clear. It also tests a bare cookie value on its own, and passwords from HTTP, FTP and -P in hts-log.txt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| nl = 0; | ||
| fputc(*buff, fp); | ||
| nl = HTS_FALSE; | ||
| fwrite(buff, 1, keep, fp); |
The old matcher sized its copy at twice the input, but each ":@" grew to ":***@", so 300 pairs in a hostile link overflowed the heap through the 404 log line, and a local argument overflowed the echo. It also stopped at the first '@' where the engine splits at the last one, and masked any ':' before an '@', such as the filter -*:*@*. The new matcher counts first, then copies. It masks the userinfo where jump_identification_const() finds it: after a scheme://, at the start of a token that goes on to a path (the engine logs http addresses with no scheme), on each start URL, and on each -P value, found from the option spans the parser recorded. The maskpw self-test holds the rows, and test 657 crawls a password through hts_log_print. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The previous round rewrote every hts-log.txt line with a heuristic that masked any "a:b@c" it found. It missed debug lines that print a bare address, and it altered text that held no password. Log lines now go out as formatted. Each log call drops the login at the source: an engine address goes through adr_nologin(), and a full link goes through URL_NOLOGIN(). The command-line echo still masks only the start URLs and the -P value, found through the option spans. Test 657 crawls with -z -Z -%H and passwords planted in the start URLs, in links of every form the engine reads, and in -P. It then greps every project file except doit.log, new.txt and new.zip, which --update needs. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fix the round-4 review of PR #2000. Two kinds of debug line still printed a login. "link #N is ready, skipping" printed urladr() raw. The two JavaScript debug lines printed the first 32 page bytes, which can hold a CSS url() with a login. The JavaScript lines now mask the whole fragment before they cut it, or print "(hidden)" when no scheme lets them find the login. adr_nologin() and url_nologin() now keep the scheme and the user and write user:***@, so a log line that compares two addresses stays readable. They take a rotating buffer of opt only for an address with a password. A crawl without a login logs what it did before. "file://" is its own host, which the suite caught as a crash. The -%H log masks the URL of a Location or Content-Location field. The start URLs on stdout and the --sitemap-url and --referer values now go through the command-line echo masking. Test 660 fails on any log call that prints an adr, url_adr or urladr() outside the helpers, with an allowlist. Test 657 now reaches the skipped image, a CSS url(), a redirect, robots.txt, a sitemap and stdout. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
xroche
changed the title
The -%H log now masks the URL login in a request line, as a proxied ftp:// target carries one, and in a Referer, as -%R sets one. The debug line for a data attribute that is no URL masks it too. The -%v1 console lines mask the login of each file they list. A login with no password is masked as ***@, because a token is often sent as the user name. adr_nologin() with a NULL opt drops the login, as back_delete(NULL, ...) logs one. The lint now also checks printf, fputs and fprintf to stdout or stderr, and skips code that a debug switch compiles out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The log helpers found a login only after a scheme the engine fetches, so an mms:// or gopher:// link printed its password. They now take any "scheme://". These print the login masked now: a quoted -%L line, a refused quoted argument, the --why answer and its echo, and the over-long seed message. The -%v display and WebHTTrack's drop the login. A -%X warning gets the same redaction as the -%H log. Lint 660 now also checks hts-ioinfo.txt writes and the lien, url, link, line and codebase variables, ignoring comments and sizeof(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
A quoted -%L line without a scheme now goes through adr_nologin, so its login is masked. A link with blanks before its scheme is masked too, because login_start() skips them. The wizard's relative-link debug line, the rejected --host-alias rule and a -%X Authorization, Proxy-Authorization or Cookie line with no colon no longer print a password. Lint 660 now flags any identifier containing adr, url, lien or link, and puts() and fprintfio() calls. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The per-site masking finds a login by its place in a URL. As a second layer, the engine now keeps each password and token it learns. They come from the start URLs, -P, --why, every recorded link, the secret -%X values and the Basic tokens it builds. hts-log.txt, hts-ioinfo.txt, the error message and the -%v display then print each of them as ***. A secret of 8 bytes or more is hidden anywhere. A shorter one is hidden only as a login, directly before an '@', so a password such as "index" or "html" leaves index.html alone. The set keeps the first 64 secrets of up to 1 KB each, and logs one warning when a page offers more. With no login, the set is empty and no line is scanned. The set lives in a new tail field, httrackp.secrets, and the front ends reach it through a new export, hts_hide_secrets(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Each secret now has a kind. A URL password is masked only in login position, at any length: followed by '@', with its userinfo starting a word or right after "://". So a password such as "download" or the host name leaves /download/file.html and the Host line alone, and mailto:info@ is not taken for a login. A header token of 8 bytes or more is masked anywhere, and a shorter one is not kept. A secret over 1 KB is skipped with one warning. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s out of the token set Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
xroche
enabled auto-merge (squash)
October 21, 2026 20:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
-%Hlog,hts-log.txt, the-%vdisplay and the argument messages hide login headers, the FTPPASScommand and a URL password (user:***@). As a backstop, every log hides each known password in a login, and each header token of 8 bytes or more anywhere. The archive and report outputs keep URLs as given (doit.log,new.txt, WARC, WACZ,--changes,-%Mand the-#Cdump), because they record what was crawled.