Repository navigation
Log in to a site with a browser, and crawl with its cookies - #257
Merged
Merged
Conversation
The Spider page gets a "Log in with a browser..." button. It opens a WebView2 window on the project's first address. When the user closes it, the program takes every cookie the browser then holds, and the browser's own User-Agent. The cookies become the Netscape file the engine preloads, and the two option fields are set to match. WhttBrowserLogin.dll grows the window and the harvest, so the ABI number moves to 2 and the loader resolves one more export. The window is modal by hand, because the caller's message loop is the one it stands in. It also hands the page every Tab and Enter while the page holds the focus. Which address the browser opens, and whose profile it opens it with, is its own unit, so tools/test-browser-login-paths.py can run those cases off Windows. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The window waited for the cookie answer with no way out. A browser that died behind it froze the whole program, because the owner stayed greyed out and every later close was swallowed. A ProcessFailed handler now ends the window, and a second close gives up on the harvest. The keyboard could not cross between the window's own controls and the page. Tab off the last control now hands the focus to the browser, and MoveFocusRequested hands it back. The loop asks Windows which window has the focus rather than tracking it, which also drops a stale-flag race. The rest answers the review. A controller built after the window went is now closed rather than orphaned. The caller tells "nothing to write" from "would not write" through a result type. The browser identity is set by the page that owns the control, which is also what the option-page wiring check demands. Three checks that passed against a wrong implementation now fail against it. A control character in the browser identity is refused, as it already is in an Accept value. The engine flattens CR, LF and TAB into spaces, and a browser login is the first thing to fill that field on its own. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The first round's fix left the keyboard dead until the browser had started. The focus sits on the frame until then, and the new gate only let the dialog manager see the three child controls. The frame now counts as one of ours. A page tabbing out when a control had failed to be created called SetFocus(NULL). That leaves the thread with no focus window at all, and no way back. It falls back to the frame, and it answers only the two reasons a page actually tabs out with. Done no longer posts WM_CLOSE. A double click sent two, and the second was what gave up on the harvest the first had started, so the user got nothing. Giving up now reports that the browser stopped, rather than reporting an empty success, and the three controls grey out while the answer is awaited. WM_DESTROY clears the window handle before it closes the controller. Should closing pump messages, the cookie answer it delivers must not call DestroyWindow on the window it is already inside. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Pressing Done and then the close box threw the login away. The close stamped the answer as "the browser stopped". But closing the browser can deliver the cookie answer it had stopped waiting for. A harvest that did arrive was then reported as a failure, and dropped. The window now only records that it gave up. What that means is decided once the window has gone, and it only means the browser stopped when nothing came back. A page handing the focus back while the harvest runs aimed it at a control the harvest had greyed out, so nothing took it. It falls back to the frame for a disabled control as it already did for a missing one. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The last commit decided the two apart by whether any cookie had arrived, and an empty jar is not the same thing. A login that set no cookie, on a profile that held none, was reported as the browser having stopped. The browser identity went with it, which is the loss that commit set out to stop. The harvest now records that the answer came, and that is what the verdict reads. A page handing the focus back while the window is being destroyed has no control left to take it. The frame fallback was then SetFocus(NULL), so the handler now leaves the focus alone once the window is going. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
A real build crashed. The user pressed Done on an ordinary page, the window shut, and the program died afterwards inside the browser's code. The cookie answer arrives as a callback. We closed the browser and destroyed the window from inside it, then returned. The browser then resumed its own dispatch on everything we had just freed. The disassembly puts the fault on the return from DispatchMessageW in our own loop. None of our frames sit above it, which is that sequence exactly. A callback now only posts a message, and the teardown runs from the loop. It closes the browser before destroying the window that hosts it, which is the order Microsoft's own Win32 sample uses. WM_DESTROY no longer touches the browser at all. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Moving the close out of WM_DESTROY left two exits uncovered. The loop breaks without tearing anything down when the application posts WM_QUIT, or when GetMessage fails. The tail then destroyed the window with the browser still open, which is the crash this round fixes, reached from the other direction. The tail now runs the same teardown as every other exit. Closing can pump messages, and a finish already posted would then come straight back in on a controller halfway through being closed. A flag makes the teardown run once. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The Spider page was the wrong home. Everything else there is a setting that is stored and replayed next run, while this one opens a browser and writes a file. It also had to reach two wizard globals for the project path and the start address. Worse, it broke the options sheet's Cancel. That sheet restores the profile when the user cancels. The cookies file was already written by then, and the restore cannot remove it. So the login was stranded, and the field pointing at it was reverted. It now sits in the Insert URL box, under the same "Forms or complex links:" heading as Capture URL. That button does the same job with a proxy and a browser of the user's own. The box also holds the address to open, and it already knows the project path. Nothing is written until the box is accepted. The harvest is kept in memory and applied on OK, beside the URL, so Cancel leaves no file and no setting behind. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The test only asked that Onlogin2() mention applyBrowserLogin somewhere. Move the call above the IDOK branch, or drop the haveHarvest guard, and it still passed. Either slip writes a cancelled login to disk. That is the whole reason the harvest is held back, so the test now asks where the call sits. Both of those were mutated and the test reds on each. The rest is tidying after the move. OptionTab8 keeps an include it no longer needs. Wid1 took browser-login.h only through another header. haveHarvest was set in OnInitDialog rather than beside the other members. One comment claimed too much. Cancel leaves no cookies file and no setting, but the browser still keeps its own profile, which is what makes the next login quick. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
…anslated The login gave back the cookies and the identity but not the address. The Add URL box stayed empty and the wizard could not continue, so Xavier called the feature unusable and he was right. A browser is opened so the user can navigate somewhere by hand, and that somewhere is the address to crawl. The window now answers with it, and the field fills the moment it closes. A test reds if that call goes. A Clear login button sits beside Go and Done. It forgets the cookies the profile holds, which is how the user logs out or logs in as somebody else. The profile is per project, so clearing one project leaves the others alone. The window reached no language catalog, so its title and buttons were English whatever the user had chosen. The caller now hands them over, and the ABI moves to 3 for the four new fields. Fourteen strings gain catalog keys, each with the English behind it. LANGSEL answers an empty string for a key the engine does not carry yet, and an empty message box tells nobody anything. The catalogs themselves are the engine's and go there separately. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The engine is rewording LANG_COOKIEFILETIP. Catalogs join on the English text. Reword lang.def without rewording the key line in all 30 catalogs and that tooltip disappears in every language at once, English included. Nothing errors, because GetTip has no fallback and simply returns an empty string. The selftest already fails on an empty catalog key, so this adds the one being reworded to the list it checks. The smoke-test matcher moves with it. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
xroche
added a commit
to xroche/httrack
that referenced
this pull request
Oct 21, 2026
This adds the 14 WinHTTrack browser login strings (xroche/httrack-windows#257) to `lang.def` and all 30 catalogs, with the translations made for that work. The cookies-file tooltip now says the file is written back too. Its English text changes in every catalog, because each catalog finds a translation by its English text. Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The address box is saved into the project's winprofile.ini in clear, so whatever the browser finishes on is written to disk. A sign-in can finish holding an access token in the fragment, and a hand-typed address can carry a password as userinfo. Both reached the file. The fragment is now dropped, which costs the crawl nothing because a fragment never goes on the wire. An address carrying userinfo is refused whole rather than stripped. The engine would otherwise authenticate with it, and the same dialog already has its own login and password fields. An address the crawler could not fetch at all, about: or edge:, is refused too. The query is kept, because a crawl start legitimately carries one. A sign-in code can ride there, so that one is still open. Found by the httrack-android session in their own diff, where all three reached shared storage. None of it was Android-specific. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The profile held a live session and sat in the project folder, so copying, zipping or backing up a project carried someone's login with it. It also grew without bound, because the browser caches in there. Neither is something the user asked for by pressing a login button. It moves under LOCALAPPDATA, one folder per run of the program, named after the process id. Closing the program removes it. A crash or a kill cannot, so the next startup sweeps every profile whose run has ended. That sweep is the only thing that ever removes one of those. A profile whose run is still alive is left alone, because a second copy of the program may be signed in right now. Skipping is the safe direction: the next startup gets it, where deleting it would sign that copy out mid-login. What this costs is a fresh login each time the program starts. A site that remembers the device will ask for its second factor again. Within one run the profile is shared, so a second capture still finds the user signed in. The delete is its own walk rather than RmDir, which raises a message box on every failure. A profile the browser still holds open fails often, so a sweep using it would greet the user with a stack of dialogs. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Bumping the browser-login count from 16 to 38 was done with an unscoped sed, and another block in the same file also expected 16. That block now demanded 38 from a loop that runs 16, so every runtime leg failed with "engine results wait ran 16 checks, expected 38". Nothing but the number moved, and the browser-login count is unaffected. Diffing every pinned count in the file against master found it in one step. That is the check worth running after any count bump: each selftest block carries its own count, and several share a value. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The walk photographed every documented screen except this one, so the pane the browser-login button landed on had no UI coverage at all. It now opens the box off the URL pane, hit-tests the button to prove nothing covers it and that it sits inside the client area, prints whether the runner's WebView2 runtime left it available, and cancels. The box is found by its address field rather than by its caption, because the caption translates and the walk takes --language. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
A login can be redirected anywhere, and the address the browser finishes on is what fills the URL box and starts the crawl. So a login that wandered off would quietly crawl a host the user never asked for, carrying the cookies it collected on the way. The box is still filled, because an empty one leaves the wizard with no URL and that is the failure this feature already shipped once. The warning names the host so the user can correct the field before Next. The comparison is the host, not the registrable domain: the engine's hts_psl_registrable_domain() is behind HTS_INTERNAL_BYTECODE and is not exported, and HTTrack's default crawl scope is the host anyway. A leading "www." is not a difference. Also fixes a drift the diff review caught: the --selftest block compared the case count against one number and printed another, so a lost case would have passed with a correct-looking message. The ubuntu harness now pins the comparison, not only the message. Signed-off-by: Xavier Roche <xroche@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The "Log in with a browser..." button in the Add URL box opens a WebView2 window on the address being added. Press OK and the cookies go to the project's own
hts-cache/cookies.txt, and the browser's own User-Agent goes in the browser identity field. Cancel writes nothing.The harvest is the whole browser profile, not one site's. A login that crosses two hosts leaves the session cookie on the host nobody asked about. That profile holds a live signed-in session, so it lives under LOCALAPPDATA, one folder per run. A sweep at startup takes what a crash left behind.
The address the browser finishes on fills the URL box. A message box warns when its host is not the one the login started from. The address still goes in, because an empty box leaves the wizard with no URL.
Test fixtures are live at https://www.httrack.com/tests/login/. The cross-host one needs an engine whose cookie matching honours host-only cookies, which the 3.50.5 tag does not.