From e5558ad0af68722eee6b3e5bced9427b47205f6d Mon Sep 17 00:00:00 2001 From: Adityaaa <144815351+addydude@users.noreply.github.com> Date: Mon, 9 Nov 2026 08:42:35 +0530 Subject: [PATCH] docs: add CONTRIBUTING guide and link it from README Covers dev setup (including the optional decaffeinator submodule), the pytest + doctor checks, branch and PR workflow, adding browser-free tests, Python 3.11 compatibility, and bug-report/security guidance. Closes #5 --- CONTRIBUTING.md | 206 ++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 8 +- 2 files changed, 213 insertions(+), 1 deletion(-) create mode 100644 CONTRIBUTING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..9086d47 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,206 @@ +# Contributing to WebVac + +Thanks for taking the time to contribute. WebVac is a small, actively +maintained project, so focused bug fixes, tests, and documentation improvements +are genuinely useful. This guide covers everything needed to go from a fresh +clone to a mergeable pull request. + +If anything here is unclear or out of date, open an issue — that is a +contribution too. + +## Good places to start + +- Issues labelled [`good first issue`](https://github.com/Shellghostt/WebVac/labels/good%20first%20issue) +- Reproducing a reported bug with a minimal command and posting the output +- Improving docs, examples, or error messages +- Adding tests for behaviour that is currently untested + +Before starting on a larger change, open or comment on an issue so we can agree +on the direction. It avoids duplicated work and surprise rework. + +## Development setup + +### 1. Fork and clone + +Fork the repository on GitHub, then clone your fork: + +```bash +git clone https://github.com//WebVac.git +cd WebVac +``` + +The `decaffeinator/` submodule is only needed for the optional VAPT task +(`--task vapt`). For normal scrape/crawl work you can skip it: + +```bash +# optional, VAPT only +git submodule update --init --recursive +``` + +### 2. Create a virtual environment and install + +Windows PowerShell: + +```powershell +python -m venv .venv +(Set-ExecutionPolicy -Scope Process -ExecutionPolicy RemoteSigned) ; (& .\.venv\Scripts\Activate.ps1) +python -m pip install --upgrade pip +pip install -e ".[dev]" +python -m patchright install +``` + +Linux/macOS: + +```bash +python3 -m venv .venv +source .venv/bin/activate +python -m pip install --upgrade pip +pip install -e ".[dev]" +python -m patchright install +``` + +`pip install -e ".[dev]"` installs WebVac in editable mode plus the test +dependencies (`pytest`). If you prefer `requirements.txt`, install it first and +then run `pip install -e ".[dev]"` as well; the extra is what adds pytest. + +### 3. Verify the setup + +```bash +python -m webvac --doctor +``` + +The doctor preflight launches a real browser and checks the output directory, +proxies, pipelines, and CapSolver configuration. A few warnings are normal on a +fresh machine (no proxy file, no CapSolver key). Failures mean the environment +is not ready. + +## Running the checks + +Run these before opening or updating a pull request: + +```bash +python -m pytest tests -q +python -m webvac --doctor +``` + +- The unit tests in `tests/` are browser-free and network-free, so they are fast + and should pass on any machine. +- The doctor smoke check needs the Patchright browser installed (step 2). + +Extra checks that CI also performs: + +```bash +python -m compileall -q webvac +``` + +CI runs on Ubuntu and Windows with Python 3.11 and 3.12. The project supports +**Python 3.11+**, so avoid syntax that only exists in 3.12. In particular, an +f-string expression cannot contain a backslash on 3.11 (PEP 701 is 3.12+); build +that string outside the f-string instead. + +### Docker (optional) + +If you are changing the container image: + +```bash +docker build -t webvac:test . +docker run --rm webvac:test --doctor +docker run --rm webvac:test --help +``` + +## Branch and pull-request workflow + +1. Branch from the latest `main`: + + ```bash + git fetch origin + git switch -c fix/short-description origin/main + ``` + + Use a short, descriptive branch name, for example `fix/robots-crawl-delay`, + `docs/contributing-guide`, or `test/storage-roundtrip`. + +2. Keep the change focused. One logical change per pull request is much easier + to review and merge than a mixed bag of unrelated edits. + +3. Add or update tests for behaviour changes. Bug fixes should include a test + that fails before the fix and passes after it. + +4. Commit with a clear message. Conventional prefixes are preferred but not + enforced: + + ```text + fix: respect crawl-delay when --ignore-crawl-delay is absent + test: cover storage round-trip for empty records + docs: document the Docker workflow + ci: pin the Ubuntu runner image + build: add Dockerfile for scrape mode + ``` + +5. Push and open a pull request **against `main`**. Describe: + + - what changed and why, + - how you verified it (commands and output), + - the issue it closes (`Closes #123`). + +6. Make sure CI is green. If a maintainer asks for changes, push follow-up + commits to the same branch; do not force-push after review unless asked. + +7. Pull requests are squash-merged by a maintainer once CI passes and review is + complete. + +## Adding tests + +Tests live in `tests/` and use pytest. They should not need a browser, network +access, or real credentials. Use `tmp_path` for filesystem work and keep each +test independent. + +A minimal pattern: + +```python +def test_something(tmp_path): + target = tmp_path / "data.json" + # arrange -> act -> assert + assert target.exists() is False +``` + +Run one file or one test while iterating: + +```bash +python -m pytest tests/test_storage.py -q +python -m pytest tests/test_proxy.py::test_name -q +``` + +## Code style + +- Follow the existing structure and naming in the package you are editing. +- Keep changes small and readable; avoid unrelated reformatting. +- Prefer the standard library and existing dependencies. Discuss new runtime + dependencies in an issue first. +- Preserve Python 3.11 compatibility (see the CI note above). +- Do not commit secrets, proxy credentials, session files, `scraped_data/`, or + local virtual environments. They are covered by `.gitignore`; keep it that way. + +## Reporting bugs + +A useful bug report includes: + +- the exact command you ran (redact secrets), +- what you expected and what happened instead, +- your OS and Python version, +- the output of `python -m webvac --doctor`, +- relevant logs or a small snippet of `{scan_session}/network/*.json` if the + failure is a blocked/failed page. + +Please do not paste credentials, tokens, or private target data into an issue. + +## Security + +If you believe you have found a security issue, do not open a public issue. +Follow [`docs/SECURITY.md`](docs/SECURITY.md) and report it privately to the +maintainer. + +## License + +By contributing, you agree that your contributions are licensed under the +project's [MIT License](LICENSE). diff --git a/README.md b/README.md index ce9eada..4b9e51e 100644 --- a/README.md +++ b/README.md @@ -381,6 +381,7 @@ Full technical docs (architecture, workflows, diagrams, config, security): - [`docs/SECURITY.md`](docs/SECURITY.md) — Secrets and responsible use - [`docs/webvac-architecture-one-page.html`](docs/webvac-architecture-one-page.html) — Visual one-pager - [`CHANGELOG.md`](CHANGELOG.md) — Release history +- [`CONTRIBUTING.md`](CONTRIBUTING.md) — Development setup, checks, and PR workflow To regenerate architecture PDF: @@ -431,7 +432,12 @@ See also [`docs/SECURITY.md`](docs/SECURITY.md). - [Adityaaa](https://github.com/Shellghostt) — author & maintainer -## Contributing Workflow +## Contributing + +See [`CONTRIBUTING.md`](CONTRIBUTING.md) for full development setup, the local +checks (`pytest` + `webvac --doctor`), branch/PR workflow, and how to add tests. +New contributors can start with issues labelled +[`good first issue`](https://github.com/Shellghostt/WebVac/labels/good%20first%20issue). 1. Create a feature branch. 2. Keep changes focused and test locally.