Repository navigation
v2rayChecker overhaul (0.5.0): batch cores, TCP prefilter, safe shutdown, geo/speedtest/export - #20
Open
mheidari98 wants to merge 27 commits into
Open
mheidari98 wants to merge 27 commits into
mheidari98 wants to merge 27 commits into
Conversation
Breaking modernization: drop Python <3.10, switch to uv + pyproject.toml (hatchling backend), bump deps, add tests + CI, document for AI agents. Changes: - Replace setup.py/setup.cfg/MANIFEST.in/requirements.txt with pyproject.toml - Migrate scripts/* into proxyUtil/cli/* as proper console_scripts - Add --version flag to all 10 CLIs (was 0/5 before) - Migrate cfRecorder from cloudflare v2 to v3+ SDK - Migrate from `from ruamel import yaml` to `YAML()` instance - Switch package data lookup to importlib.resources - Drop dead deps (numpy, cryptography, httpx, jsonlines, Pygments, PyYAML) cutting direct deps from 30+ flat-frozen pins to 8 lower-bounded ranges - Narrow 2 bare `except:` clauses - New tests/ suite: 46 cases (parsers + helpers + DNS + CLI smoke + version) - New .github/workflows/test.yml: ruff + pytest matrix on 3.10/3.11/3.12 - Switch publish workflow from `python -m build` to `uv build` - Add CLAUDE.md for AI agents - Bump 0.1.3 -> 0.2.0
- New proxyUtil/network.py: ScrapURL + downloadZray live here (only side-effecting helpers in the package); deleted from myUtil - Drop `from proxyUtil import *` from 4 legacy CLIs; every CLI uses explicit imports - proxyUtil/__init__.py: no more wildcard re-exports; only __version__ is public - Every submodule (myUtil/dnsUtil/dnsUrl/network/logFormatter/_common) defines __all__ - Drop F403/F405 ruff per-file ignores entirely (no exceptions for legacy code) - Add basedpyright (standard mode) to dev deps + CI; pre-existing legacy patterns silenced via per-rule ignores in pyproject - Add .pre-commit-config.yaml: ruff + standard hygiene hooks - Gate publish workflow with workflow_dispatch (target=test or prod); tag-pushes still publish to both - README updated for uv/ruff/pytest/basedpyright dev workflow
…rix (0.4.0)
- Delete myUtil.py and network.py. Split into focused modules:
utils, schemes, uri, parsers, shadowsocks, net, os_glue, xray,
singbox, cores. No file > 500 LOC.
- Move all CLIs into proxyUtil/cli/ (top-level shims removed; entry-points
in pyproject.toml rewired to proxyUtil.cli.<name>:main).
- New cores.REGISTRY: name -> CoreSpec(binary, schemes, write_config,
run_argv). v2rayChecker takes one --core {xray, v2ray, sing-box}; URLs
whose scheme isn't in the chosen core's set are skipped with debug log.
Drops core_paths dict and the silent xray->sing-box fallback.
- sing-box backend (proxyUtil/singbox.py) now speaks every scheme: vmess,
vless, trojan, ss, hysteria, hysteria2/hy2, tuic, anytls (>=1.11),
shadowtls v3, naive, ssh, wireguard, juicity (lossy: emitted as tuic +
bbr+native). Reality 3-flag invariant enforced. xray-only xhttp/splithttp
auto-downgraded to httpupgrade with one-line warning.
- xray backend variant coverage expanded: ws ed= early-data, gRPC mode/
authority, KCP seed/headerType, QUIC quicSecurity/key, Reality alpn,
xhttp extra JSON-decoded.
- Hot-path: replace deepcopy(mergeMultiDicts(...)) with json.loads of a
pre-rendered template per scheme (~4-8x faster per call).
- Shared helpers in utils.py: split_csv, is_truthy, format_geo. os_glue
exposes runCore/killCore platform dispatchers and augment_local_path.
cores.resolve() consolidates which/download for both v2rayChecker and
connectMe.
- 70 tests green (up from 56). New tests/test_cores.py covers registry,
Reality 3-flag invariant, xhttp downgrade, all new schemes.
- Bump version to 0.4.0; CHANGES.md and CLAUDE.md updated.
- pathlib for IO; reduce(or_,...) for dict-merge; ternaries for trivial branches - list-comp processShadowJson; _ss_userinfo dedup - finder: cache compiled regex via lru_cache, raise ValueError on miss - CustomFormatter caches one Formatter per level (was new per record)
- compile hot regexes once (_RE_SS_USERINFO, _RE_USER_HOSTPORT, _PATTERN_RES) removes quadratic regex compile in checkPatternsInList batch hot path - dispatch table for extractIPs (was if-chain) - partition() over slicing in parse_ss / parse_ss_withPlugin - dict.pop with default in parseVless; walrus in checkPatternsInList/parseTrojan - _HOSTNAME_ONLY_SCHEMES derived from FRAGMENT_TAGGED (not duplicated) - ternary in tagsChanger; consolidated parse_ssr decoder - raise ValueError (not None.groups() AttributeError) on regex miss
- match/case dispatch in build_singbox_config and _transport_block (was if-chain) - dict-merge | for juicity tuic overrides - walrus in transport ed/host extraction - pathlib for writeConfig
- factor 3 resolvers via _resolve(query_fn) — drops triple-duplicated boilerplate - isIPv4/isIPv6 via ipaddress.ip_address (was socket.inet_pton ladder) - compile findURLs regex once - walrus in scrapeDoH - alias Do53_resolver (typo Do53_reolver kept for back-compat)
- pathlib for IO across all CLIs - collect_proxies uses walrus + pathlib - ipExtractor: fix --sort crash on IPv6 (was IPv4Address-only key) - dnsChecker: triple-loop list comp via probes table; --all sets via chained = - cdnGen: dispatch builder; sys.exit replaces exit(); compact dict-comp parse - cfRecorder: pathlib + walrus IP filter - sslocal2ssURI/ssURI2sslocal: list comp over append loop
- dict-comp for proxy template format - pathlib.write_text for sorted output (was open+for-write loop) - shadowChecker: parse_ss_withPlugin via *_ unpack; pathlib pid read; walrus on ping - connectMe: collapse _ss_runner/_v2ray_runner into inline _spawn calls - clashGen: dispatch table _BEHAVIOR_PREFIX (was if-chain); dict-comp rule-providers; flattened DIRECT/REJECT rule loop into one comprehension; range(1,5) for groups
- v2rayChecker/shadowChecker: write empty file when no live proxies (not stray newline) - connectMe: don't early-return on configName=None; fall through to system-proxy cleanup
…rsing parseContent probed every line with isBase64, which fully b64-decodes and re-encodes the line to test it. On real subscription bodies that cost 32s of CPU across 36 sources; scanning scheme prefixes first brings it to 0.02s. It also kept only the first proxy per line (checkPatternsInList broke out of the scheme loop on the first hit) and could not read a body holding one base64 blob per line, or a single blob column-wrapped across lines. Both shapes are common in the wild. Verified against 84 live source bodies: identical output to the previous parser, 0 proxies lost, 0 gained. ScrapURL used requests.get(timeout=4), which bounds individual socket reads but not the request as a whole, so a server that trickles bytes could hang forever with no size cap. fetchSource streams under a wall-clock deadline and a byte cap, and returns a FetchResult carrying status, latency, bytes and the error instead of a bare list. ScrapURLs fetches many sources concurrently, preserving input order. ScrapURL keeps its old signature and return type for callers. tagChanger now raises ValueError on a malformed vmess payload rather than returning the URL with its upstream tag intact, which let callers publish configs they believed they had retagged.
…eedtest/export - one core process serves up to 100 proxies (validated + bisected via the core's own checker); TCP pre-filter drops ~62% of dead servers before any core starts - xray: stop forcing allowInsecure (removed upstream 2026-06-01, failed every TLS config); checker inbound binds 127.0.0.1; per-core ss cipher support checked up front - core crashes are config_error, not dead; probe requires 204/2xx, reports real ms - Ctrl+C/SIGTERM keep partial results and reap every core; --live, --resume, --max-live - --geo via cdn-cgi/trace, --rename/--country/--sort/--format, --stable/--verify/--retries, --speedtest; --core auto; repeatable --url and --sources - offline tests via fake core + real-core e2e (skipped without binaries)
…gments, table names Free-list check: 35/60 'live' proxies answered plain HTTP but black-holed TLS. Renamed names were written raw (spaces, |) so lines split on whitespace when re-read.
… logs, compact speedtest table
- DEFAULT_RENAME is '{flag} {cc} | {name}'; latency via custom template ({ms})
- hide per-config build errors/warnings unless -v (they broke the progress bar)
- stop the bar before the speed test and print its table afterwards; fits 80 columns
…ailure reasons Real-proxy diagnosis: Cloudflare returned HTTP 429 for large downloads (shared exit IPs), first byte often took 5-6 s (read timeout was 5 s), and a mid-stream cut discarded the measurement. Downloads now fall back to OVH/cachefly, keep partial throughput, and failures are shown in the table caption and recorded in --format json.
…y identity A real run spent 1072 of 1229 s in the prefilter: one getaddrinfo that fell back to TCP has no read timeout, and ThreadPoolExecutor joined the hung thread. - net.resolve_hosts resolves each hostname once on daemon threads under a deadline (optionally via a Do53 server); prefilter connects to resolved IPs and reports dropped configs as dns/tcp. v2rayChecker gains --dns and a resolver hint. - proxy_identity is now canonical and tagged id1: (aliases, param order, defaults, vmess types/base64 variants; fp/alpn/insecure ignored): 38.6k -> 23.4k unique on the free list. Journals from older versions simply re-test. - invalid_reason/reject_invalid drop structurally impossible configs offline. - Non-IDNA hostnames no longer crash the TCP check.
… collector The .proxy crawler needs to drop servers that never answer, but a runner outside Iran misses some servers that work from inside (16 of 577 measured). So a listed endpoint must fail three runs in a row; an unlisted one is dropped at once; and nothing is dropped if the runner itself looks offline.
They never show up in real subscriptions and only broke fixtures.
…st to the link After the DNS-hang fix a real run on an LTE home link found 8 live proxies: the modem's forwarder answered ~25 names/s (2600 of 3281 lookups timed out) and 2000 simultaneous SYNs kept only 8% of servers known to be live. - resolve_hosts asks public resolvers first (system resolver for what they cannot resolve), and its deadline now means 'no progress for N seconds'. - prefilter calibrates its socket count: 50 on a link that loses bursts, 2000 otherwise; --prefilter-workers overrides. Measured recall of known-live servers at home: 82% (was 9%).
…nswered names Measured from an Iranian LTE link: every UDP/53 destination (even web servers that are not resolvers) answers a filtered name with 10.10.34.35 in 20-45 ms, faster than the 100+ ms RTT, so it is on-path injection; TCP/53, DoT and DoH did not work. UDP resolvers sustain ~135 names/s whichever is asked, getaddrinfo only ~8/s. - Private answers from the public resolvers are failures (13% of hostnames in the free list) and are not retried or connected to. - Names a resolver said do not exist are not re-asked through getaddrinfo (it resolved 1 of 120); only names no resolver answered are. - The DNS hint now fires only on timeouts.
…)' geo label - link_is_narrow takes the two connect rates; pick_workers drops its nested closures. - The public-resolver breaker counts consecutive silent queries (one counter). - NXDOMAIN is not re-asked as AAAA; resolve_hosts stats always carry 'poisoned'. - format_geo no longer prints '(None)' when only the country code is known (trace endpoints return no country name): 322 of 429 live lines in one run had it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rework of
v2rayCheckerfound by tracing the pipeline and measuring it (13.8k-proxy bench):allowInsecureon 2026-06-01 and our templates forced it, so every TLS config failed silently as "dead". Core crashes are nowconfig_error(not dead); geo lookup no longer drops live proxies; probe requires 204/2xx and reports real ms.0.0.0.0unauthenticated, now127.0.0.1; Ctrl+C/SIGTERM keep partial results and reap every core.--core auto,--live,--resume,--max-live,--geo/--rename/--country/--sort/--format,--stable/--verify/--retries,--speedtest,--sources, repeatable--url.Breaking changes (default core
auto,-T 300, geo opt-in, stricter probe,-i/--t2killdeprecated no-ops) are listed inCHANGES.md.Notes for review
--livehere; this PR's version also covers-o -,--fsync, dedup and an atomic final sorted rewrite. Happy to credit/reconcile however you prefer.--configTOML (reasons in CHANGES.md).Test plan
uv run pytest(206 tests; offline via fake core + real-core e2e that skips without binaries)--geo --rename --stable 3 --verify --speedtest --format json/singbox/b64