diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index e4350ca..d5e1280 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -11,6 +11,7 @@ on: - 'Cargo.toml' - 'Cargo.lock' - 'README.md' + - 'docs/**' - '.cargo/**' - 'rust-toolchain*' - '.github/workflows/rust.yml' @@ -25,6 +26,7 @@ on: - 'Cargo.toml' - 'Cargo.lock' - 'README.md' + - 'docs/**' - '.cargo/**' - 'rust-toolchain*' - '.github/workflows/rust.yml' @@ -66,6 +68,8 @@ jobs: cargo test --no-default-features cargo test --features std cargo test --features entities + RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --no-default-features + RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --features entities cargo clippy --all-targets --no-default-features -- -D warnings cargo clippy --all-targets --features entities -- -D warnings - name: Benchmark smoke checks (no performance claims) diff --git a/README.md b/README.md index 0a36662..57343af 100644 --- a/README.md +++ b/README.md @@ -1,51 +1,73 @@ -# `rustedbytes-tl` +# rustedbytes-tl -[![Crates.io Version](https://img.shields.io/crates/v/rustedbytes-tl)](https://crates.io/crates/rustedbytes-tl) +[![Crates.io](https://img.shields.io/crates/v/rustedbytes-tl.svg)](https://crates.io/crates/rustedbytes-tl) -tl is a fast HTML parser written in pure Rust. +A lightweight HTML parser written in Rust. It borrows input bytes and offers +allocation-free bounded parsing by default, or heap-backed DOMs and CSS queries +with `std`. The Rust crate is imported as **`tl`**. -By default this crate builds without `std` or `alloc`. Enable `std` for the -allocating convenience API. The examples below target the current Git revision -(next breaking 0.3.x release), including APIs not present in published 0.2.0: +This README describes the current Git source. The manifest still says `0.2.0`, +but this branch contains breaking changes intended for the next `0.3.x` release. +The published [0.2.0 API reference](https://docs.rs/rustedbytes-tl/0.2.0/tl/) +does not describe all APIs below. Generate current API docs with +`cargo doc --features entities --open`. -```toml -rustedbytes-tl = { git = "https://github.com/RustedBytes/rustedbytes-tl", features = ["std"] } -``` +## Getting started -For the nightly portable SIMD path, enable `portable-simd` and build with -nightly: +Use the current source with heap-backed parsing: -```sh -cargo +nightly build --features portable-simd +```toml +[dependencies] +rustedbytes-tl = { git = "https://github.com/RustedBytes/rustedbytes-tl", features = ["std"] } ``` -## Queries and decoded text - -Enable `std` for structural CSS queries and safe owned DOMs: +Commit your application's Cargo.lock to retain the resolved Git revision. ```rust # #[cfg(feature = "std")] { -let guard = tl::VDomGuard::parse( - "".into(), - tl::ParserOptions::default(), -).unwrap(); -assert_eq!(guard.get_ref().query_selector("ul > li a[href]").unwrap().count(), 1); +let html = ""; +let dom = tl::parse(html, tl::ParserOptions::default()).unwrap(); +let handle = dom.query_selector("ul > li a[href]").unwrap().next().unwrap(); +let node = handle.get(dom.parser()).unwrap(); +assert_eq!(node.inner_text(dom.parser()), "Item"); +let tag = node.as_tag().unwrap(); +let href = tag.attributes().get("href").unwrap().unwrap(); +assert_eq!(href.as_bytes(), b"/item"); # } ``` -Enable `entities` to use `decoded_inner_text` for HTML character references. -`inner_text` continues to return the original undecoded text. -See [compatibility notes](docs/silkworm-compatibility.md) for supported selectors, -remaining HTML5 differences, query limits and release compatibility. +Keep `html` alive for every use of the borrowed DOM. If the DOM must own its +input, use the safe `tl::VDomGuard::parse(String, ParserOptions)` constructor. +See the [usage guide](https://github.com/RustedBytes/rustedbytes-tl/blob/master/docs/usage.md) for owned input, attributes, decoded text, +query errors and bounded parsing. + +## Features + +| Feature | Behavior | +| --- | --- | +| Default (none) | `no_std`, no allocation; explicit capacities in `parse`; bounded simple selectors | +| `std` | Heap-backed storage, owned DOMs, text extraction, structural CSS selectors | +| `entities` | Implies `std`; adds character-reference decoding through `decoded_inner_text` | +| `portable-simd` | Requires nightly Rust; optional SIMD parsing path, combinable with the above | + +`__INTERNALS_DO_NOT_USE` exposes instrumentation internals and is not a supported +application feature. Stable builds should select features explicitly rather +than use `--all-features`, which enables nightly-only `portable-simd`. -For checks, benchmarks, safety changes and migration guidance, see the -[Rust audit notes](docs/rust-audit.md). +## Scope and documentation -## Provenance +The parser does not implement the complete HTML5 tree-building algorithm. +Malformed HTML can produce a different tree from a browser; omitted `tbody` +elements are not inserted. CSS support is a subset, and text extraction does +not implement browser layout or visibility rules. HTML parsing and text +extraction do not sanitize untrusted content. -This crate is a fork of [`astral-tl`](https://github.com/astral-sh/astral-tl), modified to -add no-std, zero-copy parsing and other improvements. +- [Usage guide](https://github.com/RustedBytes/rustedbytes-tl/blob/master/docs/usage.md): executable examples and API behavior. +- [Development guide](https://github.com/RustedBytes/rustedbytes-tl/blob/master/docs/development.md): feature checks, docs, benchmarks and troubleshooting. +- [Migration and silkworm compatibility](https://github.com/RustedBytes/rustedbytes-tl/blob/master/docs/silkworm-compatibility.md): supported selectors and remaining compatibility work. +- [Rust audit](https://github.com/RustedBytes/rustedbytes-tl/blob/master/docs/rust-audit.md): safety fixes and measured performance, with methodology and limits. -## License +## Provenance and license -MIT. +Forked from [astral-tl](https://github.com/y21/astral-tl) and extended for bounded +`no_std` parsing and RustedBytes consumers. Licensed under MIT; see [LICENSE](https://github.com/RustedBytes/rustedbytes-tl/blob/master/LICENSE). diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..965f147 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,86 @@ +# Development and verification + +Run commands from the repository root. `std` changes public signatures and +storage behavior, so checking only one feature configuration is insufficient. +CI pins Rust 1.99.0 for stable checks and checks compilation on 1.92.0. +The manifest does not declare an MSRV; the compatibility job is not an MSRV promise. + +## Stable checks + +```sh +cargo fmt --check +cargo test --no-default-features +cargo test --features std +cargo test --features entities +cargo clippy --all-targets --no-default-features -- -D warnings +cargo clippy --all-targets --features entities -- -D warnings +``` + +README examples are included in crate rustdoc. The usage guide is also included in crate rustdoc; Cargo runs its doctests +with the selected features. +To check documentation alone: + +```sh +cargo test --doc --no-default-features +cargo test --doc --features std +cargo test --doc --features entities +RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --no-default-features +RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --features entities +``` + +Examples hide feature guards in rendered documentation so the same source can +be tested for all three configurations. Keep each example's feature requirement +clear in its surrounding text. Verify local Markdown links when moving pages. + +## Nightly and safety checks + +Stable `--all-features` fails because `portable-simd` requires nightly. The CI +nightly is pinned separately: + +```sh +cargo +nightly-2026-09-25 test --no-default-features --features portable-simd +cargo +nightly-2026-09-25 test --features entities,portable-simd +``` + +Install Miri for that nightly before focused safety checks: + +```sh +rustup component add miri --toolchain nightly-2026-09-25 +cargo +nightly-2026-09-25 miri test --lib --features std tests::audit +cargo +nightly-2026-09-25 miri test --lib --features std owned +cargo +nightly-2026-09-25 miri test --lib --features std inline +``` + +The workflow's manual `miri` input enables these focused groups. They exercise +known unsafe boundaries, not every possible program execution. + +## Benchmarks + +```sh +cargo bench --bench tl -- --test +cargo bench --features std --bench tl -- --test +cargo bench --features entities --bench audit -- --test +cargo bench --features entities --bench audit +cargo run --release --features entities --example audit_allocations +``` + +`--test` is a smoke check, not a timing measurement. `tl` benchmarks parsing; +`audit` benchmarks queries and text on a prebuilt DOM. The allocation example +measures requested allocation activity separately. See the +[audit report](rust-audit.md) for fixtures, baselines, confidence intervals and +limits. Reproduce baseline and candidate with identical toolchains, dependency +resolution, features and flags; avoid concurrent builds during timing runs. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Examples fail to find `rustedbytes_tl` | Import `tl`; the package and library names differ | +| `parse` requires const arguments | Enable `std`, or provide all six bounded capacities | +| `decoded_inner_text` is missing | Enable `entities`, which also enables `std` | +| Query is rejected | Check the supported CSS subset; rejection differs from an empty iterator | +| Tree differs from a browser | Review HTML5 recovery limits in the compatibility notes | +| Capacity error | Increase the relevant bounded budget or use heap-backed `std` parsing | + +Docs-only paths trigger CI too. This workflow checks code and docs; it does not +publish a crate, bump the version or create a release. diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..f1fbe8b --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,135 @@ +# Usage guide + +These examples target the current Git source. Import the package +`rustedbytes-tl` as `tl`. Enable `std` for heap-backed parsing, and `entities` +for decoded text; the latter already enables `std`. + +## Borrowed and owned input + +`tl::parse(&str, ParserOptions)` borrows the input. Node handles must be resolved +against the parser that produced them; a handle is not an independently owned +node or a globally unique identifier. Do not use handles from another DOM. + +For input obtained as a `String`, the guard keeps that input alive: + +```rust +# #[cfg(feature = "std")] { +let guard = tl::VDomGuard::parse( + String::from("

Hello

"), + tl::ParserOptions::default(), +).unwrap(); +let dom = guard.get_ref(); +let handle = dom.get_element_by_id("message").unwrap(); +assert_eq!(handle.get(dom.parser()).unwrap().inner_text(dom.parser()), "Hello"); +# } +``` + +DOM references and cloned nodes obtained from the guard cannot outlive it. +`get_mut_ref` also returns a shared DOM reference; its name does not grant DOM +mutation. The legacy `unsafe parse_owned` entry point remains available, but +new code should use `VDomGuard::parse`. + +`ParserOptions::default()` does not build ID/class indexes. With `std`, +`get_element_by_id` and `get_elements_by_class_name` still work by scanning. +Use `ParserOptions::default().track_ids().track_classes()` when repeated +lookups justify building indexes. CSS query support does not require these flags. + +## Queries and attributes + +With `std`, `query_selector` returns `None` for invalid or unsupported syntax. +A valid query returns an iterator, which can be empty: + +```rust +# #[cfg(feature = "std")] { +let dom = tl::parse("", Default::default()).unwrap(); +assert!(dom.query_selector("button:unsupported").is_none()); +assert_eq!(dom.query_selector("article").unwrap().count(), 0); +let button = dom.query_selector("button[disabled]").unwrap().next().unwrap(); +let attributes = button.get(dom.parser()).unwrap().as_tag().unwrap().attributes(); +assert!(attributes.get("missing").is_none()); +assert!(matches!(attributes.get("disabled"), Some(None))); +assert_eq!(attributes.get("title").unwrap().unwrap().as_bytes(), b"Save"); +# } +``` + +The nested attribute result distinguishes missing attributes (`None`), present +attributes without a value (`Some(None)`) and values (`Some(Some(bytes))`). +Attributes retain raw character references. `Bytes::as_bytes()` gives bytes; +`try_as_utf8_str()` validates UTF-8 and `as_utf8_str()` (with `std`) converts +lossily. Initial spans borrow input; explicitly replacing bytes can own data. + +Supported `std` queries include type, ID, class, universal, attribute presence +and value matching (`=`, `~=`, `^=`, `$=`, `*=`), compound selectors, comma lists, +descendant, child and sibling combinators, `:first-child`, `:nth-child(an+b)`, +`:not(...)`, and descendant-relative `:has(...)`. Element positions ignore +text and comments. Type names match ASCII-insensitively. CSS escapes, +namespaces, attribute case flags, other pseudo-classes and `:has(> a)` are +unsupported. Recursive nesting and compound size are bounded; see the +[compatibility notes](https://github.com/RustedBytes/rustedbytes-tl/blob/master/docs/silkworm-compatibility.md). + +Structural queries build indexes proportional to DOM node count. `:has` can +scan many candidates; it is not guaranteed to take linear time. Calling +`Selector::matches` directly lacks the structural context supplied by the DOM +query iterator. + +## Raw and decoded text + +`inner_text` concatenates descendant text, ignores comments and leaves entities +encoded. It does not insert layout spaces or filter hidden elements. With +`entities`, opt into decoding: + +```rust +# #[cfg(feature = "entities")] { +let dom = tl::parse("

A & B 🦀

", Default::default()).unwrap(); +let handle = dom.query_selector("p").unwrap().next().unwrap(); +let node = handle.get(dom.parser()).unwrap(); +assert_eq!(node.inner_text(dom.parser()), "A & B 🦀"); +assert_eq!(node.decoded_inner_text(dom.parser()), "A & B 🦀"); +# } +``` + +Decoding is per text node: markup boundaries cannot form a new entity. +Script/style tags preserve raw text; directly decoding a raw child uses ordinary +text-context rules. Decoding does not change source bytes or attributes. +Returned text is `Cow` and may borrow or allocate depending on the content +and tree. Do not assume every extraction is allocation-free. + +## Bounded parsing without std + +Without `std`, `parse` takes six capacities in this order: nodes, parser stack, +root nodes, tracked IDs, tracked classes and selector nodes. Include text and +comment nodes in the node budget. Capacities also affect stack/storage size; +choose them for your input rather than copying large arbitrary bounds. + +```rust +# #[cfg(not(feature = "std"))] { +let dom = tl::parse::<32, 16, 8, 8, 8, 16>( + "

Hello

", tl::ParserOptions::default(), +).unwrap(); +assert!(dom.get_element_by_id("message").is_some()); +assert_eq!(dom.query_selector("p").unwrap().count(), 1); +# } +``` + +Bounded selectors support simple matching, not the structural CSS engine +available with `std`. Here `query_selector` returns `Result`, rather than the +`std` API's `Option`. The bounded parser also reports invalid or unsupported selector syntax as +`SelectorCapacityExceeded`; that error does not prove that increasing capacity +will make the query valid. + +Insufficient storage returns `ParseError`: node, stack, root, ID, class and +selector capacity errors identify those budgets. Without `std`, each tag can store 8 ordinary attributes (ID and class +are stored separately) and 256 direct children. These fixed bounds are not +changed by the six `parse` capacities. `InvalidLength` rejects input too large for the parser's `u32` spans. +These errors are not HTML conformance validation. For no-allocation output, +use `write_outer_html` with a caller-provided `core::fmt::Write` sink. + +## Updating consumers + +The current source adds public `Selector` variants, changes bounded parsing +APIs and makes `InlineVec::inline_parts_mut` and +`InlineHashMap::inline_parts_mut` unsafe. Exhaustive enum matches and callers of +raw mutable storage need review. Prefer safe collection methods such as +`get_mut`, `as_mut_slice` (vector) and `insert` (map). Do not add an unsafe block +without meeting the documented initialized-prefix and ownership invariants. +The crate version has not yet been bumped; review the Git revision you consume. diff --git a/src/lib.rs b/src/lib.rs index bef1cd1..5f8f584 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -1,4 +1,5 @@ #![doc = include_str!("../README.md")] +#![doc = include_str!("../docs/usage.md")] #![deny(missing_docs)] #![cfg_attr(not(feature = "std"), no_std)] #![cfg_attr(feature = "portable-simd", feature(portable_simd))] @@ -44,8 +45,8 @@ const STD_INLINE_CLASSES: usize = 16; /// /// This is the "entry point" and function that is called to parse HTML. /// The input string must be kept alive, and must outlive `VDom`. -/// If you need an "owned" version that takes an input string and can be kept around forever, -/// consider using `parse_owned()`. +/// For an owned input string, use [`VDomGuard::parse`]. References obtained from +/// that guard remain bounded by its lifetime. /// /// # Errors /// Throughout the parser it is assumed that spans never overflow a `u32`. @@ -136,7 +137,9 @@ pub fn parse_query_selector( /// This uses `unsafe` code to create a self-referential-like struct. /// The given input string is first leaked and turned into raw pointer, and its lifetime will be promoted to 'static. /// Once `VDomGuard` goes out of scope, the string will be freed. -/// It should not be possible to cause UB in its current form and might become a safe function in the future. +/// This legacy signature is retained for compatibility. New code should use +/// the safe [`VDomGuard::parse`] constructor, which enforces the same ownership +/// and lifetime invariants. #[cfg(feature = "std")] pub unsafe fn parse_owned(input: String, options: ParserOptions) -> Result { VDomGuard::parse(input, options)