docs: add verified usage and development guides

.github/workflows/rust.yml

@@ -11,6 +11,7 @@ on:
       - 'Cargo.toml'
       - 'Cargo.lock'
       - 'README.md'
+      - 'docs/**'
       - '.cargo/**'
       - 'rust-toolchain*'
       - '.github/workflows/rust.yml'
@@ -25,6 +26,7 @@ on:
       - 'Cargo.toml'
       - 'Cargo.lock'
       - 'README.md'
+      - 'docs/**'
       - '.cargo/**'
       - 'rust-toolchain*'
       - '.github/workflows/rust.yml'
@@ -66,6 +68,8 @@ jobs:
           cargo test --no-default-features
           cargo test --features std
           cargo test --features entities
+          RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --no-default-features
+          RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --features entities
           cargo clippy --all-targets --no-default-features -- -D warnings
           cargo clippy --all-targets --features entities -- -D warnings
       - name: Benchmark smoke checks (no performance claims)

README.md

@@ -1,51 +1,73 @@
-# `rustedbytes-tl`
+# rustedbytes-tl
 
-[![Crates.io Version](https://img.shields.io/crates/v/rustedbytes-tl)](https://crates.io/crates/rustedbytes-tl)
+[![Crates.io](https://img.shields.io/crates/v/rustedbytes-tl.svg)](https://crates.io/crates/rustedbytes-tl)
 
-tl is a fast HTML parser written in pure Rust.
+A lightweight HTML parser written in Rust. It borrows input bytes and offers
+allocation-free bounded parsing by default, or heap-backed DOMs and CSS queries
+with `std`. The Rust crate is imported as **`tl`**.
 
-By default this crate builds without `std` or `alloc`. Enable `std` for the
-allocating convenience API. The examples below target the current Git revision
-(next breaking 0.3.x release), including APIs not present in published 0.2.0:
+This README describes the current Git source. The manifest still says `0.2.0`,
+but this branch contains breaking changes intended for the next `0.3.x` release.
+The published [0.2.0 API reference](https://docs.rs/rustedbytes-tl/0.2.0/tl/)
+does not describe all APIs below. Generate current API docs with
+`cargo doc --features entities --open`.
 
-```toml
-rustedbytes-tl = { git = "https://github.com/RustedBytes/rustedbytes-tl", features = ["std"] }
-```
+## Getting started
 
-For the nightly portable SIMD path, enable `portable-simd` and build with
-nightly:
+Use the current source with heap-backed parsing:
 
-```sh
-cargo +nightly build --features portable-simd
+```toml
+[dependencies]
+rustedbytes-tl = { git = "https://github.com/RustedBytes/rustedbytes-tl", features = ["std"] }
 ```
 
-## Queries and decoded text
-
-Enable `std` for structural CSS queries and safe owned DOMs:
+Commit your application's Cargo.lock to retain the resolved Git revision.
 
 ```rust
 # #[cfg(feature = "std")] {
-let guard = tl::VDomGuard::parse(
-    "<ul><li><a href='/item'>Item</a></li></ul>".into(),
-    tl::ParserOptions::default(),
-).unwrap();
-assert_eq!(guard.get_ref().query_selector("ul > li a[href]").unwrap().count(), 1);
+let html = "<ul><li><a href='/item'>Item</a></li></ul>";
+let dom = tl::parse(html, tl::ParserOptions::default()).unwrap();
+let handle = dom.query_selector("ul > li a[href]").unwrap().next().unwrap();
+let node = handle.get(dom.parser()).unwrap();
+assert_eq!(node.inner_text(dom.parser()), "Item");
+let tag = node.as_tag().unwrap();
+let href = tag.attributes().get("href").unwrap().unwrap();
+assert_eq!(href.as_bytes(), b"/item");
 # }
 ```
 
-Enable `entities` to use `decoded_inner_text` for HTML character references.
-`inner_text` continues to return the original undecoded text.
-See [compatibility notes](docs/silkworm-compatibility.md) for supported selectors,
-remaining HTML5 differences, query limits and release compatibility.
+Keep `html` alive for every use of the borrowed DOM. If the DOM must own its
+input, use the safe `tl::VDomGuard::parse(String, ParserOptions)` constructor.
+See the [usage guide](https://github.com/RustedBytes/rustedbytes-tl/blob/master/docs/usage.md) for owned input, attributes, decoded text,
+query errors and bounded parsing.
+
+## Features
+
+| Feature | Behavior |
+| --- | --- |
+| Default (none) | `no_std`, no allocation; explicit capacities in `parse`; bounded simple selectors |
+| `std` | Heap-backed storage, owned DOMs, text extraction, structural CSS selectors |
+| `entities` | Implies `std`; adds character-reference decoding through `decoded_inner_text` |
+| `portable-simd` | Requires nightly Rust; optional SIMD parsing path, combinable with the above |
+
+`__INTERNALS_DO_NOT_USE` exposes instrumentation internals and is not a supported
+application feature. Stable builds should select features explicitly rather
+than use `--all-features`, which enables nightly-only `portable-simd`.
 
-For checks, benchmarks, safety changes and migration guidance, see the
-[Rust audit notes](docs/rust-audit.md).
+## Scope and documentation
 
-## Provenance
+The parser does not implement the complete HTML5 tree-building algorithm.
+Malformed HTML can produce a different tree from a browser; omitted `tbody`
+elements are not inserted. CSS support is a subset, and text extraction does
+not implement browser layout or visibility rules. HTML parsing and text
+extraction do not sanitize untrusted content.
 
-This crate is a fork of [`astral-tl`](https://github.com/astral-sh/astral-tl), modified to
-add no-std, zero-copy parsing and other improvements.
+- [Usage guide](https://github.com/RustedBytes/rustedbytes-tl/blob/master/docs/usage.md): executable examples and API behavior.
+- [Development guide](https://github.com/RustedBytes/rustedbytes-tl/blob/master/docs/development.md): feature checks, docs, benchmarks and troubleshooting.
+- [Migration and silkworm compatibility](https://github.com/RustedBytes/rustedbytes-tl/blob/master/docs/silkworm-compatibility.md): supported selectors and remaining compatibility work.
+- [Rust audit](https://github.com/RustedBytes/rustedbytes-tl/blob/master/docs/rust-audit.md): safety fixes and measured performance, with methodology and limits.
 
-## License
+## Provenance and license
 
-MIT.
+Forked from [astral-tl](https://github.com/y21/astral-tl) and extended for bounded
+`no_std` parsing and RustedBytes consumers. Licensed under MIT; see [LICENSE](https://github.com/RustedBytes/rustedbytes-tl/blob/master/LICENSE).

docs/development.md

@@ -0,0 +1,86 @@
+# Development and verification
+
+Run commands from the repository root. `std` changes public signatures and
+storage behavior, so checking only one feature configuration is insufficient.
+CI pins Rust 1.99.0 for stable checks and checks compilation on 1.92.0.
+The manifest does not declare an MSRV; the compatibility job is not an MSRV promise.
+
+## Stable checks
+
+```sh
+cargo fmt --check
+cargo test --no-default-features
+cargo test --features std
+cargo test --features entities
+cargo clippy --all-targets --no-default-features -- -D warnings
+cargo clippy --all-targets --features entities -- -D warnings
+```
+
+README examples are included in crate rustdoc. The usage guide is also included in crate rustdoc; Cargo runs its doctests
+with the selected features.
+To check documentation alone:
+
+```sh
+cargo test --doc --no-default-features
+cargo test --doc --features std
+cargo test --doc --features entities
+RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --no-default-features
+RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --features entities
+```
+
+Examples hide feature guards in rendered documentation so the same source can
+be tested for all three configurations. Keep each example's feature requirement
+clear in its surrounding text. Verify local Markdown links when moving pages.
+
+## Nightly and safety checks
+
+Stable `--all-features` fails because `portable-simd` requires nightly. The CI
+nightly is pinned separately:
+
+```sh
+cargo +nightly-2026-09-25 test --no-default-features --features portable-simd
+cargo +nightly-2026-09-25 test --features entities,portable-simd
+```
+
+Install Miri for that nightly before focused safety checks:
+
+```sh
+rustup component add miri --toolchain nightly-2026-09-25
+cargo +nightly-2026-09-25 miri test --lib --features std tests::audit
+cargo +nightly-2026-09-25 miri test --lib --features std owned
+cargo +nightly-2026-09-25 miri test --lib --features std inline
+```
+
+The workflow's manual `miri` input enables these focused groups. They exercise
+known unsafe boundaries, not every possible program execution.
+
+## Benchmarks
+
+```sh
+cargo bench --bench tl -- --test
+cargo bench --features std --bench tl -- --test
+cargo bench --features entities --bench audit -- --test
+cargo bench --features entities --bench audit
+cargo run --release --features entities --example audit_allocations
+```
+
+`--test` is a smoke check, not a timing measurement. `tl` benchmarks parsing;
+`audit` benchmarks queries and text on a prebuilt DOM. The allocation example
+measures requested allocation activity separately. See the
+[audit report](rust-audit.md) for fixtures, baselines, confidence intervals and
+limits. Reproduce baseline and candidate with identical toolchains, dependency
+resolution, features and flags; avoid concurrent builds during timing runs.
+
+## Troubleshooting
+
+| Symptom | Check |
+| --- | --- |
+| Examples fail to find `rustedbytes_tl` | Import `tl`; the package and library names differ |
+| `parse` requires const arguments | Enable `std`, or provide all six bounded capacities |
+| `decoded_inner_text` is missing | Enable `entities`, which also enables `std` |
+| Query is rejected | Check the supported CSS subset; rejection differs from an empty iterator |
+| Tree differs from a browser | Review HTML5 recovery limits in the compatibility notes |
+| Capacity error | Increase the relevant bounded budget or use heap-backed `std` parsing |
+
+Docs-only paths trigger CI too. This workflow checks code and docs; it does not
+publish a crate, bump the version or create a release.

docs/usage.md

@@ -0,0 +1,135 @@
+# Usage guide
+
+These examples target the current Git source. Import the package
+`rustedbytes-tl` as `tl`. Enable `std` for heap-backed parsing, and `entities`
+for decoded text; the latter already enables `std`.
+
+## Borrowed and owned input
+
+`tl::parse(&str, ParserOptions)` borrows the input. Node handles must be resolved
+against the parser that produced them; a handle is not an independently owned
+node or a globally unique identifier. Do not use handles from another DOM.
+
+For input obtained as a `String`, the guard keeps that input alive:
+
+```rust
+# #[cfg(feature = "std")] {
+let guard = tl::VDomGuard::parse(
+    String::from("<p id='message'>Hello</p>"),
+    tl::ParserOptions::default(),
+).unwrap();
+let dom = guard.get_ref();
+let handle = dom.get_element_by_id("message").unwrap();
+assert_eq!(handle.get(dom.parser()).unwrap().inner_text(dom.parser()), "Hello");
+# }
+```
+
+DOM references and cloned nodes obtained from the guard cannot outlive it.
+`get_mut_ref` also returns a shared DOM reference; its name does not grant DOM
+mutation. The legacy `unsafe parse_owned` entry point remains available, but
+new code should use `VDomGuard::parse`.
+
+`ParserOptions::default()` does not build ID/class indexes. With `std`,
+`get_element_by_id` and `get_elements_by_class_name` still work by scanning.
+Use `ParserOptions::default().track_ids().track_classes()` when repeated
+lookups justify building indexes. CSS query support does not require these flags.
+
+## Queries and attributes
+
+With `std`, `query_selector` returns `None` for invalid or unsupported syntax.
+A valid query returns an iterator, which can be empty:
+
+```rust
+# #[cfg(feature = "std")] {
+let dom = tl::parse("<button disabled title='Save'>Save</button>", Default::default()).unwrap();
+assert!(dom.query_selector("button:unsupported").is_none());
+assert_eq!(dom.query_selector("article").unwrap().count(), 0);
+let button = dom.query_selector("button[disabled]").unwrap().next().unwrap();
+let attributes = button.get(dom.parser()).unwrap().as_tag().unwrap().attributes();
+assert!(attributes.get("missing").is_none());
+assert!(matches!(attributes.get("disabled"), Some(None)));
+assert_eq!(attributes.get("title").unwrap().unwrap().as_bytes(), b"Save");
+# }
+```
+
+The nested attribute result distinguishes missing attributes (`None`), present
+attributes without a value (`Some(None)`) and values (`Some(Some(bytes))`).
+Attributes retain raw character references. `Bytes::as_bytes()` gives bytes;
+`try_as_utf8_str()` validates UTF-8 and `as_utf8_str()` (with `std`) converts
+lossily. Initial spans borrow input; explicitly replacing bytes can own data.
+
+Supported `std` queries include type, ID, class, universal, attribute presence
+and value matching (`=`, `~=`, `^=`, `$=`, `*=`), compound selectors, comma lists,
+descendant, child and sibling combinators, `:first-child`, `:nth-child(an+b)`,
+`:not(...)`, and descendant-relative `:has(...)`. Element positions ignore
+text and comments. Type names match ASCII-insensitively. CSS escapes,
+namespaces, attribute case flags, other pseudo-classes and `:has(> a)` are
+unsupported. Recursive nesting and compound size are bounded; see the
+[compatibility notes](https://github.com/RustedBytes/rustedbytes-tl/blob/master/docs/silkworm-compatibility.md).
+
+Structural queries build indexes proportional to DOM node count. `:has` can
+scan many candidates; it is not guaranteed to take linear time. Calling
+`Selector::matches` directly lacks the structural context supplied by the DOM
+query iterator.
+
+## Raw and decoded text
+
+`inner_text` concatenates descendant text, ignores comments and leaves entities
+encoded. It does not insert layout spaces or filter hidden elements. With
+`entities`, opt into decoding:
+
+```rust
+# #[cfg(feature = "entities")] {
+let dom = tl::parse("<p>A &amp; B &#x1F980;</p>", Default::default()).unwrap();
+let handle = dom.query_selector("p").unwrap().next().unwrap();
+let node = handle.get(dom.parser()).unwrap();
+assert_eq!(node.inner_text(dom.parser()), "A &amp; B &#x1F980;");
+assert_eq!(node.decoded_inner_text(dom.parser()), "A & B 🦀");
+# }
+```
+
+Decoding is per text node: markup boundaries cannot form a new entity.
+Script/style tags preserve raw text; directly decoding a raw child uses ordinary
+text-context rules. Decoding does not change source bytes or attributes.
+Returned text is `Cow<str>` and may borrow or allocate depending on the content
+and tree. Do not assume every extraction is allocation-free.
+
+## Bounded parsing without std
+
+Without `std`, `parse` takes six capacities in this order: nodes, parser stack,
+root nodes, tracked IDs, tracked classes and selector nodes. Include text and
+comment nodes in the node budget. Capacities also affect stack/storage size;
+choose them for your input rather than copying large arbitrary bounds.
+
+```rust
+# #[cfg(not(feature = "std"))] {
+let dom = tl::parse::<32, 16, 8, 8, 8, 16>(
+    "<p id='message'>Hello</p>", tl::ParserOptions::default(),
+).unwrap();
+assert!(dom.get_element_by_id("message").is_some());
+assert_eq!(dom.query_selector("p").unwrap().count(), 1);
+# }
+```
+
+Bounded selectors support simple matching, not the structural CSS engine
+available with `std`. Here `query_selector` returns `Result`, rather than the
+`std` API's `Option`. The bounded parser also reports invalid or unsupported selector syntax as
+`SelectorCapacityExceeded`; that error does not prove that increasing capacity
+will make the query valid.
+
+Insufficient storage returns `ParseError`: node, stack, root, ID, class and
+selector capacity errors identify those budgets. Without `std`, each tag can store 8 ordinary attributes (ID and class
+are stored separately) and 256 direct children. These fixed bounds are not
+changed by the six `parse` capacities. `InvalidLength` rejects input too large for the parser's `u32` spans.
+These errors are not HTML conformance validation. For no-allocation output,
+use `write_outer_html` with a caller-provided `core::fmt::Write` sink.
+
+## Updating consumers
+
+The current source adds public `Selector` variants, changes bounded parsing
+APIs and makes `InlineVec::inline_parts_mut` and
+`InlineHashMap::inline_parts_mut` unsafe. Exhaustive enum matches and callers of
+raw mutable storage need review. Prefer safe collection methods such as
+`get_mut`, `as_mut_slice` (vector) and `insert` (map). Do not add an unsafe block
+without meeting the documented initialized-prefix and ownership invariants.
+The crate version has not yet been bumped; review the Git revision you consume.

src/lib.rs

@@ -1,4 +1,5 @@
 #![doc = include_str!("../README.md")]
+#![doc = include_str!("../docs/usage.md")]
 #![deny(missing_docs)]
 #![cfg_attr(not(feature = "std"), no_std)]
 #![cfg_attr(feature = "portable-simd", feature(portable_simd))]
@@ -44,8 +45,8 @@ const STD_INLINE_CLASSES: usize = 16;
 ///
 /// This is the "entry point" and function that is called to parse HTML.
 /// The input string must be kept alive, and must outlive `VDom`.
-/// If you need an "owned" version that takes an input string and can be kept around forever,
-/// consider using `parse_owned()`.
+/// For an owned input string, use [`VDomGuard::parse`]. References obtained from
+/// that guard remain bounded by its lifetime.
 ///
 /// # Errors
 /// Throughout the parser it is assumed that spans never overflow a `u32`.
@@ -136,7 +137,9 @@ pub fn parse_query_selector<const MAX_SELECTOR_NODES: usize>(
 /// This uses `unsafe` code to create a self-referential-like struct.
 /// The given input string is first leaked and turned into raw pointer, and its lifetime will be promoted to 'static.
 /// Once `VDomGuard` goes out of scope, the string will be freed.
-/// It should not be possible to cause UB in its current form and might become a safe function in the future.
+/// This legacy signature is retained for compatibility. New code should use
+/// the safe [`VDomGuard::parse`] constructor, which enforces the same ownership
+/// and lifetime invariants.
 #[cfg(feature = "std")]
 pub unsafe fn parse_owned(input: String, options: ParserOptions) -> Result<VDomGuard, ParseError> {
     VDomGuard::parse(input, options)