From 2518aa0b820f26f0994cfcf6c49be6683706942c Mon Sep 17 00:00:00 2001 From: Abhishek Mohan <12630584+Abhishek-Mohan@users.noreply.github.com> Date: Fri, 23 Oct 2026 18:00:43 -0700 Subject: [PATCH 1/2] docs: add a documentation style guide Add docs/source/contributor-guide/style_guide.md, which writes down the vocabulary from #4419 (pipeline membership, implementation language and data format), the rule for "native", how to name operators in prose, and the page conventions that the docs-review skill in #4420 cites. Each rule has a stable ID so that reviews can cite it. The guide follows the code where #4419 and the code differ: "Hybrid" keeps the meaning it already has in expressions.md, expressions are described separately from plan nodes, and "Arrow-native" is a claim about data format only. The Generated Content section names the generated text exactly, by the markers GenerateDocs fills in, including the config tables built from QueryPlanSerde and contrib config classes, and the cast tables built from CometCast. P18 covers attribution of third-party content, which #4420 lists among its checks. The guide's scope covers .doc strings in contrib config classes, and its vocabulary and accuracy rules also apply to doc comments, AGENTS.md and agent skills. Link the page from the contributor guide toctree and from AGENTS.md. Part of #4419. Generated-by: OpenCode --- AGENTS.md | 3 + docs/source/contributor-guide/index.md | 1 + docs/source/contributor-guide/style_guide.md | 215 +++++++++++++++++++ 3 files changed, 219 insertions(+) create mode 100644 docs/source/contributor-guide/style_guide.md diff --git a/AGENTS.md b/AGENTS.md index b7a732268dc..57d10fbc0a9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,6 +30,9 @@ Relevant entry points: - [Adding a New Expression](docs/source/contributor-guide/adding_a_new_expression.md) / [Adding a New Operator](docs/source/contributor-guide/adding_a_new_operator.md). - [Debugging Guide](docs/source/contributor-guide/debugging.md). +- [Documentation Style Guide](docs/source/contributor-guide/style_guide.md): the terms to use for + how Comet runs a query, and the conventions for docs pages. Read it before editing `docs/source`, + `README.md`, or the `.doc(...)` strings in config classes. - [Backporting to Release Branches](docs/source/contributor-guide/backporting.md): a fix that goes to one release branch also goes to every newer one. Cherry-pick with `-x`, and never add an unrelated change to a backport pull request. diff --git a/docs/source/contributor-guide/index.md b/docs/source/contributor-guide/index.md index a773c1b2ed7..12d2b9dbd16 100644 --- a/docs/source/contributor-guide/index.md +++ b/docs/source/contributor-guide/index.md @@ -98,6 +98,7 @@ Expression Audits Data Source Audits Supported Spark Configurations Configuration Conventions +Documentation Style Guide ``` ```{toctree} diff --git a/docs/source/contributor-guide/style_guide.md b/docs/source/contributor-guide/style_guide.md new file mode 100644 index 00000000000..c5848f3f214 --- /dev/null +++ b/docs/source/contributor-guide/style_guide.md @@ -0,0 +1,215 @@ + + +# Documentation Style Guide + +This guide is for anyone who writes or reviews Comet documentation: pages under `docs/source`, +`README.md`, and the user-facing `.doc(...)` strings in `CometConf.scala` and in contrib config +classes. It defines the words Comet's docs use for how a query runs, and the conventions every page +follows. The vocabulary rules (`T`, `N`, `O`) and the accuracy rules (`P13`, `P16`) also apply to doc +comments, `AGENTS.md`, and the agent skills under `.ai/skills`, with the contributor guide's +allowances. + +The sections cover how Comet runs a query, the vocabulary, the rule for "native", operator names, +page conventions, and generated content. + +Each rule has an ID, such as `N1`. Reviews cite the ID, so IDs are never renumbered or reused. +Where this guide and the code disagree, the code is right and the guide needs a fix. The +background for the vocabulary is in [#4419](https://github.com/apache/datafusion-comet/issues/4419). + +## How Comet Runs a Query + +The terms below describe this model. [Comet Plugin Architecture](plugin_overview.md) has the details. + +- Comet's planner rules replace the Spark operators Comet supports with Comet operators, most of + them named `Comet*`, and serialize their expressions to protobuf. An operator Comet does not + support stays a Spark operator. +- Most Comet operators run in Comet's Rust code, which is built on Apache DataFusion and is called + from the JVM through JNI. Some do their own work on the JVM, such as broadcast exchanges. +- Comet's query operators usually pass data to each other as Apache Arrow batches. Some expressions + run Spark's own generated JVM code over those batches. + +## Vocabulary + +Use the terms in the first column. They answer three separate questions: does Comet handle the +work, which language runs it, and what format is the data in. A term from one question says nothing +about the other two. + +| ID | Use | For | Instead of | +| --- | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- | +| T1 | _Comet pipeline_, _runs in Comet_ | Work that a Comet operator handles, whatever the language | "the native path", "runs natively" | +| T2 | _falls back to Spark_, _Spark fallback_ | Work that stays in a Spark operator | — | +| T3 | _Rust-implemented_, _runs in Rust_, _native Rust_ | Work done by Comet's Rust code, usually through DataFusion | "native" on its own | +| T4 | _JVM-side_, _runs on the JVM_ | A Comet operator whose own work runs on the JVM, such as `CometBroadcastExchange` | "non-native Comet operator" | +| T5 | _codegen dispatch_ | Spark's generated JVM code for an expression, run over Arrow batches from inside a Rust-implemented plan | "falls back" (it does not fall back) | +| T6 | _Native_, _Codegen dispatch_, _Hybrid_ | The expression kinds in the [Implementation legend](../user-guide/latest/expressions.md#implementation-legend) | Any other meaning of "Hybrid" | +| T7 | _Arrow-native_, _Arrow batches_, _Arrow columnar_ | Claims about data format: the data is held in Arrow columnar format | "vectorized" as a synonym for columnar (T10) | +| T8 | _Arrow IPC_ | The format shuffle and broadcast write | — | +| T9 | _Spark rows_ (`UnsafeRow`), _Spark columnar_ | Spark's row format, or a Spark `ColumnarBatch` that is not backed by Arrow | "row format", "JVM format" without saying which | +| T10 | _vectorized_ | Proper names, such as Spark's vectorized Parquet reader, and batch-at-a-time kernels contrasted with row-by-row loops | "vectorized" as a synonym for columnar | + +Notes on these terms: + +- **Hybrid (T6)** describes an expression that has both a Rust path and a Spark-compatible path, + usually codegen dispatch. Each use of the expression in a plan takes one path. The term comes + from `GenerateDocs`, which writes the Implementation column of `expressions.md`. Do not use + "hybrid" for a plan that mixes Comet and Spark operators. That is a plan with fallback (T2). +- **Arrow-native (T7)** is about format. It does not mean that no per-row work happens: codegen + dispatch evaluates each row, and JVM shuffle buffers `UnsafeRow`s before encoding them to Arrow + IPC. +- **Spark rows (T9)** can appear inside the Comet pipeline. JVM shuffle, for example, buffers + `UnsafeRow`s. The format alone does not say whether Spark or Comet handles the work. +- **Expressions are not plan nodes.** An expression runs inside an operator such as `CometProject` + or `CometFilter`. A codegen-dispatched expression leaves its operator Rust-implemented. + +## The Rule for "Native" + +"Native" has several meanings in Comet's docs, so on its own it does not tell the reader which one +is meant. Name the meaning instead. + +- **N1.** Do not use _runs natively_, _natively in Comet_, _the native path_, _the native + pipeline_, _native execution_, _native operator_, or _native Comet operator_. Write _in the Comet + pipeline_ (T1) or _Rust-implemented_ (T3). +- **N2.** These compounds are allowed everywhere, because the second part fixes the meaning: + _native Rust_; _native shuffle_ (paired with _JVM shuffle_); "native", optionally followed by a + format, followed by _scan_, _reader_, _write_, or _writer_, for Comet's Rust scans and writers + (_native scan_, _native Parquet reader_, _native Iceberg write_); _native memory_ and _native + library_ (the terms Comet's own log lines use); _Native_ as the expression kind (T6); and + _native implementation_ of an expression, meaning its Rust implementation. +- **N3.** In the contributor guide, "native" before a code noun is also allowed when it means the + crates under `native/` or the JNI boundary: _native code_, _native side_, _native plan_, _native + kernel_, _JVM/native boundary_, and similar. Pages outside the contributor guide, including + `about/`, `index.md`, `README.md`, and `.doc(...)` strings, write _Rust_ instead. N1 applies + everywhere. +- **N4.** Do not use "native" as a noun. Write _the Rust side_ or name the component: "hands each + block to the Rust side", not "hands each block to native". +- **N5.** Do not call Spark's own behavior native. Write _Spark's built-in shuffle_, not _Spark's + native shuffle_. +- **N6.** Quote identifiers, config keys and values, API names, log lines, and plan output exactly + as they are, in code font. `CometNativeScan`, `CometNativeExec`, and + `spark.comet.shuffle.mode=native` are not prose and do not break N1. Text that quotes a phrase in + order to discuss it, such as the "Before" column below, does not break these rules either. + +Examples: + +| Before | After | +| ---------------------------------------------------------- | ------------------------------------------------------------- | +| Runs your Spark queries on the DataFusion native engine. | Speeds up your existing Spark queries with Apache DataFusion. | +| The surrounding native operators keep running natively. | The surrounding operators stay in the Comet pipeline. | +| Keeps evaluation inside the native pipeline. | Keeps evaluation inside the Comet pipeline. | +| Takes columnar input directly from Comet native operators. | Takes Arrow batches directly from Comet operators. | + +## Operators, Plan Names, and Expressions + +[Understanding Comet Plans](../user-guide/latest/understanding-comet-plans.md) describes the nodes +a plan can contain, grouped by role. Its +[Comet Operator Reference](../user-guide/latest/understanding-comet-plans.md#comet-operator-reference) +lists them. Link to it rather than copying it. + +Do not infer an operator's kind from its class hierarchy. For example, +`CometTakeOrderedAndProjectExec` runs Rust plans without extending `CometNativeExec`. Most Comet +operators print a `Comet` prefix, but not all: `IcebergWrite` and `IcebergCommit` are Comet +operators too. + +- **O1.** In prose about plans, use the name that plan output prints. Spark drops the `Exec` suffix + from most names (`CometProjectExec` prints as `CometProject`), but some operators keep it, such as + `CometWindowExec`, and `CometShuffleExchangeExec` prints as `CometExchange` or + `CometColumnarExchange`. This is the `nodeName`, which formatted explain output and the Spark UI + print. Some classes also override `simpleString`, which changes the line `df.explain()` prints: + `CometTakeOrderedAndProjectExec` prints as `CometTakeOrderedAndProjectExec(...)` there. Quote plan + output exactly as it was printed (`N6`), and check the operator reference when unsure. +- **O2.** Use the class name, with `Exec`, only when the text is about the code, including text + about how a planner rule rewrites a plan. +- **O3.** The current names are the correct names. Do not describe renames that have not happened. +- **O4.** Expressions are not operators. Describe an expression by its kind (T6), not as a node in + the plan. + +## Page Conventions + +- **P1.** Every new Markdown file starts with the ASF license header, as in this file. +- **P2.** A page has one H1, and heading levels are not skipped. +- **P3.** The first paragraph says who the page is for and what it covers. A long page then gives + a short overview of its sections. +- **P4.** On pages under `docs/source`, link to another page by its path, either relative to the + current page or starting with `/` for the `docs/source` root, not by the published docs URL, + because the site's address can change. `README.md` is rendered by GitHub, so it links to the + published docs. +- **P5.** Do not hand-edit generated content. Change the source it comes from instead. See + [Generated Content](#generated-content). +- **P6.** Every new page is added to a toctree. Pages for users go under `docs/source/user-guide/` + or `docs/source/about/`, and pages for contributors go under `docs/source/contributor-guide`. +- **P7.** The user guide describes what Comet does today, in the present tense. Plans go in the + [roadmap](roadmap.md). Do not leave `TODO` or `XXX` markers in published pages. +- **P8.** A claim that depends on a version, such as "Spark 3.5 and later", "Spark 3.5+", or "Spark + 4.0 only", matches the versions the code supports. The same applies to other dependencies, such + as Iceberg. +- **P9.** Prefer the active voice, and address the reader as "you". +- **P10.** Do not use marketing words such as "blazing fast", "seamless", or "powerful". State the + measured result, or leave the claim out. +- **P11.** Use a table to compare several items, such as operators, configs, or support levels. +- **P12.** Plan output in a page is real output from `explain`, in Spark's tree format with `+-` + connectors, and is shown in a code block. +- **P13.** Config keys, class names, and file paths named in a page exist in the code. Code + examples are valid in the language their block declares. Follow + [Configuration Conventions](config_conventions.md) for new configs. +- **P14.** A new user guide page includes at least one example a reader can run and links to + related pages. A new contributor guide page that describes a procedure says how to check that it + worked. +- **P15.** A change that adds an operator updates the + [Comet Operator Reference](../user-guide/latest/understanding-comet-plans.md#comet-operator-reference) + and [Supported Spark Operators](../user-guide/latest/operators.md) in the same pull request. A + change that adds an expression updates its row in + [Spark Expression Support](../user-guide/latest/expressions.md). +- **P16.** Describe the code as it is now. Remove text about code that has been removed, keep + sample output, such as explain output and messages, in step with the code that produces it, and + keep each page consistent with itself. +- **P17.** A change that moves, renames, or removes a page adds an entry to `redirects` in + `docs/source/conf.py`, so that existing links keep working. +- **P18.** Text, diagrams, and images taken from another project or website say where they came + from, and their license is compatible with the Apache License. See the ASF's + [third-party licensing policy](https://www.apache.org/legal/resolved.html). + +### Generated Content + +These parts of the docs are written by tools. Edit the source they come from instead. The rest of +each file, including the prose around the markers and the `index.md` pages, is written by hand. + +- **Config tables**, the text between `` and + ``. `GenerateDocs` builds them from the config entries in `CometConf.scala` + and from the contrib config classes registered with `CometConfigProvider`. The `enable_expr` and + `enable_agg_expr` tables come from the expression serde maps in `QueryPlanSerde` instead. +- **Expression compatibility sections**, the text between `` and + `` in `compatibility/expressions/_category_template/`. `GenerateDocs` builds + them from the expression serdes. +- **Cast tables**, the text between the `CAST_*_TABLE` markers. `GenerateDocs` builds them from + `CometCast.supportedTypes`, `CometCast.isSupported`, and Spark's `Cast.canCast`. +- **Per-version expression pages** built from `_category_template/`. On `main`, the docs build + creates them in a temporary tree. Release branches commit a generated copy, made with + `dev/generate-release-docs.sh`. +- **The Implementation column** in `expressions.md`, built by `GenerateDocs` from the expression + serdes in `QueryPlanSerde`. +- **`changelog/`**, written by the release process. + +The expression audits, data source audits, and benchmark results under `contributor-guide/` are +records of past work, not pages to restyle. + +## Changing This Guide + +Propose changes in a pull request. Add new rules with new IDs, and do not renumber or reuse an ID, +even when a rule is removed. From f7644b8f04f75bc207d8bf8056b129e200e56e53 Mon Sep 17 00:00:00 2001 From: Abhishek Mohan <12630584+Abhishek-Mohan@users.noreply.github.com> Date: Fri, 23 Oct 2026 00:33:06 -0700 Subject: [PATCH 2/2] docs: add a docs-review skill Add .ai/skills/review-comet-docs/SKILL.md, a sibling of review-comet-pr for documentation. It has three modes: PR mode reviews the docs a pull request changes, branch mode reviews a branch before it becomes a PR, and audit mode reviews an existing file or directory, including files not yet committed. The skill reads the documentation style guide on every run and cites its rule IDs rather than restating the rules. It finds the Apache remote and the PR's real base, reads everything from the reviewed revision, checks whether main has moved since, and checks only the added lines in PR and branch mode. Mechanical checks come first: vague "native" (including in Markdown italics), config keys (including keys built from a name at run time), operator names, links, anchors, images, external links, structure, redirects for moved pages, and version claims. Then it runs judgment checks, the most important being whether each claim, and each condition in it, still matches the code. Besides docs pages, it checks changed .doc strings, doc comments, AGENTS.md and agent skills. Findings are must-fix, should-fix, or nit, with nits dropped when the skill runs as part of a review-comet-pr review. It never edits files or posts comments. A short table of fixed cases says how to check the skill after changing it. In review-comet-pr, route docs changes to the new skill, mention documentation in its description, and remove "docs only" from the list of PRs it covers on its own. Fix two things in it that no longer match: `gh pr checks --failed` no longer exists, and the opt-in key is spark.comet.expression..allowIncompatible. In review-comet-pr and review-comet-expression-pr, point at the style guide for exactly which docs text is generated. Closes #4420. Generated-by: OpenCode --- .ai/skills/review-comet-docs/SKILL.md | 322 ++++++++++++++++++ .../review-comet-expression-pr/SKILL.md | 7 +- .ai/skills/review-comet-pr/SKILL.md | 23 +- 3 files changed, 340 insertions(+), 12 deletions(-) create mode 100644 .ai/skills/review-comet-docs/SKILL.md diff --git a/.ai/skills/review-comet-docs/SKILL.md b/.ai/skills/review-comet-docs/SKILL.md new file mode 100644 index 00000000000..79cfad10f51 --- /dev/null +++ b/.ai/skills/review-comet-docs/SKILL.md @@ -0,0 +1,322 @@ +--- +name: review-comet-docs +description: Use when reviewing the documentation in a DataFusion Comet pull request or branch, or when auditing an existing page or directory under docs/source or README.md. Checks wording against the documentation style guide and checks that config keys, operator names, links, images and generated content match the code. Advisory only, it never edits files or posts comments. Load alongside review-comet-pr when a PR changes docs. +argument-hint: +--- + + + +Review Comet docs: $ARGUMENTS + +Pick the mode from the argument: + +- **PR mode**: a number. Review the docs a pull request changes. +- **Audit mode**: a file or directory that exists in the working tree. Review it as it is now. +- **Branch mode**: a branch, tag, or commit that `git rev-parse --verify` accepts. Review what it + changes compared with `main`, for example before opening a PR. Otherwise, ask which one is meant. + +In PR mode, **REQUIRED BACKGROUND:** use `review-comet-pr` for its steps 2 to 4 (PR metadata, +existing comments, and the diff) and its CI check. This skill only covers documentation. + +This skill gives advice to a human. Never edit files, commit, push, or post comments. + +**Shell notes.** Run every command from the repository root. Each tool call may start a new shell, +so save the variables you set and load them again in later calls: + +```bash +STATE="$(git rev-parse --git-dir)/review-comet-docs.env" +printf 'R=%s\nHEAD_REF=%s\nBASE=%s\nADDED=%s\n' "$R" "$HEAD_REF" "$BASE" "$ADDED" > "$STATE" +. "$STATE" +``` + +In zsh, `$HEAD_REF:spark/...` is read as a history modifier, so always write `"${HEAD_REF}:"`. + +## 1. Select the Revisions + +Find the remote that points at `apache/datafusion-comet`, because its name differs between clones: + +```bash +R=$(git remote -v | awk '/apache\/datafusion-comet(\.git)? \(fetch\)/ {print $1; exit}') +``` + +If `R` is empty, use `https://github.com/apache/datafusion-comet.git` in its place. + +**PR mode.** Read the base branch and changed files, then fetch the base and the head without +touching the working tree. The leading `+` lets the fetch replace a ref after a force push. + +```bash +gh pr view --repo apache/datafusion-comet --json baseRefName,headRefOid,files +git fetch "$R" "+refs/heads/:refs/review/base-" "+pull//head:refs/review/pr-" +HEAD_REF=refs/review/pr-; BASE=$(git merge-base refs/review/base- "$HEAD_REF") +``` + +Check that `git rev-parse "$HEAD_REF"` equals `headRefOid`. If it does not, the PR changed while you +fetched it: run both commands again, at most twice. + +**Branch mode.** Fetch `main` the same way, then compare with it: + +```bash +git fetch "$R" "+refs/heads/main:refs/review/main" +HEAD_REF=; BASE=$(git merge-base refs/review/main "$HEAD_REF") +``` + +**Audit mode.** There is no base. List the files, including new files that are not committed yet, +and search them with `--untracked`: + +```bash +git ls-files --cached --others --exclude-standard -- +``` + +In PR and branch mode, read every file, including linked pages and the code a claim describes, from +`$HEAD_REF`. If `git rev-list --count "$BASE"..refs/review/base-` (or `..refs/review/main`) is +not zero, `main` has moved since the change branched. Check the claims against `main` too, and report +what has changed there. + +## 2. Read the Style Guide + +Read `docs/source/contributor-guide/style_guide.md` in full on every run. If the change edits the +guide, read its version with `git show "${HEAD_REF}:docs/source/contributor-guide/style_guide.md"`. +Otherwise, read it from the working tree. If the file does not exist, stop and say so. + +The guide is the rulebook, and its first section says what it covers. Cite its rule IDs (`T3`, `N1`, +`P16`, and so on), and do not restate or reinterpret its rules here. If the guide and this skill +disagree, the guide wins. + +## 3. Select the Files + +In PR and branch mode, `` below means these changed files: + +- `.md` and `.rst` files under `docs/source`, and `README.md`; +- files that change `.doc(...)` strings in `CometConf.scala` or in contrib config classes; +- files that change doc comments (Scaladoc, Javadoc, and Rust `///` comments); +- `AGENTS.md` and `.ai/skills/**/SKILL.md`. + +For doc comments, `AGENTS.md`, and skills, check the vocabulary and accuracy rules the guide names +for them, not the page conventions. Check the changed lines and the paragraphs around them, not the +whole page. Report a problem in unchanged text only when the change makes it wrong, or list it +separately as already on the page. Also read the code diff for the behavior the changed docs +describe, because docs are usually only as accurate as the code change beside them. + +In every mode: + +- Skip the content the guide's **Generated Content** section lists as generated or as records. + Review the hand-written text around the generated markers. Text between the markers that differs + from what `GenerateDocs` produces is a hand edit, and a finding (`P5`). +- Do the judgment checks in step 5 for at most 10 pages per run. List any pages you skipped so the + user can run the skill again on them. + +## 4. Mechanical Checks + +These checks produce candidates, not findings. Read each hit in context before reporting it. For the +wording checks, ignore hits inside fenced code blocks, inline code, quoted plan output or logs, and +text that quotes a phrase in order to discuss it (`N6`), such as the style guide's own examples. The +accuracy checks still apply to code font: config keys, operator names, and code examples are usually +written that way. For each check, record what it checked, the number of hits, and how many of them +are findings. + +Use `git grep -P` or `perl`, as written. The system `grep` on macOS has no `-P`, and `git grep -E` +does not support `\b`, so it silently finds nothing. Plain `\b` also misses Markdown italics such as +`_native execution_`, because `_` counts as a word character, so the patterns below use lookarounds. + +**Lines to check.** In PR and branch mode, list the added lines with their file and line number, and +run the line checks below on this list only. In audit mode, run the line checks on the files with +`git grep -n --untracked`. + +```bash +ADDED=$(mktemp "$(git rev-parse --git-dir)/review-added.XXXXXX") +git diff --no-ext-diff --src-prefix=a/ --dst-prefix=b/ -U0 "$BASE" "$HEAD_REF" -- \ + | awk '/^--- /{h=1; next} + h && /^\+\+\+ b\//{f=substr($0,7); sub(/\t$/,"",f); h=0; next} + /^@@/{h=0; split($3,a,/[+,]/); n=a[2]; next} + /^\+/{print f":"n": "substr($0,2); n++}' > "$ADDED" +``` + +**"Native" and "vectorized".** Check each hit against `N1`-`N6` and `T10`. Which page the hit is on +matters, because `N3` allows more in the contributor guide: + +```bash +perl -ne 'print if /^.+?:\d+: .*(?:(? # audit +``` + +**Config keys.** Every `spark.comet.*` key a page names must exist (`P13`). List the keys the text +names, then the keys the code defines at the revision under review. `` is `$HEAD_REF`, or +nothing in audit mode. The second command reads definitions that span two lines, expands +`createExecEnabledConfig`, and leaves out deprecated alternative names and unexpanded templates: + +```bash +perl -ne 'print "$1\n" while /(spark\.comet\.[\w.<>]*[\w>])/g' "$ADDED" | sort -u +git grep -h -A1 -P '\b(conf|ConfigBuilder|createExecEnabledConfig)\(' -- spark/src/main contrib \ + | perl -0777 -ne ' + while (/\b(?:conf|ConfigBuilder)\(\s*s?"([^"]+)"/g) { + ($k = $1) =~ s/\$\{?COMET_EXEC_CONFIG_PREFIX\}?/spark.comet.exec/; + $k =~ s/\$\{?COMET_PREFIX\}?/spark.comet/; + print "$k\n" if $k =~ /^spark\.comet\./ && $k !~ /\$/; + } + while (/\bcreateExecEnabledConfig\(\s*"([^"]+)"/g) { print "spark.comet.exec.$1.enabled\n" }' \ + | sort -u +``` + +If the second command prints nothing, it is broken: stop and report that, rather than flagging every +key. Some keys are built from a name at run time: + +- `spark.comet.expression..enabled` and `.allowIncompatible`: `` is the Spark expression + class name, as in the `QueryPlanSerde` serde maps, not a Comet or protobuf name. Some expressions + use their own opt-in key instead, set by `nativeOptInConfigKeyOverride`, such as + `spark.comet.caseConversion.enabled`. +- `spark.comet.operator..allowIncompatible`: `` is the Spark operator class name, such as + `ProjectExec`. +- `spark.comet.datafusion.`: not a Comet config. Comet passes it to DataFusion only when + `spark.comet.exec.respectDataFusionConfigs=true`, which is off by default. Check the key against + DataFusion. + +Before reporting a missing key, search the code for it, because a few keys are defined in other +ways, such as `spark.comet.version` in `Plugins.scala`. A deprecated key that a page still uses is a +finding (`P16`). + +**Operator names.** Check operator names against `O1`, class names against `O2`, and expressions +against `O4`. List the concrete operator classes and the classes that override their printed name: + +```bash +git grep -h -o -P 'case class (Comet|Iceberg)[A-Za-z]+Exec\b' -- spark/src/main contrib | sort -u +git grep -n -P 'override (def|val|lazy val) (nodeName|simpleString)' -- spark/src/main contrib +``` + +`spark/src/main` includes the per-Spark-version source directories. If the first command prints +nothing, it is broken: stop and report that. A class's `nodeName` is its name without `Exec`, unless +it overrides `nodeName`. Compare each name in the text with these, as `O1` describes, not with a rule +about suffixes. When the page is an operator reference, also report concrete operators that it +leaves out. Leave out planning markers such as `CometScanExec`, which never appear in a final plan, +and contrib operators, which belong on their contrib page. + +**Links and images.** For each link or image in the text, inline or reference-style +(`[text]: target`): + +- A Markdown link or image path under `docs/source` is relative to the page, or to `docs/source` when + it starts with `/`. Check that the file exists at the revision under review. A raw HTML link to a + `.html` page, or a link with no extension, refers to the `.md` or `.rst` source of the same name. +- An `#anchor` must match a target on that page: an explicit label, or a heading slug. MyST makes + slugs for H1-H4 headings by lowercasing the text, turning each space into `-`, and removing + everything except letters, digits, `_`, and `-`. A repeated heading gets a numbered suffix. +- A link to Comet's published docs by URL breaks `P4` on pages under `docs/source`, but not in + `README.md`. +- For external links on the lines under review (at most 20 in audit mode), check the status with + `curl -s -o /dev/null -w '%{http_code}' -I -L --max-time 10 `. Report a 404 as a finding. If + the request fails or the network is unavailable, say the link was not checked. + +**Structure.** Check for one H1 and no skipped heading levels (`P2`), the ASF header on new files +(`P1`), and new pages missing from a toctree (`P6`). The heading command skips fenced code, where +`#` starts a shell comment. Read the file with `git show "${HEAD_REF}:"`, or from the working +tree in audit mode: + +````bash +git show "${HEAD_REF}:" | awk '/^[[:space:]]*(```|~~~)/{f=!f; next} !f && /^#{1,6} /{print FNR": "$0}' +git grep -n '' -- 'docs/source/*.md' 'docs/source/*.rst' +```` + +For the toctree check, search for the page's file name without `.md`. The pathspec matches every page, +including `docs/source/index.md`. A hit counts only inside a `{toctree}` block or a `.. toctree::` +directive, not in an ordinary link. + +**Moved and removed pages.** In PR and branch mode, list the pages the change renames or deletes. +Each one needs an entry in `redirects` in `docs/source/conf.py` (`P17`): + +```bash +git diff --name-status --diff-filter=DR "$BASE" "$HEAD_REF" -- docs/source +``` + +**Version claims and markers.** Find version claims to check against `P8`, and `TODO` or `XXX` +markers, which break `P7`: + +```bash +perl -ne 'print if /^.+?:\d+: .*(?:(?:Spark|Iceberg) [0-9]\.[0-9x]+\+?|\b(?:TODO|XXX)\b)/' "$ADDED" +git grep -n --untracked -P '(Spark|Iceberg) [0-9]\.[0-9x]+\+?|\b(TODO|XXX)\b' -- # audit +``` + +CI runs prettier and the Apache RAT license check on PRs, so do not repeat them. If CI has not run +them yet, for example on a draft, say they were not checked. Mention them otherwise only when CI has +failed on them. + +## 5. Judgment Checks + +- **Is it true?** For every sentence that names a class, rule, config, default, or file, read the + code it describes at the revision under review. A sentence can follow every wording rule and + still be wrong: a scan that no longer exists, a config default that has changed, or an operator + described as Rust-implemented that runs on the JVM. Check numbers, versions, and names first, and + check that a condition in the text is the whole condition in the code. Pages that describe + architecture or test infrastructure go stale most often. A false or outdated claim breaks `P16`. + This is the most valuable check in this skill. +- **Does it agree with the rest of the docs?** A new sentence can contradict an older one on the + same page, or a count such as "five configs" can stop matching the list below it (`P16`). When a + change rewrites the description of a config or class, search `docs/source` for the same name and + check the other descriptions. +- **Which axis?** When you suggest a replacement for vague "native", choose the term the code + supports: _in the Comet pipeline_ (`T1`), _Rust-implemented_ (`T3`), or _JVM-side_ (`T4`). Never + suggest _Rust-implemented_ without checking the code. +- **Page conventions.** Check `P3`, `P7`, `P8`, `P9`, `P10`, `P11`, `P12`, `P14`, and `P18` by + reading the page. For plan output (`P12`, `P16`), check that the operator names exist, that the + tree shape is one Spark could print, and that messages in it match the strings in the code. Do not + ask for new plan output to be invented. +- **Same-change updates.** If the change adds an operator or expression, check the pages `P15` + lists. If it adds a config, check that its `.doc(...)` string follows the guide, since + `GenerateDocs` publishes it, and that any page that documents the config by hand mentions it. + +## 6. Severity + +- **Must-fix.** The page is wrong or broken: a false or outdated claim (`P16`), a config key, class, + or file that does not exist (`P13`), a broken link, anchor, or image, a hand edit to generated + content, a new file without the ASF header, or a new page missing from a toctree. When a sentence + is both false and badly worded, report it once, as must-fix, with a rewrite that fixes both. +- **Should-fix.** The page breaks a style guide rule, but what it says is true. Most `T`, `N`, `O`, + and `P` findings belong here. +- **Nit.** A one-off change that would make the text clearer, but that no rule requires. Nits carry + no rule ID. + +A problem that keeps coming up and that no rule covers is a **guide gap**, not a finding. Report it +so the guide can be extended. + +## 7. Output + +When this skill runs as part of a `review-comet-pr` review, put its findings into that skill's +**Findings** and **Suggested Review Comments**, and drop the nits. That skill's review bar has no +third tier, and its rules win for anything posted to a PR. On its own, use this format: + +1. **Scope.** The mode, the revisions reviewed (the base and head commits, and which guide version), + the files reviewed, and the files skipped, with the reason. +2. **Mechanical checks.** A table with each check, its hit count, and how many hits are findings. +3. **Findings**, grouped as must-fix, should-fix, and nit. For each one give the `path:line`, the + rule ID, the text quoted from the page, and a suggested rewrite. Keep rewrites short and plain, + and change only what the rule requires. +4. **Guide gaps.** + +If there are no findings, say so. + +## Checking This Skill + +After changing this skill or the style guide, run it on these cases and compare the results. The +expected results were checked on `main` in October 2026, so a later fix to a page can change them. + +| Run | Expected | +| ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | +| PR mode on #6786 | Must-fix: `datasources.md` and the new `.doc` string claim more than `shouldBridgeUnsupportedScan` does | +| Audit `docs/source/user-guide/latest/understanding-comet-plans.md` | No finding for Comet's _native shuffle_, `CometNativeScan`, `CometWindowExec`, or config keys; `N5` for "Spark's native shuffle" | +| Audit `docs/source/contributor-guide/timezones.md` | No finding for _native code_ or _native plan_, or for `ToJson` and `ToCsv`, which are protobuf message names | +| Audit `docs/source/user-guide/latest/compatibility/expressions` | Lists all 16 files, reviews 10 of the templates and index pages, names the skipped ones, and skips only text between markers | +| Audit a scratch page containing `_native execution_` | `N1` finding | diff --git a/.ai/skills/review-comet-expression-pr/SKILL.md b/.ai/skills/review-comet-expression-pr/SKILL.md index c1a80773961..a5a22fe8145 100644 --- a/.ai/skills/review-comet-expression-pr/SKILL.md +++ b/.ai/skills/review-comet-expression-pr/SKILL.md @@ -225,9 +225,10 @@ and the requirement to record a performance audit. Hold the PR to both. ### Generated, review the source instead -The compatibility pages under `docs/source/user-guide/latest/compatibility/expressions/` and -`docs/source/user-guide/latest/configs.md` are produced by `GenerateDocs` and regenerated by CI on -merge. Never ask the contributor to hand-edit them. Check what feeds them: +`GenerateDocs` fills in the expression compatibility tables and the config tables when the docs are +built, and CI rebuilds and publishes the docs when a merge changes their inputs. The **Generated Content** section of +`docs/source/contributor-guide/style_guide.md` says exactly which text is generated. Never ask the +contributor to hand-edit that text. Check what feeds it: - Expression marked `Incompatible(Some("..."))` but `getIncompatibleReasons()` is empty, so the compat page shows it as supported with no caveats. diff --git a/.ai/skills/review-comet-pr/SKILL.md b/.ai/skills/review-comet-pr/SKILL.md index ea1f57a9d1f..64735411f42 100644 --- a/.ai/skills/review-comet-pr/SKILL.md +++ b/.ai/skills/review-comet-pr/SKILL.md @@ -1,6 +1,6 @@ --- name: review-comet-pr -description: Use when reviewing a DataFusion Comet pull request. Covers the workflow that applies to every PR and routes to the area-specific review skills for expressions, FFI, memory management, shuffle, and Iceberg writes. Provides guidance to a human reviewer rather than posting comments. +description: Use when reviewing a DataFusion Comet pull request. Covers the workflow that applies to every PR and routes to the area-specific review skills for expressions, FFI, memory management, shuffle, Iceberg writes, and documentation. Provides guidance to a human reviewer rather than posting comments. argument-hint: --- @@ -43,6 +43,7 @@ change. | `native/core/src/execution/memory_pools/`, `CometTaskMemoryManager`, `CometShuffleMemoryAllocator`, `MemoryConsumer` / `try_grow` / `reserve` call sites, pool configs | `review-comet-memory-pr` | | `native/shuffle/`, `spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/`, `spark/src/main/java/org/apache/spark/shuffle/`, `CometShuffleExchangeExec` | `review-comet-shuffle-pr` | | `IcebergWriteStrategy`, `IcebergWriteExec`, `IcebergCommitExec`, `CometIcebergWriteExec`, `CometIcebergNativeWrite`, `iceberg_write.rs`, `iceberg_partition_path.rs`, the iceberg-rust pin | `review-comet-iceberg-write-pr` | +| `docs/source/`, `README.md`, `.doc(...)` strings in config classes, doc comments (Scaladoc, Javadoc, Rust `///`) | `review-comet-docs` | For a new operator, read `docs/source/contributor-guide/adding_a_new_operator.md` alongside this skill. There is no dedicated operator review skill yet. @@ -52,8 +53,8 @@ For a PR that deals with timestamps or the session timezone, read cuts across the serde, the native kernels, the scans and the JVM/native boundary, so no area skill owns it. "Timestamps and timezones" in step 5 says what to check. -If the PR falls outside all of these, for example build, CI, docs only, or release tooling, this -skill on its own is the review. +If the PR falls outside all of these, for example build, CI, or release tooling, this skill on its +own is the review. ## 2. Gather PR Metadata @@ -95,7 +96,7 @@ Anything that reports `getSupportLevel()` must report it honestly: - `Compatible()`, matches Spark exactly - `Incompatible(Some("reason"))`, differs in documented ways and requires - `spark.comet.expr.allowIncompatible=true` + `spark.comet.expression..allowIncompatible=true` - `Unsupported(Some("reason"))`, cannot be implemented A PR that quietly marks something `Compatible` while the diff shows a known divergence is the single @@ -264,7 +265,8 @@ For a change on a default path, look for tests with the inputs that broke earlie ### CI -`gh pr checks --repo apache/datafusion-comet`, and again with `--failed` for detail. +`gh pr checks --repo apache/datafusion-comet`. For only the failed checks, add +`--json name,state,bucket,link --jq '.[] | select(.bucket == "fail")'`. Summarize failures in the review. Do not compare against failures on `main`. @@ -299,10 +301,13 @@ same PR, because a follow-up that is not filed as an issue does not happen. Each area skill names the doc for its subsystem and the specific claims in it that go stale. When you load an area skill, read its doc and hold the diff against it. -**Do not ask contributors to hand-edit generated docs.** The compatibility guide pages under -`docs/source/user-guide/latest/compatibility/expressions/` and -`docs/source/user-guide/latest/configs.md` are produced by `GenerateDocs` and regenerated by CI on -every merge to `main`. Review the source that feeds them instead. +**Do not ask contributors to hand-edit generated docs.** `GenerateDocs` fills in the config tables +and the expression compatibility tables when the docs are built, and CI rebuilds and publishes the +docs when a merge to `main` changes their inputs. The +**Generated Content** section of `docs/source/contributor-guide/style_guide.md` says exactly which +text is generated. Review the source that feeds it instead. The prose around the generated tables, +the `_category_template/` files, and the `index.md` pages are written by hand and are reviewed +normally. ## Review Bar