Documentation Guidelines

This document outlines the guidelines for writing Kotlin DataFrame website documentation.

For KDocs guidelines, see KDOC_GUIDELINES.md.

General documentation information

Documentation website information

Address: https://kotlin.github.io/dataframe
Deployed through GitHub Actions.
- configuration file
- For local build instructions, see docs/README.md.
Built using JetBrains Writerside.
- Indexed with Algolia.

Sources information

All website documentation sources can be found in the docs/StardustDocs directory.
All samples code is stored as tests inside the :samples module. See Samples information.
- Samples are copied from tests code using Korro.
- Some text and HTML outputs are generated by :samples tests running. HTML outputs are automatically saved into docs/StardustDocs/resources.

Old documentation samples are still placed in :core module. Sample migration into :samples module is in progress — issue #898; :core still has a legacy samples source set + samplesTest/korro tasks that will be removed. See Migration from :core.

Topic structure

All documentation website topics can be one of the following types:

operation documentation
guides (tutorials)
others

These guidelines are mostly applied to operation documentation topics, but all the principles and standards apply to all the others.

General rules for all pages:

When mentioning any operation or any Kotlin DataFrame structures the first time on the page, add a link to the corresponding page. For example:
```
[`DataFrame`](DataFrame.md), [`DataRow`](DataRow.md), [`DataColumn`](DataColumn.md),
[`ColumnGroup`](DataColumn.md#columngroup), [`ValueColumn`](DataColumn.md#valuecolumn),
[`FrameColumn`](DataColumn.md#framecolumn), [`ColumnSet`](ColumnSelectors.md#column-resolvers),
[column selectors](ColumnSelectors.md), [`GroupBy`](groupBy.md)
```
Note that Writerside resolves topics by file name, so use just DataFrame.md, not a relative path.
- If the operation takes a row expression or a row condition — don't forget to link it (DataRow.md#row-expressions / DataRow.md#row-conditions)! Also add a note on what the lambda receives and how the row values are accessed — for any kind of row expression or row condition (RowExpression, RowValueExpression, RowFilter, RowValueFilter, …). For example, for a RowFilter (see filter.md):
```
The row condition receives each row both as `this` and as `it`, so the values of the row can be accessed directly
using [Extension Properties](extensionPropertiesApi.md) or
[`String` column accessors](StringApi.md#string-column-accessors),
including the values of nested columns of a [`ColumnGroup`](DataColumn.md#columngroup).
```
  Adapt it to the lambda type: a RowValueExpression / RowValueFilter receives the row as this and the current value as it; a RowColumnExpression receives the row and the column as the parameters row and col.
- When linking dataframe structures, also link ColumnSet and other column resolvers (ColumnSelectors.md#column-resolvers) where they are mentioned.
Follow Spelling conventions.
Simple, clear language, without obscure words or complex vocabulary; avoid narrow concepts.
Use sample code created by Korro.

Operation documentation topics

Operation documentation topics are the most important part of the documentation.

Note that Kotlin DataFrame operations are often a sequence of methods rather than a single method; all methods — parts of the operation should be on the single page dedicated to this operation.

Operation documentation topic must contain comprehensive information about the operation:

Detailed yet short operation description. Use simple words and clearly explain what is happening.
For complex operations: all methods — parts of the operation should be on the single page dedicated to this operation. Each operation step should have its own section.
All method overloads; sometimes familiar (isEmpty/isNotEmpty) or shortcut (drop/dropNulls) operations are placed on the same page.
All argument descriptions; if any argument is enum specific to this operation, describe its possible values and how they affect the operation result.
Describe in detail any non-obvious behavior and edge cases (empty, containing null, frame columns and column groups, etc.), but avoid mentioning implementation details. For most important cases, add an example sample.
Documentation on the website should be aligned with the operation KDoc. The documentation on the website may contain more detailed information than in KDoc, but certainly not less! Type parameters can be omitted if they are not used in the operation directly.
Topic should contain an actual code example (extracted from tests-samples with Korro samples) with an explicit input and output (generated with Sample Helper). See Operation Examples
Topic should contain links to all related operations (including reverse if there is one).
If there's a relevant page on the API reference, link to it.

Operation Examples

For an operation that selects columns or uses row values, the basic usage example must always have two versions — one with the Extension Properties API and one with the String API — shown through <tab>.
- Korro inserts these tabs automatically when it sees _properties or _strings suffixes in the test name — do not write <tabs>/<tab> markup manually for such samples. Use the base name (without the suffix) in the FUN directive.
- Some operations can't be used with plain column names (Strings): write such a second version with String column accessors (col<T>("name"), getValue<T>("name"), …) instead of skipping it. Don't use the outdated Invoked String API.
- Sometimes it's also useful to show an example with the Columns Selection DSL.
Examples should contain clear explanations (both as code comments and text in the section).
Use illustrative meaningful example (do not show dropNulls example on dataframe without nulls).
Show an initial dataset for examples first: put it inside the ### Examples section, right before the first example (and not somewhere earlier on the page), with a short lead-in like "The following dataframe will be used in the examples below:". If the page describes several operations, the dataset is shown once — before the very first example of the page.
The required number of examples depends on the complexity of the operation, number of overloads, and other factors.

Extended formatting

Coloring (and formatting in general) of output dataframes helps better understand the result. It's done by using .format and .formatHeader methods. Use them to highlight important columns and rows.

Highlight rows for rows-oriented operations (e.g., filter), and column headers for columns-oriented operations (e.g., select); some operations are both rows- and columns-oriented (e.g., join). There's a default formatting for headers for selected columns: .defaultHeaderFormatting { }.

Place formatting out of the sample, after // SampleEnd:

@Test
fun selectTwoCols() {
    // SampleStart
    df.select { col1 and col2 }
    // SampleEnd
        .defaultHeaderFormatting { col1 and col2 }
        .saveDfHtmlSample()
}

Structure

Title
Writerside summaries
Operation description.
1. First — what it actually does ("Adds ...", "Computes ...", "Removes...", "Sorts ...").
2. Then — what it returns ("Returns ...").
An API reference link like 'API reference: name'
Method(s) signature. For complex operations — complete operation grammar.
Any important notes.
See also section — links to reverse or generally reversed operations.
### Parameters — parameters description with their types.
### Examples — examples (code sample with outputs).
The DataColumn overload section, if the operation has one.

Parameters and examples are always separate headers (### Parameters, ### Examples), not bold paragraphs, so that every operation section looks the same.

If the operation is complex and consists of several steps, each step can be described in a separate subsection keeping this structure.

Pages with several operations

If a topic describes several operations (e.g., sortBy and sortByDesc), then:

Each operation gets its own ## section with the structure above.
Don't list the operations of the page anywhere: the page starts with the first operation section right after the summaries, and the sections themselves describe the operations.
The "See also" section is written only once — in the first operation section, in its usual place (after the operation description and notes, before ### Parameters) — and links only to operations that are not described on this page. Don't repeat "See also" in every operation section.

Signatures

Write signatures as pseudocode with types: a Kotlin-like call with the receiver, parameter names and their types, lambda parameters and the lambda result type. Use a text code block:

df.sortWith(comparator: Comparator<DataRow>)
df.sortWith { row1: DataRow, row2: DataRow -> Int }

Optional parts are wrapped in square brackets, alternatives are separated with |, like in the operation grammar of complex operations.

Always write the receiver, named after its type:

Type	Name
`DataFrame`	`df`
`DataColumn`	`col`
`DataRow`	`row`
`ColumnGroup`	`colGroup`
`FrameColumn`	`frameCol`
`GroupBy`	`groupBy`
`Pivot`	`pivot`

Parameters always keep their names from the library — check them in the function declaration. A user can pass any argument by name (df.concat(frames = listOf(df1, df2))), and the page must match the KDoc and IDE hints, so never rename them for the docs: df.concat(vararg frames: DataFrame), not df.concat(vararg dfs: DataFrame); col.concat(vararg other: DataColumn), not col.concat(vararg cols: DataColumn).

The prefixes above are only for the names that are not part of the API: receivers, lambda parameters ({ row: DataRow -> Boolean }), and receivers of other kinds (dfs: Iterable<DataFrame>, cols: Iterable<DataColumn>).

Type parameters (<T>, <C>, <R>, …) are written — in the signature and in the description — only if they carry meaning:

they take part in the function logic — for example, a reified type parameter that selects columns or values by type, like colsOf<T>();
or they connect the types of the input and the result — for example, collectionColumn.concat(): List<T> for a DataColumn<Collection<T>>: the result elements have the type of the collections' elements.

Otherwise omit them: Iterable<DataRow>, not Iterable<DataRow<T>>; df.concat(vararg frames: DataFrame): DataFrame, not DataFrame<T>.concat(vararg frames: DataFrame<T>): DataFrame<T> (the schema type T is the same everywhere and says nothing new).

If a receiver or an argument needs a more specific type than its name tells (a column of collections, an Iterable of rows), declare it on a separate line of the signature block, after the calls:

collectionColumn.concat(): List<T>

collectionColumn: DataColumn<Collection<T>>

A receiver of a more specific kind may get a descriptive name (collectionColumn) instead of the plain prefix.

Parameters

Types are written not only in the signature, but in the parameter description as well: give each parameter its type as pseudocode (the same way as in the signature) or describe it in words. For lambdas, describe what the lambda receives and what it must return.

* `keepColumns: Boolean = false` — whether the original columns are kept in the result.
* `expression: (DataRow) -> Value` — a lambda that computes the new value for each row.

Columns arguments. Most operations select columns either by string names or with the Columns Selection DSL, and the parameter description must say which of them this operation supports:

Both ways — say that columns can be selected by string names or with the Columns Selection DSL, and link the DSL page.
Only the DSL (no String overload) — say so explicitly.
The ways differ between overloads (like in rename, where the string overload takes the columns directly and the DSL one takes a selector) — describe each overload separately, so it's clear what exactly each of them accepts.
The operation uses its own specialized DSL (like the Sort DSL of sortBy or the Pivot DSL of pivot) — name that DSL, link its section, and explain what it adds on top of the Columns Selection DSL.

`DataColumn` overloads

If the operation also has an overload for a single DataColumn, describe it at the end of the operation section under the header:

### `funName` on `DataColumn`

with its own description, signature, parameters, and examples (see filter.md).

If the operation is defined for several receivers (DataFrame, DataRow, DataColumn, GroupBy, Iterable, …) and the overloads differ substantially, describe the DataFrame operation first, without a header — the page starts with it right after the summaries, like a single-operation page. Then give every other receiver its own ## `funName` on `Receiver` section with the same structure, the DataColumn one included (see concat.md). The "See also" section is written only once, in the DataFrame part.

Writerside summaries

Every topic must have Writerside summaries right after the title (and the Korro IMPORT directive, if any). They are used for the page metadata, cards, and link previews.

For operation topics, use the same template for all pages, with the operation name substituted:

<web-summary>
Discover the `operationName` operation for Kotlin DataFrame.
</web-summary>

<card-summary>
Discover the `operationName` operation for Kotlin DataFrame.
</card-summary>

<link-summary>
Discover the `operationName` operation for Kotlin DataFrame.
</link-summary>

All three summaries have the same text. If a topic has no summaries yet, add them; if it already has meaningful custom ones, keep them.

Links from KDocs (`DocumentationUrls`)

Operation KDocs link to the documentation website through DocumentationUrls: KDoc snippets with URLs, for example:

/** [See `sortWith` on the documentation website.]({@include [Url]}/sortby.html#sortwith) */
public typealias SortWith = Nothing

They are included into operation KDocs as For more information: {@include [DocumentationUrls.SortWith]}.

Nothing checks these URLs automatically, so a stale one silently leads to a missing page or section. Whenever a topic or a section is added, renamed, moved, or split, open DocumentationUrls.kt and verify every URL that points to the affected page:

the page part is the topic file name in lowercase with the .html extension (sortWith.md → sortwith.html);
the anchor part is the Writerside anchor of the section (the heading in lowercase, punctuation dropped, spaces replaced with -);
if a new operation topic has no entry yet and its KDoc refers to the website, add one following the style of the neighboring declarations.

Guide topics

Guide pages don't have strict structure rules. Their purpose is to show Kotlin DataFrame best practices and provide users with clear examples that can be applied in their actual work.

Try to show intermediate results of operations.

Samples information

All code samples are written as usual tests inside the :samples module: samples/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples.

Make sure samples are valid by running these tests.
Code samples are extracted from the source code using Korro.
HTML dataframes and svg plots are automatically generated using Sample Helper. and save into a resources directory and an images directory respectively.

General rules:

Provide maximum quality code (meaningful variable names, avoid unnecessary complexity, etc.).
Do not write samples manually, all samples should be written as tests and extracted via Korro.
One test code file for one documentation page.
Tests should not be ignored (if possible) and should pass.
Use Compiler Plugin / Extension Property API for examples primarily. For operation documentation, show both APIs if possible.
Keep the structure between topic and sample files.
Use meaningful examples (e.g., use dataset with nulls for null-related operations).

Korro

Korro is a tool for extracting code samples from the source code.

Write a sample inside a test method.
Use // SampleStart and // SampleEnd comments to mark sample code. If not specified, the whole body is used (not recommended!).

Provide a test class path inside IMPORT korro directive (at the beginning of the topic file):

<!---IMPORT org.jetbrains.kotlinx.dataframe.samples.api.PivotSamples-->

Place FUN and END directives in the desired place for sample:

<!---FUN pivotMedianBy-->

<!---END-->

For grouped samples (test names with the _properties / _strings suffixes), write the base test name in the FUN directive, without the suffix (valueCountsSelector_properties + valueCountsSelector_strings → ). Korro finds all suffixed variants itself and wraps them into <tabs>. The generated outputs have the base name too — Sample Helper cuts off the suffix (./resources/valueCountsSelector.html).

Code samples and their outputs will be inserted between them after korro gradle task run: the code, then everything the test saved with Sample Helper — the <inline-frame> for an HTML table, the <img> for a plot, the text output — in the order it was saved. Don't add an <inline-frame> / <img> for a sample by hand — it would duplicate the generated one. Both files must be included in korro configuration in the Gradle build script (korro{ docs { from(...) }; samples { from(...) } }).

Sample Helper

SampleHelper is a base test class that allows saving DataFrames (or other Kotlin DataFrame entities) as interactive HTML, Kandy plots as SVG, as well as String or CodeString as text samples with the special methods by running tests. Resulting samples have names equal to the test method name without the _properties / _strings suffix (test_name.html or test_name.svg; test_name_properties → test_name.html). Every saved output is also written as a Korro output line, so Korro inserts it into the topic inside the sample's … block automatically.

Inherit test class from DataFrameSampleHelper.

Use .saveDfHtmlSample() for saving DataFrame, DataColumn, GroupBy or any other Kotlin DataFrame structure as an interactive HTML table sample (if the receiver is not suitable, try to use .toDataFrame(), .values() or .frames()). Korro inserts this table into the topic as an <inline-frame> right after the code sample, inside the … block — don't write it by hand.
- The only <inline-frame> written by hand is the one for an output shown without its code (for example, a highlighted copy of the input dataframe): a test that only saves an output, without // SampleStart / // SampleEnd and without FUN marks in the topic.
- For a _properties / _strings pair, save the output in one variant only (usually _properties): both variants write the same file.
- HTML outputs are saved into the docs/StardustDocs/resources folder. Subfolder names are defined by DataFrameSampleHelper arguments: DataFrameSampleHelper("<pageName>", "api") writes resources/api/<pageName>/<testName>.html.
- The <inline-frame> src is nevertheless always the flat file name — ./resources/<testName>.html, never ./resources/api/<pageName>/<testName>.html (the generated iframes already use it; keep it in mind for a hand-written one and when checking links). Writerside finds the file through _shadow_resources, which registers resources by file name only; a nested src builds fine but renders an empty frame.
Use .savePlotSVGSample() for saving a Kandy plot as SVG into docs/StardustDocs/images; Korro inserts it as an <img>.
Use .saveSample() for saving text sample from the String or CodeString (.saveTextSample() for a String that should be shown as a text block).
- Text samples are saved into a temporary build folder and exported automatically while running korro.

Migration from `:core`

Issue #898.

Some samples are still generated from :core tests using Korro. Some of them use an outdated way to save HTML samples (called "dataframe explainer") and have @TransformDataFrameExpressions annotation. This is an outdated way to save HTML samples! Do not use it! We want to migrate all samples to the new way of saving HTML samples using Sample Helper.

All samples should be placed as tests in the :samples module only!

To migrate to new samples, remove all old Korro marks and samples, and replace them with new samples written in :samples tests (with fresh Korro marks) and outputs created using Sample Helper.

Standard operation documentation writing workflow

Use a reference topic as a template, do not create a new one from scratch. The reference topics, which fully follow these guidelines, are filter.md (with FilterSamples.kt) and concat.md (with ConcatSamples.kt). Other topics may be outdated: take only the mechanics from them (Korro marks, samples), never the structure. When writing / updating with agents, provide the template topic as a context.
Create a topic as .md file in docs/StardustDocs/topics if it doesn't exist yet. Add it to the site map.
Write the topic content following rules and structure.
- Find KDoc operation in the generated source folder if it exists and well-written. Try to reuse it for the page.
Create a test file in the :samples module by this path. Write code samples as tests. Mark out code and the topic file with Korro marks (IMPORT in the beginning of the file, FUN and END). Add example outputs generation using Sample Helper methods (.save..()). Korro inserts the outputs (<inline-frame> for HTML dataframes, <img> for plots, text outputs) into the FUN/END block itself — don't add them to the topic by hand.
Add the test file and topic file paths to Korro config in the Gradle build script (korro{ docs { from(...) }; samples { from(...) } }).
Run Gradle tasks of the :samples module: clean, build, test, korro. korro must be run after test!
- It's very important to run clean before build, test and korro: otherwise outdated korro output lines (text samples in samples/build/) from a previous build, including those of removed or renamed samples, may survive and end up in the documentation.
- clean does not touch HTML outputs: they are written straight into the tracked docs/StardustDocs/resources folder. So before running the tests, delete the page's output folder docs/StardustDocs/resources/<subFolder>/<pageName>/ — the tests recreate it, and HTML of removed or renamed samples doesn't survive (and doesn't get registered by updateShadowResources).
Check if code samples and their outputs (<inline-frame>s) appear inside the FUN/END blocks of the topic, and HTML frames are added to resources.
Run updateShadowResources Gradle task. This will update a special _shadow_resources file to build website correctly.
Check the resulting page manually, by running the website manually.

Navigation Menu

Repository navigation

DOCUMENTATION_GUIDELINES.md

DOCUMENTATION_GUIDELINES.md

Documentation Guidelines

General documentation information

Documentation website information

Sources information

Topic structure

Operation documentation topics

Operation Examples

Extended formatting

Structure

Pages with several operations

Signatures

Parameters

`DataColumn` overloads

Writerside summaries

Links from KDocs (`DocumentationUrls`)

Guide topics

Samples information

Korro

Sample Helper

Migration from `:core`

Standard operation documentation writing workflow

FilesExpand file tree

DOCUMENTATION_GUIDELINES.md

Latest commit

History

DOCUMENTATION_GUIDELINES.md

File metadata and controls

Documentation Guidelines

General documentation information

Documentation website information

Sources information

Topic structure

Operation documentation topics

Operation Examples

Extended formatting

Structure

Pages with several operations

Signatures

Parameters

DataColumn overloads

Writerside summaries

Links from KDocs (DocumentationUrls)

Guide topics

Samples information

Korro

Sample Helper

Migration from :core

Standard operation documentation writing workflow

`DataColumn` overloads

Links from KDocs (`DocumentationUrls`)

Migration from `:core`