A Python library that splits Markdown into hierarchical sections. It keeps code blocks intact and links each section to its parent section.
- Lines inside fenced code blocks (
```or~~~, including fences in list items), HTML comment blocks,<pre>,<script>,<style>and<textarea>blocks and YAML front matter are never treated as headers. Section text keeps them exactly as written - Splits on headers from H1 to H6.
MarkdownSplitter(max_level=3)keeps deeper headers as text of the enclosing section - Gives each section an id and its parent section's id, so the header hierarchy can be rebuilt as a tree or loaded into a graph. Headers with the same text are kept as separate sections
- Text before the first header, or a document without headers, is returned as a level-0 section with an empty header
section.breadcrumb()returns the section's header path, such asDocker > Installation, to send with the section text to an embedding model or LLM- Records where each section is in the source text.
text[start:end]withstart, end = section.metadata.spanreturns the section's header line and text as written.metadata.char_countis the length of that span, not ofcontentorto_markdown()
pip install markdown-chunkifyThe splitter depends on Pydantic.
from markdown_chunkify import MarkdownSplitter
splitter = MarkdownSplitter()
# Split from text
sections = splitter.split_text(markdown_text)
# Split from file
sections = splitter.split_file("document.md")MarkdownSplitter is immutable, so one instance can be reused.
metadata.id is the section's position in document order, starting at 0, so sections[i].metadata.id == i. metadata.parent_id is the id of the enclosing section, or None for top-level sections and text before the first header.
text = "# Docker\n## Installation\n## Usage\n# Kubernetes"
sections = MarkdownSplitter().split_text(text)
[(s.header, s.metadata.id, s.metadata.parent_id) for s in sections]
# [('Docker', 0, None), ('Installation', 1, 0), ('Usage', 2, 0), ('Kubernetes', 3, None)]
sections[sections[1].metadata.parent_id].header # 'Docker'The other relationships follow from these two fields:
- A section's children are the sections whose
parent_idis itsid. - Sections with the same
parent_idare siblings, whatever their header levels. In# A,### C,## B, both C and B are children of A. A skipped level adds no placeholder section. - The previous and next sections in reading order have ids
id - 1andid + 1.
Ids are unique within one document. To use them across documents, combine them with your own document key, such as f"{doc_key}:{section.metadata.id}". Ids are positions, not content versions: inserting or removing a section renumbers the sections after it, and editing a section's text keeps its id. To detect changed sections between runs, for example to skip re-extracting entities from unchanged text, hash content. Changing max_level changes the tree and the ids.
section.breadcrumb() joins the parent headers and the section's own header. Sent with the section text, it tells an embedding model or LLM what the section is about when the text itself says "it" or "this step".
text = "# Docker\n## Installation\nRun the installer."
section = MarkdownSplitter().split_text(text)[1]
section.breadcrumb() # 'Docker > Installation'
section.breadcrumb(separator=" / ") # 'Docker / Installation'
prompt_text = f"{section.breadcrumb()}\n\n{section.content}"Text before the first header returns an empty string. The breadcrumb is computed from metadata.parents, so it is not part of the serialized section.
PDF parsers often produce ligatures (fi, fl), non-breaking spaces and fullwidth characters. A chunk with find doesn't match a search for find, and entity extraction can miss the word. The splitter doesn't clean text. Clean it before calling split_text, so the spans index into the cleaned string you pass in.
import unicodedata
from markdown_chunkify import MarkdownSplitter
clean = unicodedata.normalize("NFKC", pdf_markdown)
clean = clean.replace("\u00ad", "").replace("\u200b", "") # Soft hyphen, zero-width space
sections = MarkdownSplitter().split_text(clean)NFKC normalization from the standard library replaces ligatures, non-breaking spaces and fullwidth characters, and keeps accents:
| Input | NFKC output |
|---|---|
find flow |
find flow |
a\xa0b (non-breaking space) |
a b |
ＡＢＣ１２３ (fullwidth) |
ABC123 |
Müller |
Müller |
x² |
x2 |
co\xadoperate (soft hyphen) |
unchanged |
zero\u200bwidth (zero-width space) |
unchanged |
NFKC also folds superscripts and subscripts, so x² becomes x2. It leaves soft hyphens and zero-width spaces in the text, so remove them with str.replace.
- Only ATX headers (
#to######) are recognized. Setext headers, underlined with===or---, stay in the section text. - Headers and fences inside blockquotes (
> # Title) are not recognized. - A line indented by 4 or more spaces that starts with
```or~~~is treated as a fence, even inside an indented code block. If that fence is never closed, the headers after it are treated as text and a warning is logged. - An HTML comment or a
<pre>,<script>,<style>or<textarea>block is only recognized when it starts the line. One that opens mid-line (text <!--ortext <pre>) does not hide the#lines inside it. - Headers inside
<details>are split, as GitHub renders Markdown after a blank line inside<details>. The<details>and</details>tags stay in the text of the sections around them. - A header with no text (
#,## ##, or only whitespace after the#) is kept as text of the enclosing section, not an empty section. CommonMark renders it as an empty heading. - A document that starts with
---is treated as having YAML front matter only if the line after it is akey:entry and every line up to the next---or...looks like YAML (key: value,- item, indented lines, comments or blank lines). Otherwise the---is a horizontal rule and the headers after it are split as usual. - Sections have no size limit. A document without headers is returned as one section and a long section stays whole, so split long sections before embedding them.
Section and SectionMetadata are exported from the package root:
from markdown_chunkify import Section
from markdown_chunkify import SectionMetadataThe models are frozen, so to change a section, call section.model_copy(update={"content": "..."}), which returns a changed copy. The freezing is shallow: metadata.parents is a plain dict, so treat it as read-only.
A section returned by the split_text and split_file methods. It holds the section's header, text, header level and metadata.
class Section(BaseModel):
model_config = ConfigDict(frozen=True)
# The header of the section (without #), empty for text before the first header
header: str
# The content of the section
content: str
# Number of # symbols (1-6), 0 for text before the first header
level: int
# Hierarchy and position information
metadata: SectionMetadata
# Convert the section back to Markdown
def to_markdown(self) -> str: ...
# Join the parent headers and the section's header, such as "Docker > Installation"
def breadcrumb(self, separator: str = " > ") -> str: ...A section's id, its parent's id, its parent headers and where it is in the source text.
class SectionMetadata(BaseModel):
model_config = ConfigDict(frozen=True)
# Position in document order, starting at 0
id: int
# Id of the enclosing section, None for top-level sections
parent_id: int | None
# Parent headers by level, such as {"h1": "Guide", "h2": "Install"}
parents: dict[str, str]
# Start (inclusive) and end (exclusive) offsets in the source text
span: tuple[int, int]
# Number of characters in the span, computed from it
char_count: intThe span starts at the header line and ends at the last non-whitespace character of the section text. It does not include subsections, leading blank lines or trailing whitespace. Offsets count Python characters (code points), not UTF-8 bytes or UTF-16 units, in the string passed to split_text, including a byte order mark. split_file offsets index into the file's contents as decoded with its encoding, with line endings kept as written.
Python 3.12 or later.
Report bugs and request features in GitHub issues. For a large change, open an issue to discuss it before you start.
Set up a development environment with uv:
git clone https://github.com/Gal-Gilor/markdown-chunkify.git
cd markdown-chunkify
uv syncRun the tests and Ruff before opening a pull request:
uv run pytest -vv
uv run ruff check .
uv run ruff format .Open pull requests against main. The Lint, test and build workflow runs Ruff, the tests on Python 3.12, 3.13 and 3.14, and a wheel build on each pull request. Add tests for changed behavior, and add user-facing changes to CHANGELOG.md.
To release a version, update version in pyproject.toml and the changelog, then publish a GitHub release tagged v<version>, such as v0.3.0. The Release workflow builds the package and publishes it to PyPI with trusted publishing.
Apache License 2.0. See LICENSE.