Skip to content

Commit a3b0a1f

Browse files
author
Zoran Simic
committed
Added docs
1 parent 9727737 commit a3b0a1f

5 files changed

Lines changed: 306 additions & 0 deletions

File tree

‎ARCHITECTURE.md‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
## Architecture
2+
3+
### Core Classes Hierarchy
4+
5+
```
6+
ModuleBuilder (abstract base)
7+
├── PythonBuilder (abstract, extends ModuleBuilder)
8+
│ └── Cpython (concrete implementation in cpython.py)
9+
└── External modules (in external/xcpython.py, xtkinter.py)
10+
├── Bdb, Bzip2, Gdbm, LibFFI, Mpdec, Openssl, Readline
11+
├── Sqlite, Uuid, Xz, Zlib, Zstd, TkInter
12+
13+
BuildSetup (coordinates overall compilation)
14+
├── BuildContext (handles isolation hacks on macOS)
15+
├── Folders (manages build directory structure)
16+
└── ModuleCollection (manages module selection and dependencies)
17+
18+
PPG (Global state singleton)
19+
├── config: Config (YAML configuration)
20+
├── target: PlatformId (target OS/arch)
21+
├── cpython: CPythonFamily (available versions)
22+
└── families: dict (extensible family support)
23+
24+
Config
25+
├── Folders (build/dist/sources/logs directories)
26+
├── target platform detection
27+
├── YAML config merging (platform-specific overrides)
28+
29+
PythonInspector
30+
├── Inspect Python installation for portability
31+
├── LibAutoCorrect (rewrite paths to be relative)
32+
└── Report system and shared lib detection
33+
34+
Tracker/Trackable
35+
├── Categorizes found issues/objects by type
36+
└── Provides detailed reports
37+
```
38+
39+
### Key Design Patterns
40+
41+
1. **Hierarchical Module Building**: External modules are compiled first, then Python itself, using the same build framework.
42+
43+
2. **Environment Variable Injection**: `xenv_*` methods dynamically provide environment variables (CPATH, LDFLAGS, PATH, etc.) for compilation.
44+
45+
3. **Platform Abstraction**: `PPG.target` (PlatformId) encapsulates platform logic. Compile methods named `_do_<platform>_compile()` dispatch to platform-specific implementations.
46+
47+
4. **Configuration Precedence**: YAML config supports platform-specific overrides (windows.ext, macos.env, etc.). Most specific setting wins.
48+
49+
5. **Folder Masking (macOS)**: On macOS, `/usr/local` is temporarily masked with a RAM disk to prevent accidental dynamic linking.
50+
51+
6. **Build Isolation**: All external modules compiled to a shared `build/deps/` folder, Python finds them via CPATH/LDFLAGS.
52+
53+
7. **Lazy Version Fetching**: `VersionFamily` caches available versions, fetching from python.org on first access.
54+
55+
8. **Telltale Detection**: Modules check for marker files (`m_telltale`) to determine if they're already available on the system (as shared libs).
56+
57+
9. **Log Aggregation**: Each module logs to a separate file (`01-openssl.log`, `02-cpython.log`, etc.) under `build/logs/`.
58+
59+
60+
## CI/CD
61+
62+
### GitHub Actions
63+
64+
**tests.yml** (main branch & PRs):
65+
- Matrix test on py3.10, 3.11, 3.12, 3.13, 3.14
66+
- Runs: `uvx --with tox-uv tox -e py`
67+
- Coverage upload to coveralls.io (parallel, then finish)
68+
- Linter job: docs + style checks on 3.14
69+
70+
**release.yml** (version tags v*):
71+
- Triggers on `v[0-9]*` tags
72+
- Runs all tests + docs + style
73+
- Builds distribution with `uv build`
74+
- Publishes to PyPI via trusted publishing (OIDC)

‎CLAUDE.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## What This Project Does
6+
7+
portable-python is a CLI tool and Python library for compiling portable CPython binaries from source. Binaries are statically linked so they can be extracted to any folder and used without installation. Supports Linux and macOS (not Windows).
8+
9+
## Common Commands
10+
11+
```bash
12+
# Run all tests (tox manages envs for py310-314, coverage, docs, style)
13+
tox
14+
15+
# Run tests for a single Python version
16+
tox -e py313
17+
18+
# Run a single test
19+
pytest tests/test_build.py::test_build_rc -vv
20+
21+
# Lint check / auto-fix
22+
tox -e style
23+
tox -e reformat
24+
25+
# Run tests directly (from venv)
26+
pytest tests/
27+
28+
# CI uses: uvx --with tox-uv tox -e py
29+
```
30+
31+
## Architecture
32+
33+
**Key classes:**
34+
35+
- `PPG` (versions.py) — Global singleton holding config, target platform, and version families. All modules access shared state through this.
36+
- `BuildSetup` (__init__.py) — Coordinates overall compilation: downloads sources, builds external modules, then CPython.
37+
- `ModuleBuilder` (__init__.py) — Abstract base for anything that gets compiled. Both external C modules and Python itself extend this.
38+
- `PythonBuilder` (__init__.py) — Extends ModuleBuilder for Python implementations.
39+
- `Cpython` (cpython.py) — Concrete PythonBuilder that handles CPython's configure/make/install, optimization, and finalization.
40+
- `Config` (config.py) — Loads and merges YAML configuration (portable-python.yml) with platform-specific overrides.
41+
- `PythonInspector` (inspector.py) — Validates portability of a built Python by checking shared library dependencies and paths.
42+
43+
**External modules** (external/xcpython.py): `Bdb`, `Bzip2`, `Gdbm`, `LibFFI`, `Mpdec`, `Openssl`, `Readline`, `Sqlite`, `Uuid`, `Xz`, `Zlib`, `Zstd` — each is a ModuleBuilder subclass that compiles a C library statically before CPython is built.
44+
45+
**Key patterns:**
46+
47+
- Platform-specific compile logic uses `_do_linux_compile()` / `_do_macos_compile()` method dispatch.
48+
- Environment injection: `xenv_*` methods provide CPATH, LDFLAGS, PATH etc. for compilation.
49+
- On macOS, `/usr/local` is masked with a RAM disk (`FolderMask`) to prevent accidental dynamic linking.
50+
- External modules compile to a shared `build/deps/` prefix; CPython finds them via CPATH/LDFLAGS.
51+
- Telltale detection: modules check for marker files (`m_telltale`) to determine if system already has the library.
52+
- No patches to upstream CPython source — relies solely on configure flags.
53+
54+
**runez** is the foundational utility library (file ops, system info, CLI decorators, logging, Version/PythonSpec types). Check runez before reimplementing anything.
55+
56+
**Additional pointers:**
57+
58+
- `ModuleCollection.selected` contains only the modules chosen for a build — not all candidates.
59+
- Build logs go to `build/logs/NN-modulename.log` (e.g. `01-openssl.log`, `02-cpython.log`).
60+
- YAML config supports platform-specific overrides and path templates — see CONFIGURATION.md.
61+
- See ARCHITECTURE.md for class hierarchy and design patterns, DEVELOP.md for common tasks and dependencies.
62+
63+
## Testing
64+
65+
- pytest with 100% code coverage target
66+
- Tests mock `runez.run()` to avoid actual compilation — uses `--dryrun` mode
67+
- `conftest.py` provides a `cli` fixture (from runez) and forbids HTTP calls (`GlobalHttpCalls.forbid()`)
68+
- Sample YAML configs in `tests/sample-config*.yml` for testing configuration parsing
69+
70+
## Linting
71+
72+
Ruff handles all linting and formatting. Key settings in pyproject.toml:
73+
- Line length: 140
74+
- McCabe complexity: max 18
75+
- Security checks (S rules) disabled in tests
76+
- Numpy-style docstrings

‎CONFIGURATION.md‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# Configuration (portable-python.yml)
2+
3+
portable-python is configured via a YAML file (default: `portable-python.yml` in current directory, override with `--config`).
4+
5+
6+
## Key Sections
7+
8+
### folders
9+
10+
- `build`, `dist`, `sources`, `logs`, `destdir`, `ppp-marker`
11+
- All support path templates: `{build}`, `{version}`, `{abi_suffix}`
12+
13+
### cpython-modules (CSV)
14+
15+
Which external modules to auto-select.
16+
Default is configured in `DEFAULT_CONFIG` (see `src/portable_python/config.py`).
17+
Examples: openssl, zlib, xz, sqlite, bzip2, gdbm, libffi, readline, uuid
18+
19+
### cpython-configure (list)
20+
21+
Extra `./configure` args for CPython.
22+
Default includes: `--enable-optimizations`, `--with-lto`, `--with-ensurepip=upgrade`
23+
24+
### cpython-clean-1st-pass (list)
25+
26+
Files to remove before `compileall` — removes test files, idle, 2to3 (~94 MB savings).
27+
28+
### cpython-clean-2nd-pass (list)
29+
30+
Files to remove after `compileall` — removes pycaches for seldom-used libs (~1.8 MB savings).
31+
32+
33+
## Per-module config
34+
35+
Each external module can be customized with these keys (replace `{module}` with the module name, e.g. `openssl`):
36+
37+
| Key | Purpose |
38+
|-----|---------|
39+
| `{module}-version` | Version to use |
40+
| `{module}-url` | URL to download from |
41+
| `{module}-src-suffix` | File extension if not in URL |
42+
| `{module}-configure` | Custom configure args |
43+
| `{module}-http-headers` | HTTP headers for download |
44+
| `{module}-patches` | File patches to apply |
45+
| `{module}-debian` | Package name on Debian (for dependency detection) |
46+
47+
48+
## Platform-specific overrides
49+
50+
Configuration supports platform-specific sections (e.g. `windows.ext`, `macos.env`, etc.).
51+
The most specific setting wins.
52+
53+
`MACOSX_DEPLOYMENT_TARGET` defaults to 13 (Ventura).

‎DEVELOP.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,3 +66,48 @@ Now inside docker, you run a build:
6666
```shell
6767
portable-python build 3.13.2
6868
```
69+
70+
71+
# Key Dependencies
72+
73+
| Package | Version | Purpose |
74+
|---------|---------|---------|
75+
| click | <9 | CLI framework |
76+
| pyyaml | <7 | Configuration parsing |
77+
| requests | <3 | HTTP downloads |
78+
| runez | <6 | Utilities (file ops, system, logging) |
79+
| urllib3 | <3 | HTTP transport |
80+
| pytest-cov | - | Coverage reporting (dev only) |
81+
82+
**runez** is central: provides file ops (`ls_dir`, `touch`, `compress`/`decompress`), system info (platform detection), CLI decorators (`@click.group`), logging, and version handling (`Version`, `PythonSpec`).
83+
84+
85+
# Common Tasks
86+
87+
## Add a New External Module
88+
89+
1. Create class in `src/portable_python/external/xcpython.py` extending `ModuleBuilder`
90+
2. Set `m_name`, `m_telltale`, `m_debian`, `version` property
91+
3. Implement `url` property (or override `_do_linux_compile()` / `_do_macos_compile()`)
92+
4. Add to `Cpython.candidate_modules()` if it's a CPython sub-module
93+
5. Add tests in `tests/test_setup.py`
94+
95+
## Add a New Config Option
96+
97+
1. Update `DEFAULT_CONFIG` in `src/portable_python/config.py`
98+
2. Use `PPG.config.get_value("key")` to retrieve it in code
99+
3. Add tests to `tests/test_setup.py`
100+
101+
## Fix a Portability Issue
102+
103+
1. Run `portable-python inspect <PATH>` to diagnose
104+
2. If lib is being dynamically linked, add to module list or update isolation
105+
3. Use `LibAutoCorrect.run()` logic (or extend it) to fix paths
106+
4. Add test case to `tests/test_inspector.py`
107+
108+
## Bump Python Support
109+
110+
1. Update `pyproject.toml` classifiers and `setup.py` `python_requires`
111+
2. Update `.github/workflows/tests.yml` matrix
112+
3. Update `CPythonFamily.min_version` if needed
113+
4. Run full test matrix with `tox`

‎README.rst‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,64 @@ Note that you can use ``--dryrun`` mode to inspect what would be done without do
128128
Would tar build/3.9.7 -> dist/cpython-3.9.7-macos-x86_64.tar.gz
129129

130130

131+
CLI reference
132+
-------------
133+
134+
Main entry point::
135+
136+
portable-python [OPTIONS] COMMAND [ARGS]
137+
138+
**Global options**:
139+
140+
- ``--config PATH``: Config file (default: ``portable-python.yml``)
141+
- ``--quiet``: Turn off DEBUG logging
142+
- ``--dryrun`` / ``-n``: Show what would be done
143+
- ``--target PLATFORM``: Override detected platform (for testing)
144+
145+
146+
**build** ``<PYTHON_SPEC>`` - Build a portable Python binary::
147+
148+
portable-python build 3.13.2 -m openssl,zlib
149+
150+
- ``--modules, -m CSV``: External modules to include
151+
- ``--prefix, -p PATH``: Use ``--prefix`` (non-portable)
152+
153+
154+
**build-report** ``[PYTHON_SPEC]`` - Show module status and what will be compiled:
155+
156+
- ``--modules, -m CSV``: Specific modules to check
157+
- Validates that modules can be built
158+
159+
160+
**inspect** ``<PATH>`` - Check if a Python installation is portable::
161+
162+
portable-python inspect /usr/bin/python3
163+
164+
- ``--modules, -m MODULES``: Which modules to inspect
165+
- ``--verbose, -v``: Show full ``.so`` report
166+
- ``--prefix, -p``: Built with ``--prefix`` (not portable)
167+
- ``--skip-so, -s``: Don't check all ``.so`` files
168+
169+
170+
**list** ``[FAMILY]`` - List available versions (default: cpython)::
171+
172+
portable-python list cpython
173+
174+
- ``--json``: Output as JSON
175+
176+
177+
**diagnostics** - Show system diagnostics
178+
179+
180+
**recompress** ``<PATH> <EXT>`` - Re-compress existing binary tarball (for comparing compression sizes)
181+
182+
183+
**lib-auto-correct** ``<PATH>`` - Auto-correct exes/libs to use relative paths:
184+
185+
- ``--commit``: Actually perform changes (dryrun by default)
186+
- ``--prefix, -p PATH``: Expected ``--prefix`` from build
187+
188+
131189
Library
132190
-------
133191

0 commit comments

Comments
 (0)