Skip to content

Commit 62de0a3

Browse files
zsimiczsimic
authored
Merge pull request #76 from codrsquad/added-docs
Metadata in pyproject.toml
2 parents 9727737 + 9e62f4d commit 62de0a3

14 files changed

Lines changed: 387 additions & 89 deletions

‎.github/workflows/release.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ jobs:
2020
python-version: "3.14"
2121

2222
- uses: astral-sh/setup-uv@v7
23-
- run: uvx --with tox-uv tox -e py,docs,style
23+
- run: uvx --with tox-uv tox -e py,style
2424
- run: uv build
2525
- name: Publish to PyPI
2626
uses: pypa/gh-action-pypi-publish@release/v1

‎.github/workflows/tests.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,4 +56,4 @@ jobs:
5656
python-version: "3.14"
5757

5858
- uses: astral-sh/setup-uv@v7
59-
- run: uvx --with tox-uv tox -e docs,style
59+
- run: uvx --with tox-uv tox -e style

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,4 @@ root/
2525
venv*/
2626

2727
pp*.yml
28+
uv.lock

‎ARCHITECTURE.md‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
## Architecture
2+
3+
### Core Classes Hierarchy
4+
5+
```
6+
ModuleBuilder (abstract base)
7+
├── PythonBuilder (abstract, extends ModuleBuilder)
8+
│ └── Cpython (concrete implementation in cpython.py)
9+
└── External modules (in external/xcpython.py, xtkinter.py)
10+
├── Bdb, Bzip2, Gdbm, LibFFI, Mpdec, Openssl, Readline
11+
├── Sqlite, Uuid, Xz, Zlib, Zstd, TkInter
12+
13+
BuildSetup (coordinates overall compilation)
14+
├── BuildContext (handles isolation hacks on macOS)
15+
├── Folders (manages build directory structure)
16+
└── ModuleCollection (manages module selection and dependencies)
17+
18+
PPG (Global state singleton)
19+
├── config: Config (YAML configuration)
20+
├── target: PlatformId (target OS/arch)
21+
├── cpython: CPythonFamily (available versions)
22+
└── families: dict (extensible family support)
23+
24+
Config
25+
├── Folders (build/dist/sources/logs directories)
26+
├── target platform detection
27+
├── YAML config merging (platform-specific overrides)
28+
29+
PythonInspector
30+
├── Inspect Python installation for portability
31+
├── LibAutoCorrect (rewrite paths to be relative)
32+
└── Report system and shared lib detection
33+
34+
Tracker/Trackable
35+
├── Categorizes found issues/objects by type
36+
└── Provides detailed reports
37+
```
38+
39+
### Key Design Patterns
40+
41+
1. **Hierarchical Module Building**: External modules are compiled first, then Python itself, using the same build framework.
42+
43+
2. **Environment Variable Injection**: `xenv_*` methods dynamically provide environment variables (CPATH, LDFLAGS, PATH, etc.) for compilation.
44+
45+
3. **Platform Abstraction**: `PPG.target` (PlatformId) encapsulates platform logic. Compile methods named `_do_<platform>_compile()` dispatch to platform-specific implementations.
46+
47+
4. **Configuration Precedence**: YAML config supports platform-specific overrides (windows.ext, macos.env, etc.). Most specific setting wins.
48+
49+
5. **Folder Masking (macOS)**: On macOS, `/usr/local` is temporarily masked with a RAM disk to prevent accidental dynamic linking.
50+
51+
6. **Build Isolation**: All external modules compiled to a shared `build/deps/` folder, Python finds them via CPATH/LDFLAGS.
52+
53+
7. **Lazy Version Fetching**: `VersionFamily` caches available versions, fetching from python.org on first access.
54+
55+
8. **Telltale Detection**: Modules check for marker files (`m_telltale`) to determine if they're already available on the system (as shared libs).
56+
57+
9. **Log Aggregation**: Each module logs to a separate file (`01-openssl.log`, `02-cpython.log`, etc.) under `build/logs/`.
58+
59+
60+
## CI/CD
61+
62+
### GitHub Actions
63+
64+
**tests.yml** (main branch & PRs):
65+
- Matrix test on py3.10, 3.11, 3.12, 3.13, 3.14
66+
- Runs: `uvx --with tox-uv tox -e py`
67+
- Coverage upload to coveralls.io (parallel, then finish)
68+
- Linter job: docs + style checks on 3.14
69+
70+
**release.yml** (version tags v*):
71+
- Triggers on `v[0-9]*` tags
72+
- Runs all tests + docs + style
73+
- Builds distribution with `uv build`
74+
- Publishes to PyPI via trusted publishing (OIDC)

‎CLAUDE.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## What This Project Does
6+
7+
portable-python is a CLI tool and Python library for compiling portable CPython binaries from source. Binaries are statically linked so they can be extracted to any folder and used without installation. Supports Linux and macOS (not Windows).
8+
9+
## Common Commands
10+
11+
```bash
12+
# Run all tests (tox manages envs for py310-314, coverage, docs, style)
13+
tox
14+
15+
# Run tests for a single Python version
16+
tox -e py313
17+
18+
# Run a single test
19+
pytest tests/test_build.py::test_build_rc -vv
20+
21+
# Lint check / auto-fix
22+
tox -e style
23+
tox -e reformat
24+
25+
# Run tests directly (from venv)
26+
pytest tests/
27+
28+
# CI uses: uvx --with tox-uv tox -e py
29+
```
30+
31+
## Architecture
32+
33+
**Key classes:**
34+
35+
- `PPG` (versions.py) — Global singleton holding config, target platform, and version families. All modules access shared state through this.
36+
- `BuildSetup` (__init__.py) — Coordinates overall compilation: downloads sources, builds external modules, then CPython.
37+
- `ModuleBuilder` (__init__.py) — Abstract base for anything that gets compiled. Both external C modules and Python itself extend this.
38+
- `PythonBuilder` (__init__.py) — Extends ModuleBuilder for Python implementations.
39+
- `Cpython` (cpython.py) — Concrete PythonBuilder that handles CPython's configure/make/install, optimization, and finalization.
40+
- `Config` (config.py) — Loads and merges YAML configuration (portable-python.yml) with platform-specific overrides.
41+
- `PythonInspector` (inspector.py) — Validates portability of a built Python by checking shared library dependencies and paths.
42+
43+
**External modules** (external/xcpython.py): `Bdb`, `Bzip2`, `Gdbm`, `LibFFI`, `Mpdec`, `Openssl`, `Readline`, `Sqlite`, `Uuid`, `Xz`, `Zlib`, `Zstd` — each is a ModuleBuilder subclass that compiles a C library statically before CPython is built.
44+
45+
**Key patterns:**
46+
47+
- Platform-specific compile logic uses `_do_linux_compile()` / `_do_macos_compile()` method dispatch.
48+
- Environment injection: `xenv_*` methods provide CPATH, LDFLAGS, PATH etc. for compilation.
49+
- On macOS, `/usr/local` is masked with a RAM disk (`FolderMask`) to prevent accidental dynamic linking.
50+
- External modules compile to a shared `build/deps/` prefix; CPython finds them via CPATH/LDFLAGS.
51+
- Telltale detection: modules check for marker files (`m_telltale`) to determine if system already has the library.
52+
- No patches to upstream CPython source — relies solely on configure flags.
53+
54+
**runez** is the foundational utility library (file ops, system info, CLI decorators, logging, Version/PythonSpec types). Check runez before reimplementing anything.
55+
56+
**Additional pointers:**
57+
58+
- `ModuleCollection.selected` contains only the modules chosen for a build — not all candidates.
59+
- Build logs go to `build/logs/NN-modulename.log` (e.g. `01-openssl.log`, `02-cpython.log`).
60+
- YAML config supports platform-specific overrides and path templates — see CONFIGURATION.md.
61+
- See ARCHITECTURE.md for class hierarchy and design patterns, DEVELOP.md for common tasks and dependencies.
62+
63+
## Testing
64+
65+
- pytest with 100% code coverage target
66+
- Tests mock `runez.run()` to avoid actual compilation — uses `--dryrun` mode
67+
- `conftest.py` provides a `cli` fixture (from runez) and forbids HTTP calls (`GlobalHttpCalls.forbid()`)
68+
- Sample YAML configs in `tests/sample-config*.yml` for testing configuration parsing
69+
70+
## Linting
71+
72+
Ruff handles all linting and formatting. Key settings in pyproject.toml:
73+
- Line length: 140
74+
- McCabe complexity: max 18
75+
- Security checks (S rules) disabled in tests
76+
- Numpy-style docstrings

‎CONFIGURATION.md‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# Configuration (portable-python.yml)
2+
3+
portable-python is configured via a YAML file (default: `portable-python.yml` in current directory, override with `--config`).
4+
5+
6+
## Key Sections
7+
8+
### folders
9+
10+
- `build`, `dist`, `sources`, `logs`, `destdir`, `ppp-marker`
11+
- All support path templates: `{build}`, `{version}`, `{abi_suffix}`
12+
13+
### cpython-modules (CSV)
14+
15+
Which external modules to auto-select.
16+
Default is configured in `DEFAULT_CONFIG` (see `src/portable_python/config.py`).
17+
Examples: openssl, zlib, xz, sqlite, bzip2, gdbm, libffi, readline, uuid
18+
19+
### cpython-configure (list)
20+
21+
Extra `./configure` args for CPython.
22+
Default includes: `--enable-optimizations`, `--with-lto`, `--with-ensurepip=upgrade`
23+
24+
### cpython-clean-1st-pass (list)
25+
26+
Files to remove before `compileall` — removes test files, idle, 2to3 (~94 MB savings).
27+
28+
### cpython-clean-2nd-pass (list)
29+
30+
Files to remove after `compileall` — removes pycaches for seldom-used libs (~1.8 MB savings).
31+
32+
33+
## Per-module config
34+
35+
Each external module can be customized with these keys (replace `{module}` with the module name, e.g. `openssl`):
36+
37+
| Key | Purpose |
38+
|-----|---------|
39+
| `{module}-version` | Version to use |
40+
| `{module}-url` | URL to download from |
41+
| `{module}-src-suffix` | File extension if not in URL |
42+
| `{module}-configure` | Custom configure args |
43+
| `{module}-http-headers` | HTTP headers for download |
44+
| `{module}-patches` | File patches to apply |
45+
| `{module}-debian` | Package name on Debian (for dependency detection) |
46+
47+
48+
## Platform-specific overrides
49+
50+
Configuration supports platform-specific sections (e.g. `windows.ext`, `macos.env`, etc.).
51+
The most specific setting wins.
52+
53+
`MACOSX_DEPLOYMENT_TARGET` defaults to 13 (Ventura).

‎DEVELOP.md‎

Lines changed: 46 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,7 @@
33
Create a dev venv:
44

55
```shell
6-
uv venv
7-
uv pip install -r requirements.txt -r tests/requirements.txt
8-
uv pip install -e .
6+
uv sync
97
```
108

119
You can then run `portable-python` from that venv:
@@ -66,3 +64,48 @@ Now inside docker, you run a build:
6664
```shell
6765
portable-python build 3.13.2
6866
```
67+
68+
69+
# Key Dependencies
70+
71+
| Package | Version | Purpose |
72+
|---------|---------|---------|
73+
| click | <9 | CLI framework |
74+
| pyyaml | <7 | Configuration parsing |
75+
| requests | <3 | HTTP downloads |
76+
| runez | <6 | Utilities (file ops, system, logging) |
77+
| urllib3 | <3 | HTTP transport |
78+
| pytest-cov | - | Coverage reporting (dev only) |
79+
80+
**runez** is central: provides file ops (`ls_dir`, `touch`, `compress`/`decompress`), system info (platform detection), CLI decorators (`@click.group`), logging, and version handling (`Version`, `PythonSpec`).
81+
82+
83+
# Common Tasks
84+
85+
## Add a New External Module
86+
87+
1. Create class in `src/portable_python/external/xcpython.py` extending `ModuleBuilder`
88+
2. Set `m_name`, `m_telltale`, `m_debian`, `version` property
89+
3. Implement `url` property (or override `_do_linux_compile()` / `_do_macos_compile()`)
90+
4. Add to `Cpython.candidate_modules()` if it's a CPython sub-module
91+
5. Add tests in `tests/test_setup.py`
92+
93+
## Add a New Config Option
94+
95+
1. Update `DEFAULT_CONFIG` in `src/portable_python/config.py`
96+
2. Use `PPG.config.get_value("key")` to retrieve it in code
97+
3. Add tests to `tests/test_setup.py`
98+
99+
## Fix a Portability Issue
100+
101+
1. Run `portable-python inspect <PATH>` to diagnose
102+
2. If lib is being dynamically linked, add to module list or update isolation
103+
3. Use `LibAutoCorrect.run()` logic (or extend it) to fix paths
104+
4. Add test case to `tests/test_inspector.py`
105+
106+
## Bump Python Support
107+
108+
1. Update `pyproject.toml` classifiers and `requires-python`
109+
2. Update `.github/workflows/tests.yml` matrix
110+
3. Update `CPythonFamily.min_version` if needed
111+
4. Run full test matrix with `tox`

‎MANIFEST.in‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,2 @@
11
include LICENSE
22
include README.rst
3-
include DEVELOP.md
4-
include SECURITY.md
5-
include requirements.txt

‎README.rst‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,64 @@ Note that you can use ``--dryrun`` mode to inspect what would be done without do
128128
Would tar build/3.9.7 -> dist/cpython-3.9.7-macos-x86_64.tar.gz
129129

130130

131+
CLI reference
132+
-------------
133+
134+
Main entry point::
135+
136+
portable-python [OPTIONS] COMMAND [ARGS]
137+
138+
**Global options**:
139+
140+
- ``--config PATH``: Config file (default: ``portable-python.yml``)
141+
- ``--quiet``: Turn off DEBUG logging
142+
- ``--dryrun`` / ``-n``: Show what would be done
143+
- ``--target PLATFORM``: Override detected platform (for testing)
144+
145+
146+
**build** ``<PYTHON_SPEC>`` - Build a portable Python binary::
147+
148+
portable-python build 3.13.2 -m openssl,zlib
149+
150+
- ``--modules, -m CSV``: External modules to include
151+
- ``--prefix, -p PATH``: Use ``--prefix`` (non-portable)
152+
153+
154+
**build-report** ``[PYTHON_SPEC]`` - Show module status and what will be compiled:
155+
156+
- ``--modules, -m CSV``: Specific modules to check
157+
- Validates that modules can be built
158+
159+
160+
**inspect** ``<PATH>`` - Check if a Python installation is portable::
161+
162+
portable-python inspect /usr/bin/python3
163+
164+
- ``--modules, -m MODULES``: Which modules to inspect
165+
- ``--verbose, -v``: Show full ``.so`` report
166+
- ``--prefix, -p``: Built with ``--prefix`` (not portable)
167+
- ``--skip-so, -s``: Don't check all ``.so`` files
168+
169+
170+
**list** ``[FAMILY]`` - List available versions (default: cpython)::
171+
172+
portable-python list cpython
173+
174+
- ``--json``: Output as JSON
175+
176+
177+
**diagnostics** - Show system diagnostics
178+
179+
180+
**recompress** ``<PATH> <EXT>`` - Re-compress existing binary tarball (for comparing compression sizes)
181+
182+
183+
**lib-auto-correct** ``<PATH>`` - Auto-correct exes/libs to use relative paths:
184+
185+
- ``--commit``: Actually perform changes (dryrun by default)
186+
- ``--prefix, -p PATH``: Expected ``--prefix`` from build
187+
188+
131189
Library
132190
-------
133191

0 commit comments

Comments
 (0)