Skip to content

PythonReminingPlan.md

Latest commit

 

History

History
168 lines (120 loc) · 7.22 KB

PythonReminingPlan.md

File metadata and controls

168 lines (120 loc) · 7.22 KB
Raw

Python/FastAPI Re-mining Plan

Motivation

The original Python mining used mutual dependents (fastapi+bcrypt, fastapi+pyjwt, fastapi+redis, fastapi+itsdangerous) to pre-filter repos per pattern before cloning. This caused:

  1. Missed repos: Repos using alternative libraries (e.g., python-jose instead of pyjwt, argon2-cffi instead of bcrypt) were excluded from the pool entirely.
  2. Inconsistency with Java approach: For Java/Spring, we cloned ALL spring-boot-starter-security dependents (649), applied a primary filter (import org.springframework.security), then grepped for pattern-specific keywords independently. This captured implementations regardless of which library they used.
  3. Small filtered pool: Only 147 unique repos passed the from fastapi.security import filter — out of a potential 6,452 fastapi dependents.

New Approach

Mirror the Java approach exactly:

  1. Clone ALL fastapi dependents (6,452 repos) in batches
  2. Primary filter: from fastapi.security import in any .py file
  3. Pattern-specific grep: search the filtered repos for each of the 5 KU Leuven patterns using keyword signatures derived from the catalogue
  4. Save all commands in a reproducible scripts/python_pattern_grep.sh

Step 1: Batched Cloning

Source

Existing JSONL: build/volumes/data/dependent_repos_info/python_Pypi_fastapi_dependents_1_cleaned.jsonl (6,452 repos)

Clone directory

build/volumes/data/cloned_repos/ (same location as original, now empty)

Batch strategy

The Rust cloner supports --start-index and --end-index. We clone in batches of ~1,000 repos to manage disk space:

Batch Range Command
1 0–999 crawl --start-index 0 --end-index 1000
2 1000–1999 crawl --start-index 1000 --end-index 2000
3 2000–2999 crawl --start-index 2000 --end-index 3000
4 3000–3999 crawl --start-index 3000 --end-index 4000
5 4000–4999 crawl --start-index 4000 --end-index 5000
6 5000–5999 crawl --start-index 5000 --end-index 6000
7 6000–6452 crawl --start-index 6000 --end-index 6452

After each batch

  1. Run primary filter on the new repos
  2. Append passing repos to build/volumes/data/grep_results/00_primary_filter_fastapi_security.txt
  3. Run scripts/remove_filtered_repos.sh python to delete repos that don't pass the primary filter
  4. Check df -h / — proceed to next batch only if >= 10GB free

Full clone command template

LIBRARIES_IO_API_KEY=677d27c0d1367a899bac03f0ffc316c5 \
./security_pattern_miner_rs/target/release/security-pattern-miner crawl \
  --input-jsonl ./build/volumes/data/dependent_repos_info/python_Pypi_fastapi_dependents_1_cleaned.jsonl \
  --clone-dir ./build/volumes/data/cloned_repos \
  --start-index START --end-index END \
  --max-concurrent-clones 4

Step 2: Primary Filter

grep -rl "from fastapi.security" --include="*.py" REPO_DIR

This is the Python equivalent of Java's import org.springframework.security. It identifies repos that explicitly use FastAPI's built-in security utilities (OAuth2PasswordBearer, APIKeyHeader, HTTPBearer, SecurityScopes, etc.).

Expected yield

From the original mining, ~132 out of 399 cloned repos (33%) passed this filter. With 6,452 total dependents, we might expect 5,000+ successful clones and ~1,500–2,000 passing the primary filter (optimistic estimate — many fastapi repos are pure REST APIs without security).

Step 3: Pattern-Specific Grep

All grep commands will be saved in scripts/python_pattern_grep.sh.

P1: Password-based authentication

Catalogue roles mapped to Python/FastAPI:

  • Hasher: bcrypt, argon2, passlib, hashlib, pbkdf2
  • Password store: database ORMs storing password hashes
  • Comparator: verify, checkpw, hash comparison
  • Verification manager: login endpoint logic
  • Credential: username + password form data
grep -rqlE "bcrypt|argon2|passlib|Pbkdf2|CryptContext|pwd_context|hash_password|verify_password|checkpw|password_hash|PasswordHasher" --include="*.py"

P2: Verifiable token-based authentication

Catalogue roles mapped to Python/FastAPI:

  • Verifier + Cryptographer: JWT decode/verify
  • Key manager: secret key management
  • Token generator: JWT encode/sign
grep -rqlE "import jwt|from jwt |from jose |from authlib|JWTError|jwt\.encode|jwt\.decode|JWTBearer|OAuth2PasswordBearer|access_token|create.*token|verify.*token" --include="*.py"

P3: Opaque token-based authentication

Catalogue roles mapped to Python/FastAPI:

  • Principal provider: server-side session/token store (Redis, DB)
  • Token generator: secrets.token_urlsafe, uuid4
  • Verifier: session lookup
grep -rqlE "SessionMiddleware|session\[|SessionBackend|RedisBackend|aioredis|session_id|session_token|InMemoryBackend|itsdangerous|URLSafeTimedSerializer|starlette\.middleware\.sessions" --include="*.py"

P4: Obscure token-based access control

Catalogue roles mapped to Python/FastAPI:

  • Validator: combined authn + authz on API key
  • Hasher: hash API key for storage
  • Token manager: API key CRUD + privilege mapping
grep -rqlE "APIKeyHeader|APIKeyQuery|APIKeyCookie|APIKey|api_key|X-API-Key|x-api-key|PersonalAccessToken" --include="*.py"

P5: Session-based access control

Catalogue roles mapped to Python/FastAPI:

  • Authentication enforcer: session verification
  • Authorisation enforcer + Decider: permission/role checks
  • Policy provider: RBAC/ABAC policy

Requires BOTH session AND authorization keywords (intersection):

# Session keywords:
grep -rqlE "SessionMiddleware|session\[|SessionBackend|HTTPBearer|OAuth2PasswordBearer|Depends.*current_user|get_current_user" --include="*.py"
# Authorization keywords:
grep -rqlE "permission|role|rbac|authorize|access_control|policy|privilege|scope|SecurityScopes|is_admin|is_superuser|has_permission|check_permission" --include="*.py"

Step 4: Cross-pattern Analysis & Documentation

  • Compute overlap matrix (repos matching 1, 2, ..., 5 patterns)
  • Update MiningLog.md with the new Python re-mining section
  • Compare with Java results

Disk Budget

Item Estimated Size
Available disk ~65 GB
6,452 repos cloned (estimate) ~60–80 GB (before pruning)
After primary filter pruning ~15–25 GB (keeping ~30% of repos)
Safety margin 10 GB

The batch-and-prune approach keeps disk usage manageable: clone 1,000, prune ~700 that fail the filter, clone the next 1,000.

Output Files

File Description
build/volumes/data/grep_results/00_primary_filter_fastapi_security.txt Repos passing primary filter
build/volumes/data/grep_results/p1_password_based_authentication.txt P1 repos
build/volumes/data/grep_results/p2_verifiable_token_authentication.txt P2 repos
build/volumes/data/grep_results/p3_opaque_token_authentication.txt P3 repos
build/volumes/data/grep_results/p4_obscure_token_access_control.txt P4 repos
build/volumes/data/grep_results/p5_session_based_access_control.txt P5 repos
build/volumes/data/grep_results/overlap_all5.txt Repos with all 5 patterns
scripts/python_pattern_grep.sh Reproducible grep script