Refactor: dead code, single CRUD path, polymorphism, single-responsibility, no global state

.cursorrules

@@ -14,40 +14,47 @@ Pydantic schema. No models.py, schemas.py, Column(), or @validates in user code.
 - Pydantic v2 — Field(), create_model(), model_validate(), model_dump()
 - Starlette — Request, JSONResponse, Response, Router, Route
 - Uvicorn — ASGI server
-- PyJWT — JWT auth (JWTAuthentication is FROZEN — do not modify)
-- Redis — caching via frozen RedisCache/cache_manager
+- PyJWT: JWT auth
+- Redis: caching via RedisCache/cache_manager
 - PyYAML — YAML config loading
 - pytest + httpx + SQLite — testing only
 
 ## Module Map
 | File | Role |
 |------|------|
-| lightapi/exceptions.py | ConfigurationError, SerializationError |
-| lightapi/methods.py    | HttpMethod.GET/POST/PUT/PATCH/DELETE markers |
-| lightapi/config.py     | Authentication, Filtering, Pagination, Serializer, Cache |
+| lightapi/__init__.py   | Public API re-exports (`__all__`) |
+| lightapi/lightapi.py   | LightApi: register(), build_app(), run(), from_config(), from_dict() |
+| lightapi/rest.py       | RestEndpointMeta, RestEndpoint auto-CRUD (sync and async share one core) |
+| lightapi/endpoint_handler.py | EndpointHandler (one per route), EndpointGuard, MiddlewareChain, verb tables |
+| lightapi/repository.py | Repository: row persistence, timestamps, optimistic locking |
+| lightapi/schema.py     | SchemaFactory (pydantic schemas, row_to_dict, patch_schema), FieldProjection |
+| lightapi/table_mapping.py | TableSource: DeclaredTable (columns from fields), ReflectedTable |
+| lightapi/session_manager.py | SessionManager (engine, metadata, registry, run_blocking) |
+| lightapi/session.py    | get_sync_session, get_async_session (public) |
+| lightapi/config.py     | Config (JWT settings), Authentication, Filtering, Pagination, Cache, Serializer |
+| lightapi/constants.py  | Enums: HttpVerb, ExecutionMode, TableReflection, JwtAlgorithm, ApiError, ResponseKey |
+| lightapi/methods.py    | HttpMethod.GET/POST/PUT/PATCH/DELETE marker mixins (public) |
 | lightapi/fields.py     | lightapi.Field() wrapper (strips foreign_key/unique/index/exclude) |
-| lightapi/schema.py     | SchemaFactory, _row_to_dict, _apply_fields, resolve_fields |
-| lightapi/rest.py       | RestEndpointMeta, RestEndpoint auto-CRUD |
-| lightapi/auth.py       | AllowAny, IsAuthenticated, IsAdminUser (JWTAuthentication FROZEN) |
-| lightapi/filters.py    | FilterBackend, SearchFilter, OrderingFilter |
-| lightapi/pagination.py | PageNumberPaginator, CursorPaginator |
-| lightapi/middleware.py | Middleware base |
-| lightapi/__init__.py   | LightApi, @app.route, @app.middleware, YAML, select shim |
-
-## FROZEN — DO NOT MODIFY
-- lightapi/auth.py: JWTAuthentication, BaseAuthentication (class bodies)
-- lightapi/cache.py: RedisCache, BaseCache
-- lightapi/swagger.py: SwaggerGenerator
-- lightapi/lightapi.py: preserved as legacy until __init__.py rewrite absorbs it
-- .github/workflows/
-- pyproject.toml metadata fields (name, version, description, authors, etc.)
+| lightapi/authentication/ | BaseAuthentication, JWTAuthentication, BasicAuthentication, permissions |
+| lightapi/auth.py       | Compatibility re-exports of lightapi.authentication |
+| lightapi/_login.py     | LoginRequest, LoginEndpoint (/auth/login, /auth/token) |
+| lightapi/rate_limiter.py | RateLimiter, RateLimitWindow |
+| lightapi/cache.py      | RedisCacheBackend, EndpointCache (per route), legacy BaseCache/RedisCache |
+| lightapi/filters.py    | BaseFilter, FieldFilter, SearchFilter, OrderingFilter |
+| lightapi/pagination.py | PaginationStyle, Cursor, NoPagination, PageNumberPaginator, CursorPaginator |
+| lightapi/yaml_config.py | Pydantic models of lightapi.yaml and how they build the app (LightAPIConfig) |
+| lightapi/_dict_config_loader.py | DictConfig behind the deprecated LightApi.from_dict |
+| lightapi/core.py       | Middleware, CORSMiddleware, AuthenticationMiddleware, Response (compat) |
+| lightapi/health.py     | /healthz |
+| lightapi/server.py     | `lightapi serve` command |
+| lightapi/exceptions.py | ConfigurationError, SerializationError |
 
 ## Key Invariants (enforce in all generated code)
 1. RestEndpoint IS the SQLAlchemy model AND the Pydantic schema.
 2. Fields = annotated class attrs with Field(). Column() never appears in user code.
 3. queryset is the only name for static and dynamic query definitions.
 4. All validation via Pydantic Field(). @validates must not exist.
-5. Every DB row passes through _row_to_dict → _apply_fields → model_validate → model_dump.
+5. Every DB row passes through SchemaFactory.row_to_dict → FieldProjection.apply → model_validate → model_dump.
 6. IsAdminUser checks payload["is_admin"] == True.
 7. version (int, default 1) is auto-injected; PUT/PATCH require it; 409 on mismatch.
 8. GET list always returns 200 with {"results": []} on empty; never 404.

.gitignore

@@ -89,6 +89,7 @@ celerybeat.pid
 *.sage.py
 .env
 .venv
+.venv310/
 env/
 venv/
 ENV/
@@ -108,3 +109,5 @@ cython_debug/
 .idea
 alembic
 *.db
+
+tools/e2e/.out/

CHANGELOG.md

@@ -7,6 +7,184 @@ Versions align with [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ---
 
+## [Unreleased]
+
+### Deprecated
+- **`LightApi.from_dict`** emits a `DeprecationWarning` and is removed in 0.2.0. Use
+  `RestEndpoint` subclasses with `register()`, or `from_config()` with a YAML file.
+
+### Changed
+- **The authentication backend is built once per route**, on the first request that needs
+  it, instead of once per request. A backend subclass with its own `__init__` is therefore
+  instantiated once. JWT settings and the secret are still read on every request.
+- **A permission class decides whether login is required**: `BasePermission` has
+  `requires_authentication = True` and `AllowAny` sets it to `False`. The only visible
+  difference: a subclass of `AllowAny` used as a permission no longer forces authentication
+  (before, only `AllowAny` itself, by identity, made a method public).
+- **`/auth/login` with a body that is not valid UTF-8 returns 422** (it was a 500); a body
+  that is not JSON still returns the same 422.
+- **`Meta.reflect` rejects unknown values**: anything other than `True`, `False`, `"full"`
+  or `"partial"` now raises `ConfigurationError` when the class is defined. Before, a value
+  such as `"yes"` was silently treated as "not reflected" and produced a table with only the
+  automatic columns.
+- **Sync CRUD runs in a worker thread**: with a sync `Engine`, `list`, `retrieve`,
+  `create`, `update`, `destroy` and the response-cache calls ran on the event loop
+  thread, so one slow query held up every other request on that worker. They now
+  go through `starlette.concurrency.run_in_threadpool`. Engines on `StaticPool` or
+  `SingletonThreadPool` (in-memory SQLite) still run in place, because those pools
+  cannot be shared across threads.
+
+### Removed (internal, no public name affected)
+- `RestEndpoint._meta["reflect"]`: the key was written but never read; `Meta.reflect` is
+  read from the class itself.
+- `lightapi.constants.HTTPStatus`: a copy of eleven members of the standard library's
+  `http.HTTPStatus`, which the package now uses directly. Status codes in responses are
+  unchanged.
+- `lightapi.auth_checker` and `lightapi.middleware_runner`: `EndpointGuard` and the
+  middleware runner (now `MiddlewareChain`) live in `lightapi.endpoint_handler`, the only
+  module that used them.
+- `lightapi.cache_helper` and the module functions `lightapi.cache.get_cached`,
+  `set_cached` and `_ping_redis`: the response cache is `lightapi.cache.EndpointCache`
+  (built once per route) over `RedisCacheBackend.shared()`, the process-wide client that is
+  still created on first use. `lightapi.cache.invalidate_cache_prefix` stays.
+- `lightapi.yaml_loader`, `lightapi.yaml_names` and `lightapi.yaml_schema`: the YAML
+  document's models and the code that turns them into endpoints now live together in
+  `lightapi.yaml_config` (`LightAPIConfig.from_file(...).build_app(...)`,
+  `LightAPIConfig.resolve_name`). The YAML format is unchanged.
+- `lightapi._dict_config_loader.load_from_dict`: `LightApi.from_dict` (deprecated) now goes
+  through `DictConfig` in the same module.
+
+### Fixed
+- **`Meta.cache` was ignored on async engines**: cached GETs and cache invalidation
+  only ran on the sync path, so an app on an `AsyncEngine` never read or wrote the
+  cache. Both engine kinds now use it. The Redis calls run in a worker thread.
+- **Sync verb overrides were ignored**: a plain `def get(self, request)` (or `post`,
+  `put`, `patch`, `delete`) on an endpoint was skipped and the built-in CRUD ran
+  instead; only `async def` overrides were served. Both are served now, on sync and
+  async engines. A column that happens to be called `post` or `get` is still a
+  column, not an override.
+- **`/auth/login` ignored a backend's `validate_credentials`**: the login route
+  always validated through a bare `JWTAuthentication()`, so overriding
+  `validate_credentials()` on a subclass (the README's "custom authentication")
+  never ran and login returned `401`. The route now uses the backend the endpoint
+  configured, sync or `async def`. An app-level `login_validator` still wins.
+- **Auth settings were dropped for backend subclasses**: `jwt_expiration`,
+  `jwt_algorithm` and `jwt_extra_claims` only reached a backend whose class was named
+  exactly `JWTAuthentication`. Subclasses get them too, unless they define their own
+  `__init__`.
+- **`LIGHTAPI_REDIS_URL` was read at import time**: the Redis client behind
+  `Meta.cache` was built when `lightapi` was imported, so a URL set afterwards (in
+  code, or by a test) was ignored. It is built on the first cache call now.
+- **Redis failures are logged**: a cache read, write, invalidation or ping that fails
+  still falls back to the database as before, but now leaves a warning with the
+  traceback (`lightapi.cache`) instead of nothing. A `login_validator` that raises still
+  answers `401`, and the exception is logged with its traceback (`lightapi._login`), as is
+  a failed `CREATE TABLE` at `build_app()`.
+- **The Redis client has a read timeout**: `socket_timeout=1` next to the existing
+  `socket_connect_timeout=1`, so a Redis that accepts the connection and hangs costs one
+  second and a cache miss, not a stuck worker.
+- **Verb overrides and CRUD functions are resolved once, at `register()`**: a `get`/`post`
+  method assigned to the endpoint class after `register()` is no longer picked up per
+  request. Middleware classes are checked for `async def process` once per class.
+- **`Pagination(style=...)` validates without rewriting the frozen field**; an invalid style
+  raises `ConfigurationError` as before, and `Pagination.style` keeps the value as given
+  (`PaginationStyle` member or its string).
+- **`Authentication().build_backend()` without a backend raises `ConfigurationError`**
+  instead of `AttributeError`; a CRUD call on an endpoint class that was never passed to
+  `register()` raises `RuntimeError` naming the missing call instead of SQLAlchemy's
+  `ArgumentError`.
+- **Async apps create their tables in a Starlette lifespan** instead of the deprecated
+  `on_startup` handler; same moment, no deprecation warning. `created_at`/`updated_at`
+  column defaults no longer use the deprecated `datetime.utcnow()`; the stored value is the
+  same naive UTC.
+- **The container HEALTHCHECK probes `/healthz` over a raw socket**: it hit `/`, which is a
+  404 on every LightAPI app, so `docker inspect` reported every healthy container as
+  unhealthy. The probe also stops importing `urllib.request`, which alone took 1.8 to 2.9 s
+  inside the image on Docker Desktop and blew the 3 s timeout; the socket probe takes 0.7 s.
+- **PATCH validates every field constraint**: the PATCH schema rebuilt each field as an
+  optional copy and dropped its `Field()` constraints, so `min_length`, `ge`,
+  `decimal_places` and the rest applied to POST and PUT but not to PATCH (`{"name": ""}`
+  against `min_length=3` was a 200). A PATCH now meets the same constraints as a PUT and
+  answers 422 like it; only the "every field is optional" part is unchanged.
+- **`decimal_places` on a non-Decimal field is refused when the class is defined**: a
+  `ConfigurationError` names the field. Before this release the option was silently
+  ignored on `float` or `int`; once forwarded to pydantic it would have made every write a
+  500 (`TypeError: Unable to apply constraint`). Same through YAML.
+- **`Field(decimal_places=N)` is enforced**: a `Decimal` with more than N decimal places
+  is a 422. Before, the option reached neither pydantic nor the DDL (`Numeric(scale=N)`
+  without a precision renders as plain `NUMERIC` on every dialect), so "1.999" was stored
+  as sent on a `decimal_places=2` column. The column type is unchanged.
+- **A database constraint violation is a 409**: a duplicate on a `unique=True` column, a
+  DELETE of a row another table still references, or any other `IntegrityError` on POST,
+  PUT, PATCH or DELETE answers `409 {"detail": "constraint violation"}` and leaves the
+  session usable. It was an unhandled 500. The server logs the table and the driver's
+  text at INFO, because a NOT NULL or CHECK failure usually means the schema and the
+  table disagree.
+- **Error messages never echo the database password**: the `ConfigurationError` for a
+  URL without an async driver renders the URL with the password masked, and the YAML
+  load error lists each problem as location and message only, so a typo like `databse:`
+  no longer prints the block it rejected (which held the URL, or `password:`) into the
+  logs.
+- **A YAML key the format does not know fails at load time**: `defaults.filtering`,
+  `meta.paginate` or a misspelt top-level key now raise `ConfigurationError` naming the
+  key, as the configuration guide promises. Before, pydantic dropped the key and the app
+  started without the feature. A key under `fields.<name>` that is not a constraint
+  (`min_lenght: 3`) is refused the same way; before, the constraint silently vanished.
+- **Async mode is detected from the engine or the URL**: `LightApi(engine=AsyncEngine)`
+  and `LightApi(database_url="sqlite+aiosqlite://...")` (or a YAML document with such a
+  URL) run async without `mode="async"`, as the README always said. Before, the sync
+  default never created the tables and every request was a 500 (with a "Table creation
+  skipped" warning); the docs' own async testing fixture hit it verbatim. An explicit
+  `mode="sync"` with an async engine or driver is a `ConfigurationError`.
+- **An `async def validate_credentials` on a Basic backend is refused at `register()`**:
+  the request guard calls it without `await`, and a coroutine is truthy, so every Basic
+  credential used to pass. The error names the backend; `JWTAuthentication` subclasses
+  may keep an async validator (only the login route calls it, and that one awaits).
+- **An integer filter value beyond 64 bits is a 422** (`?pages=99999999999999999999`),
+  like any other value the column cannot hold. It was an `OverflowError` 500 on SQLite.
+- **Async reflection no longer leaves a connection bound to a dead loop in the pool**:
+  `Meta.reflect` on an `AsyncEngine` reflects on a loop of its own at `register()` and now
+  disposes the pool afterwards (not for an in-memory SQLite `StaticPool`, which holds the
+  database), so asyncpg's first request no longer finds a connection from another loop.
+- **`mode="async"` works from a database URL**: `LightApi(database_url=..., mode="async")`
+  and a YAML document with `mode: async` build an `AsyncEngine`, so `lightapi serve` can
+  start an async app. Before, both failed at `register()` with "requires AsyncEngine". A
+  URL without an async driver now raises `ConfigurationError` naming the drivers that work.
+- **Reflected tables accept what the table and the endpoint allow to be absent**: a
+  nullable column of a reflected table is optional on POST (stored as null), and on
+  `reflect = "partial"` a declared `Field(default=...)` or `Field(min_length=...)` is
+  honoured. Before, every non-auto column was required on POST, and the declared
+  `Field()` was dropped once the class was mapped, so `notes: str = Field(default="")`
+  still answered 422 "Field required".
+- **A filter value the column cannot parse is a 422**: `?pages=lots` on an integer
+  column answers `422 {"detail": [{"loc": ["query", "pages"], ...}]}`. Before, the raw
+  string reached the database: an empty list on SQLite, a 500 (`invalid input syntax for
+  type integer`) on Postgres. `lightapi.filters._coerce_filter_value` raises
+  `FilterValueError` for such a value; custom backends that call it get the same.
+- **A protected endpoint's 401 is the backend's own reply**: the guard answered every
+  failed authentication with `{"detail": "Authentication required"}` and
+  `WWW-Authenticate: Bearer`, whatever the backend. It now returns the backend's
+  `get_auth_error_response(request)`, so a JWT endpoint answers the documented
+  `{"detail": "Authentication credentials invalid."}` with `Bearer`, a Basic endpoint
+  answers `{"error": "authentication failed"}` with `Basic realm="Restricted Area"` (the
+  challenge a browser needs to prompt), and a backend subclass that overrides the hook is
+  finally heard. Status 401 unchanged.
+- **A method the endpoint does not allow answers the documented JSON 405**:
+  `{"detail": "Method Not Allowed. Allowed: GET"}` with the `Allow` header, on the CRUD
+  routes and on `/auth/login`. Before, Starlette answered with its `text/plain` "Method
+  Not Allowed" (the JSON reply existed in the code but only `HEAD` ever reached it). The
+  routes still list only the verbs they serve, so two classes may share a path split by
+  method, and a disallowed verb is refused before the authentication guard, the
+  middlewares and the login rate limiter see it. `/auth/login` answers the same body as
+  the CRUD routes; its former `{"detail": "method not allowed"}` is gone. Status and
+  `Allow` are unchanged.
+- **An async app on asyncpg no longer fails its first request after startup** ("Future
+  attached to a different loop"). The startup created the tables on a worker thread with
+  its own event loop, and the connection that did it went back to the pool bound to that
+  loop. The lifespan now runs the DDL on the loop that serves the requests. `build_app()`
+  called from inside an already running loop (a pytest-asyncio test) still uses the worker
+  thread; create the tables yourself in that case.
+
 ## [Unreleased] — 0.1.24
 
 ### Fixed

Dockerfile

@@ -54,12 +54,15 @@ USER lightapi
 
 EXPOSE 8000
 
-# Lightweight health check — touches the root path expecting any HTTP response.
+# Health check against the liveness route the app always registers. A raw
+# socket, not urllib: importing urllib.request alone took 1.8 to 2.9 s inside
+# this image on Docker Desktop, which blew the 3 s timeout; the socket probe
+# measures 0.7 s end to end there.
 HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
-    CMD python -c "import urllib.request, sys; \
-import os; \
-url='http://127.0.0.1:'+os.environ.get('LIGHTAPI_PORT','8000')+'/'; \
-sys.exit(0 if urllib.request.urlopen(url, timeout=2).status else 1)" \
+    CMD python -c "import socket, os; \
+s = socket.create_connection(('127.0.0.1', int(os.environ.get('LIGHTAPI_PORT', '8000'))), timeout=2); \
+s.sendall(b'GET /healthz HTTP/1.0\r\nHost: localhost\r\n\r\n'); \
+raise SystemExit(0 if b' 200 ' in s.recv(64) else 1)" \
     || exit 1
 
 ENTRYPOINT ["python", "/app/entrypoint.py"]