What
The inference HTTP server has:
- No authentication
- No rate limiting
- No request validation middleware
Why
The inference engine exposes 10 API endpoints over HTTP. Without auth, any process on the machine (or network, if bound to 0.0.0.0) can submit prompts and pull models, potentially exhausting resources.
How
- Add a simple JSON body schema validation middleware
- Add optional token-based auth (configurable via env var)
- Add per-IP rate limiting with configurable requests/minute
What
The inference HTTP server has:
Why
The inference engine exposes 10 API endpoints over HTTP. Without auth, any process on the machine (or network, if bound to 0.0.0.0) can submit prompts and pull models, potentially exhausting resources.
How