Skip to content

Add request validation and rate limiting #14

Description

@jeanmachuca

What

The inference HTTP server has:

  • No authentication
  • No rate limiting
  • No request validation middleware

Why

The inference engine exposes 10 API endpoints over HTTP. Without auth, any process on the machine (or network, if bound to 0.0.0.0) can submit prompts and pull models, potentially exhausting resources.

How

  • Add a simple JSON body schema validation middleware
  • Add optional token-based auth (configurable via env var)
  • Add per-IP rate limiting with configurable requests/minute

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions