Skip to content

LLMClient: OLLAMA_HOST/OPENAI_BASE_URL ignored, keyless local servers unusable, explicit model can be overridden #184

Closed

Description

@srpatcha

ebuild/eos_ai/llm_integration.py on master:

  • ignores OLLAMA_HOST (a schemeless host:port, as Ollama documents it), OPENAI_BASE_URL and the model env vars;
  • is_available() requires an API key for custom, so a keyless local OpenAI-compatible server (vLLM, llama.cpp) is never used, and auto() needs EOS_LLM_API_KEY as well as EOS_LLM_URL;
  • sends Authorization: Bearer with an empty key;
  • treats a response with no choices as an empty success instead of a failure;
  • model defaults to llama3 for every provider, so LLMClient(provider='openai') asks OpenAI for llama3.

PR #127 addresses all of these.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions