Skip to content

Autocomplete silently produces nothing when Ollama Modelfile num_ctx equals Continue's default maxTokens (4096) — pruneLength computes a negative prompt budget and prunes prefix/suffix to empty #13038

Open

Description

@japric
No description provided.

Activity

  1. japric commented on Jul 27, 2026

    @japric
    Author

    Before submitting your bug report

    Relevant environment info

    • OS: Ubuntu 22.04
    • Continue version: 1.2.1
    • IDE version: VSCode 1.103.2
    • Model: qwen2.5-coder:1.5b (Ollama 0.13.5, local)
    • GPU: NVIDIA MX250 (2 GB VRAM) — relevant only because it motivated me to pin num_ctx to 4096 to keep the model fully in VRAM
    • config (~/.continue/config.yaml):
    name: Local Agent
    version: 1.0.0
    schema: v1
    models:
      - name: Qwen 2.5 Coder 1.5b AllGPU
        provider: ollama
        model: qwen2.5_coder-1.5b:allgpu
        roles:
          - autocomplete
          - chat
        # NOTE: no explicit contextLength / maxTokens here — both fall back to defaults
      - name: Autodetect
        provider: ollama
        model: AUTODETECT
        roles:
          - autocomplete
          - chat
          - edit
          - apply
    context:
      - provider: code
      - provider: docs
      - provider: diff
      - provider: terminal
      - provider: problems
      - provider: folder
      - provider: codebase

    The Ollama Modelfile for qwen2.5_coder-1.5b:allgpu contains:

    FROM <…qwen2.5-coder-1.5b-instruct-q4_k_m.gguf…>
    PARAMETER num_ctx 4096
    PARAMETER num_gpu 32
    

    For comparison, the qwen2.5_coder-1.5b:latest tag has no num_ctx line in its Modelfile (everything else identical, same weight blob).

    Description

    Tab autocomplete silently produces nothing — no completion item, no error message, no log line — when the Ollama Modelfile for the selected autocomplete model contains PARAMETER num_ctx 4096 and the user has not explicitly overridden maxTokens in config.yaml. The same model tag without the num_ctx line (:latest) autocomplete works fine.

    Direct curl to http://localhost:11434/api/generate with identical prompts returns correct completions for both tags, so the Ollama side is healthy. The failure is 100% client-side, inside Continue's autocomplete pipeline.

    Root cause (traced in the bundled extension.js)

    When the Ollama provider is instantiated, Continue calls POST /api/show and parses the parameters string line by line (extension.js:731409–731451). A num_ctx line overwrites this._contextLength:

    // extension.js:731428
    case "num_ctx":
      this._contextLength = options.contextLength ?? Number.parseInt(value2);
      break;

    If the user did not set contextLength in config.yaml, the Modelfile value wins. With num_ctx 4096 in the Modelfile, Continue sees contextLength = 4096.

    If the user did not set maxTokens either, maxTokens falls back to DEFAULT_MAX_TOKENS = 4096 (extension.js:239811).

    For models without a num_ctx line in their Modelfile, contextLength falls back to DEFAULT_CONTEXT_LENGTH = 32768 (extension.js:239813 + the getter at 243810).

    The autocomplete pipeline then calls pruneLength (extension.js:252546):

    function pruneLength(llm, prompt2) {
      const contextLength = llm.contextLength;
      const reservedTokens = llm.completionOptions.maxTokens ?? DEFAULT_MAX_TOKENS;
      const safetyBuffer = getTokenCountingBufferSafety(contextLength);   // Math.min(1000, contextLength * 0.02)
      const maxAllowedPromptTokens = contextLength - reservedTokens - safetyBuffer;
      const promptTokenCount = countTokens(prompt2, llm.model);
      return promptTokenCount - maxAllowedPromptTokens;
    }

    There is no Math.max(0, maxAllowedPromptTokens) clamp. Substituting the two tags:

    Tag contextLength maxTokens safetyBuffer maxAllowedPromptTokens prune for any non-empty prompt
    :latest (no num_ctx) 32768 4096 655.36 28017 negative → no pruning ✅
    :allgpu (num_ctx 4096) 4096 4096 81.92 −81.92 always positive → always over-prunes ❌

    renderPromptWithTokenLimit (extension.js:252585–252624) enters the prune > 0 branch and computes:

    const allowedPrefixTokens = Math.max(0, prefixTokenCount - dropPrefix);  // = 0 when dropPrefix > prefixTokenCount
    const allowedSuffixTokens = Math.max(0, suffixTokenCount - dropSuffix);  // = 0 when dropSuffix > suffixTokenCount
    prefix = pruneLinesFromTop(prefix, allowedPrefixTokens, helper.modelName);
    suffix = pruneLinesFromBottom(suffix, allowedSuffixTokens, helper.modelName);

    pruneLinesFromTop(prompt, 0, …) (extension.js:243283–243297) loops while (totalTokens > 0 && start < length) start++ until start === length, then returns lines.slice(length).join("\n") — the empty string. Same for suffix.

    The request is then sent to Ollama with prefix === "" and suffix === "\n" (the fallback). The model has no anchor and emits an empty or garbage completion. postprocessCompletion (extension.js:251045) drops it via if (isBlank(completion)) return void 0;, and provideInlineCompletionItems returns undefined (extension.js:254898). No error, no log, no UI feedback.

    Note that _streamFim (extension.js:731688–731722) does not gate on contextLength and always fires the request, and num_ctx: this.contextLength (extension.js:731507) is what Continue POSTs back to Ollama as the KV cache size — so the bug is purely in the client-side prompt-budget arithmetic, not in the Ollama round-trip.

    Why this is distinct from #7544

    #7544 is the same family (contextLength − maxTokens − safetyBuffer goes negative or too small) but in the edit pipeline, where it surfaces as a visible Error streaming edit diffs: Token limit reached. File/range likely too large for this edit. This issue is the autocomplete variant, which has no such error path — the failure is completely silent. The config validator at extension.js:239188 only checks model.defaultCompletionOptions fields that the user wrote explicitly; it does not validate the _contextLength value derived from /api/show, so no warning is emitted either.

    Why #7702 does not fix this

    #7702 (merged in Continue 1.2.11) overrides the Ollama contextLength getter to fall back to 8192 instead of 32768 when no num_ctx is present. That helps the no-num_ctx case only when the user has not set contextLength in config. It does not help any Modelfile that explicitly sets num_ctx 4096 (Continue still reads 4096 via /api/show), and it does not touch the pruneLength arithmetic at extension.js:252550. So upgrading Continue does not fix this case.

    To reproduce

    1. Pull a small coder model in Ollama and create two tags — one with num_ctx 4096 pinned, one without:

      ollama pull qwen2.5-coder:1.5b
      ollama show qwen2.5-coder:1.5b --modelfile > /tmp/M.allgpu
      printf 'PARAMETER num_ctx 4096\n' >> /tmp/M.allgpu
      ollama create qwen2.5-coder:1.5b:allgpu -f /tmp/M.allgpu
    2. In ~/.continue/config.yaml add one model entry with roles: [autocomplete] and no explicit contextLength or maxTokens:

      models:
        - name: AllGPU
          provider: ollama
          model: qwen2.5-coder:1.5b:allgpu
          roles: [autocomplete]
    3. Reload VSCode (Developer: Reload Window).

    4. Open a small Python file, e.g.:

      # helloworld
      def hello(): print("Hello, world!")
      
      if __name__ == "__main__":
          █
    5. Trigger autocomplete at the cursor. No completion appears. No error in the Continue output channel.

    6. Switch the autocomplete model to qwen2.5-coder:1.5b (the :latest tag without num_ctx).

    7. Reload VSCode. Trigger autocomplete at the same cursor. hello() is suggested.

    8. Direct API sanity check (both tags return the same correct completion):

      curl -s http://localhost:11434/api/generate -d '{
        "model": "qwen2.5-coder:1.5b:allgpu",
        "prompt": "# helloworld\ndef hello(): print(\"Hello, world!\")\n\nif __name__ == \"__main__\":\n    ",
        "stream": false,
        "options": {"temperature": 0.01, "num_predict": 40, "stop": ["\n\n"]}
      }'

    Expected behavior

    When contextLength is less than or equal to reservedTokens + safetyBuffer, Continue should not compute a negative prompt budget and silently prune the prompt to empty. At minimum:

    1. pruneLength should clamp maxAllowedPromptTokens to Math.max(0, …) (or MIN_RESPONSE_TOKENS), so the worst case is "badly trimmed but still sends something", not "sends nothing and tells no-one".
    2. The config validator at extension.js:239188 should also validate the /api/show-derived _contextLength vs maxTokens and emit a non-fatal warning so users at least see that their inputs are being truncated.
    3. postprocessCompletion should distinguish "blank because the model produced nothing useful" from "blank because the client sent an empty prompt"; the latter should log a warning.

    Workarounds (for anyone hitting this today)

    Any one of these fixes it:

    1. In config.yaml, give the model an explicit small maxTokens so the budget is positive:

      - name: AllGPU
        provider: ollama
        model: qwen2.5-coder:1.5b:allgpu
        roles: [autocomplete]
        defaultCompletionOptions:
          maxTokens: 128    # 4096 − 128 − 81 = 3887 > 0
    2. In config.yaml, give the model an explicit larger contextLength (Continue will POST it back to Ollama as num_ctx, so the server KV cache must actually fit):

      defaultCompletionOptions:
        contextLength: 8192
    3. Remove the PARAMETER num_ctx 4096 line from the Ollama Modelfile (Continue will fall back to DEFAULT_CONTEXT_LENGTH).

    Related issues / PRs

    Log output

    No log output is produced by this bug — that is part of the problem. The Continue output channel shows normal stream providerID=ollama modelID=… step=… lines and then a normal done for Ollama's response; the truncation happens before the request and the discard happens after, both silently.

  2. added 2 commits that reference this issue on Jul 28, 2026
    8fe6f59
    a32931c
  3. added a commit that references this issue on Aug 7, 2026
    86f4a17
  4. corbenicai commented on Nov 16, 2026

    @corbenicai

    Hello, I looked at the autocomplete silence you get when Ollama num_ctx is 4096 and Continue falls back to the default maxTokens of 4096.

    pruneLength going negative is expected in that case: Continue thinks there is no room left for the prefix and suffix, so it prunes both to empty and the model returns nothing, with no error. A practical fix is to set contextLength or maxTokens on that model in ~/.continue/config.yaml to a value clearly below the Ollama num_ctx (or raise num_ctx if VRAM allows), so the prompt budget stays positive. Your Qwen 2.5 Coder 1.5b AllGPU entry currently leaves both unset, which is why it hits the default 4096 against the pinned num_ctx.

    This is a budget bug, not a missing model, so I would not change the Ollama Modelfile alone without also setting those Continue fields.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions