flm serve OpenAI-compatible endpoint hardcodes max_tokens default = 4096 → long responses are silently truncated (finish_reason: "length") when the client omits max_tokens

[camilo6castell]

Title

flm serve OpenAI-compatible endpoint hardcodes max_tokens default = 4096 → long responses are silently truncated (finish_reason: "length") when the client omits max_tokens

Environment

• FLM version: 1.0.6 (Linux package fastflowlm 1.0.6-1, Arch Linux, x86_64)
• Hardware: AMD Ryzen™ AI NPU (XDNA2, NPU2 kernels)
• Model: qwen3.5:9b (Qwen3.5-9B-NPU2), server started with --ctx-len 64000 --pmode turbo
• Client: OpenAI-compatible (opencode 2.0.12) + direct curl

Summary

The /v1/chat/completions handler falls back to a hardcoded max_tokens = 4096 when the request body does not include max_tokens/max_completion_tokens:

request.value("max_tokens", 4096)   // rest_handler.cpp, three request sites (~lines 655 / 1106 / 1420 in 1.0.6)

Several common OpenAI-compatible clients never send max_tokens (verified: opencode 2.0.12 sends only model/store/stream/stream_options). For those clients this 4096 cap becomes an invisible hard ceiling: any generation longer than 4096 tokens is cut mid-sentence with finish_reason: "length", even when the model would have stopped naturally. This truncates long outputs (webpages, code files) with no way for the client to detect the cause other than parsing finish_reason.

Reproduction

1. Start: flm serve qwen3.5:9b --ctx-len 64000 -p 52626
2. POST without max_tokens:

curl -s http://127.0.0.1:52626/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.5:9b","messages":[{"role":"user","content":"Build a complete single-file HTML page with embedded JS (form, validation, dark mode)..."}]}'

Observed: "finish_reason":"length", "completion_tokens":4096, output cut mid-JS (at <section id="contact), never reaching </html>.

3. Same message WITH "max_tokens":12000: "finish_reason":"stop", completed </html>, "completion_tokens":3772.

Evidence (same model, same server)

Request	max_tokens	finish_reason	completion_tokens	Result
A	(absent)	length	4096	Truncated mid-JS
B	12000	stop	3772	Completed </html>
C	12000 (+ think:true, reasoning_effort:"medium")	stop	4940	Completed

Decode speed ≈ 9.2 tok/s → each truncated run wastes ~7 minutes of NPU time while producing a broken response.

Expected behavior

Following the OpenAI convention (and Ollama's), when the client omits the output budget, generation should be bounded only by the context window — not by a fixed 4096 ceiling. If a default is kept, it should be derived from the active context, e.g. min(16384, MAX_L - prompt_tokens), and ideally configurable (--max-tokens env/flag, or per-model default from model_list.json's default_context_length).

Related observations (same session, feel free to split)

1. Default sampler for Qwen3.5 in 1.0.6: presence_penalty 1.5, frequency_penalty 1.0, temperature 0.7, top_p 0.8. The aggressive penalties visibly degrade long code generation; opencode also does not forward client-provided sampling options, so these defaults apply in practice.
2. With "think": true + "reasoning_effort", FLM produced ~4.4K chars of reasoning but never emitted the response marker: reasoning was appended to content while reasoning_content stayed empty.

Suggested fix

In src/server/rest_handler.cpp, replace the three hardcoded defaults:

// before
request.value("max_tokens", 4096)

// after
request.value("max_tokens", std::min(16384, max_len - prompt_len))

Happy to open a PR if this behavior is confirmed as unintentional.