[lavaman131]

Reactions read by the decisions model never got a reply

What was seen

On a hosted deployment, chopin answered the toolbar's ❓ and 👎 within about 5 seconds, in Chat and in threads. 🤔, 😕 and 🙋 on chopin's messages never got a reply, in 7 tries in Chat and threads. No reaction picked from the emoji picker started a Planner turn at all.

Cause

❓, 👎 and ✅ are read in code. Every other emoji goes to the decisions classifier with two questions: a Choice ("what does the reaction mean?") and a companion Noul ("Does the reaction ask the Planner to respond?"). A reply needed the Choice at 0.75 likely and 0.7 confident, and the Noul at 0.8.

I asked the same classifier the deployment uses (openai-decisions/gpt-6-luna) about the same kind of reaction from a local script. The Choice was right: 🤔 read as confused at 0.93, 😕 at 0.98. The Noul was not. A reaction doesn't literally "ask", so the model put a thinking face at 0.36 and a confused face at 0.39, far under 0.8. A magnifying glass read as "wants more" at 0.91 had a Noul of 0.03. So the Noul vetoed almost every reply. The call itself never failed.

🙋 had a second problem: none of the Choice's descriptions covered a raised hand, so it read as none (0.43 likely, 0.32 confident).

The failure was invisible. The log line for a reading had the scores but not why it didn't reply, and a failed call logged only Error.

Fix

• One question. The reply is decided by the Choice alone. The companion Noul is gone from the request. The Choice already says whether the reaction is confusion, disagreement or a wish for more, which are the kinds that ask for a reply, and its probability and confidence are what is calibrated below.
• Clearer descriptions.
  • "Confused" also covers doubting the message.
  • "More" covers having a question or something to add, with a raised hand as an example.
  • A plus sign moved from "more" to "approve", since a plus one agrees.
    The model still decides every emoji outside the toolbar's three.
• Thresholds from measurements. A reply now needs a replying kind at 0.6 likely and 0.55 confident (below). A reading that is only logged needs 0.5 and 0.4.
• Visible failures.
  • Every reading's log line now says why it didn't reply: unreadable, unsure, not-asking or below-reply-threshold.
  • A failed call logs [chat] reactions could not be read with a reason (HTTP 429, timeout, invalid-answer, invalid-request, unknown-model, …), the number of reactions and the size of the state. It never logs the messages.
• Only reactions on chopin's own messages are read, in Chat and in threads, as before.

Measurements

These come from the classifier named above. There were five conversations, each with a Planner message and the messages before it. Each emoji was asked one at a time twice and in one batch twice, so 20 readings per emoji. The last column applies the new rule.

Emoji	Read as (count of 20)	Probability	Confidence	Replies now
🤔	confused 20	0.63–0.94	0.56–0.93	20 of 20
😕	confused 20	0.89–1.00	0.87–1.00	20 of 20
🙋	more 16, confused 4	0.65–1.00	0.58–1.00	20 of 20
🔍	more 20	0.79–0.96	0.75–0.95	20 of 20
➕	approve 20	0.80–0.97	0.76–0.96	0 of 20
👀	ack 20	0.45–0.99	0.34–0.99	0 of 20
👍	approve 20	0.90–1.00	0.88–1.00	0 of 20
🎉	approve 20	0.86–0.99	0.83–0.99	0 of 20
❤️	approve 18, none 2	0.50–0.96	0.40–0.95	0 of 20
🙏	approve 20	0.44–0.95	0.33–0.94	0 of 20
😐	none 16, confused 4	0.39–0.93	0.27–0.92	0 of 20
🤷	confused 10, none 10	0.39–0.58	0.27–0.50	0 of 20

• The weakest reading that should reply was 🤔 at 0.63 / 0.56.
• The strongest that should not was 🤷 read as confused at 0.58 / 0.50.
• The thresholds sit between those two.

Under the old rule, the same readings gave 56 wrong outcomes out of 260, all of them missing replies. Under the new rule there are none. (❓ is read in code and never reaches the model.)

Tests

• reaction-intent.test.ts:
  • the request is one Choice per reaction;
  • routing by probability and confidence, with each reason;
  • a table of recorded classifier readings for 🤔 😕 🙋 🔍 (reply) and 👀 👍 🎉 ❤️ ➕ 🤷 😐 (no reply);
  • failures log a reason and sizes, never the message.
• reaction-reply.test.ts: updated to one question per reaction. A likely intent with too little confidence earns no turn.
• Browser specs' fake classifier (e2e/reaction-judgments.ts): it now answers with readings like the real model's, and covers 😕 and 🙋. Before, it answered a Noul of 0.96 that the real model never gives.

Checks run

• bun run types: passes.
• bun run ci: passes.
• bun test apps/server/src/chat: 1008 pass, 0 fail (before rebasing onto 8cb7e62; after it, the three reaction suites pass 81 of 81).
• Playwright, --workers=2: message-reactions, chat-thread-reactions, reaction-reader and reaction-access: 11 passed, 1 failed. The failure was chat-thread-reactions' toolbar test, whose click on Chat's toolbar timed out because the transcript stack intercepted the pointer under two workers. Run alone, chat-thread-reactions passed 4 of 4. This change touches no web code.

Judgement calls

• Noul removed, not reworded. I also tried a reworded Noul ("would the person want the Planner to answer it…"). It fixed confused faces (0.69–0.94) but still read 🙋 and 🔍 anywhere from 0.03 to 0.68. Any Noul bar either kept those from replying or let nothing through, so it couldn't be a reliable gate.
• The rule's margin is narrow: about 0.03 on likelihood, between a thinking face in one conversation and a shrug. A shrug that sometimes gets a reply would be a small cost: one short Planner turn.
• Hosted log not checked. I didn't read the hosted server's log. The local readings reproduce the reported behaviour exactly, with the same model and the same question.