Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,11 @@ All notable changes to this project will be documented in this file. See [standa
- First-class Kiro support: skills, steering rules, JSON subagents with CLI `agentSpawn` session-start hooks, and MCP sync to `.kiro/` (for [#500](https://github.com/Tencent/teamai-cli/issues/500)).
- Multi-project management: `role` and `project` together resolve resource namespaces, and project-private learnings are isolated ([#426](https://github.com/Tencent/teamai-cli/pull/426), for [#375](https://github.com/Tencent/teamai-cli/issues/375)).
- Data partitions auto-migrate a legacy `.teamai`, resume interrupted migrations, smoke-check the clone, and keep a git-ignored backup ([#439](https://github.com/Tencent/teamai-cli/pull/439), for [#374](https://github.com/Tencent/teamai-cli/issues/374)).
- Teams add their own course-correction words via `sharing.intervention.correctionKeywords` in `teamai.yaml`. The built-in list still covers only Chinese, English and Japanese, so corrections typed in other languages count only once the team configures them. The `UserPromptSubmit` hook now stores a `correction` flag on each dashboard prompt event (for [#564](https://github.com/Tencent/teamai-cli/issues/564)).

### 🐛 Bug Fixes

- Course-correction detection matches keywords in space-separated scripts as whole words, so Spanish "segundo" no longer counts as `undo` (for [#564](https://github.com/Tencent/teamai-cli/issues/564)).
- MCP `requires` is resolved from `PATH` (including Windows `PATHEXT`), so `teamai mcp inject` no longer skips servers such as `uvx` on Windows ([#540](https://github.com/Tencent/teamai-cli/pull/540), for [#539](https://github.com/Tencent/teamai-cli/issues/539)).
- The GitHub and CNB providers resolve their CLI to a launchable absolute path and start it through cross-spawn, so on Windows they no longer answer "installed" while every call fails silently ([#520](https://github.com/Tencent/teamai-cli/pull/520)).
- `enabledAgents` now also gates CLI builtin deploy, CLAUDE.md-class injects, and last-pull skip-sync targets, so an already-installed tool outside the whitelist is not written to ([#510](https://github.com/Tencent/teamai-cli/issues/510)).
Expand Down
14 changes: 13 additions & 1 deletion docs/usage-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -1240,10 +1240,20 @@ Each session card shows a `⚠ N` badge, counting the **number of human interven
|------|------|----------|
| `interrupt` | User pressed ESC to interrupt the agent mid-execution | An interrupted turn in the transcript |
| `toolReject` | User rejected a tool call (permission deny) | A tool_result marked as rejected in the transcript |
| `correction` | Within 60s after the agent stops, the user submits a follow-up prompt containing a correction keyword ("not right" / "redo" / "wrong" / 「違う」 / 「やり直し」 / etc. — Chinese, English and Japanese) | The stop → prompt_submit event pattern |
| `correction` | Within 60s after the agent stops, the user submits a follow-up prompt containing a correction keyword ("not right" / "redo" / "wrong" / 「違う」 / 「やり直し」 / etc. — Chinese, English and Japanese built in, plus any team keywords) | The stop → prompt_submit event pattern |

> Privacy: only counts are tracked — no prompt or transcript text is ever stored.

Keywords in a space-separated script (English, Spanish, ...) must appear as a whole word, so Spanish "segundo" does not count as `undo`. Chinese and Japanese keywords match as substrings. The built-in list covers only Chinese, English and Japanese; a correction typed in any other language is not detected until the team adds its own words in `teamai.yaml`. Team words are merged with the built-in list and matched case-insensitively under the same rules:

```yaml
sharing:
intervention:
correctionKeywords: [rehazlo, deshaz, "no era eso", "otra vez"]
```

The prompt is checked when the `UserPromptSubmit` hook captures it, so a change to the team keywords applies to new prompts after the next `teamai pull`; sessions recorded earlier are not re-evaluated.

Intervention data is automatically aggregated and reported to the team's `stats/<user>.yaml` during `teamai pull`, and shown in the "Session Autonomy" leaderboard of `teamai digest`, with team averages and per-person intervention rate rankings — useful for verifying whether a skill/rule reduces intervention rates after rollout. Tools without a transcript (e.g. Cursor) degrade gracefully, tracking only `correction`.

#### Conversation Volume & Token Usage
Expand Down Expand Up @@ -1569,6 +1579,8 @@ sharing:
enabled: false # optional; strip AI-tool commit trailers team-wide
contributeHint:
enabled: true # optional; false = no /teamai-share-learnings nudge after high-friction sessions
intervention:
correctionKeywords: [] # optional; extra course-correction words merged with the built-in zh/en/ja list
```

### config.yaml (local config)
Expand Down
14 changes: 13 additions & 1 deletion docs/usage-guide.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -1207,10 +1207,20 @@ teamai dashboard --port 8080
|------|------|----------|
| `interrupt` | 用户在 agent 执行中途按 ESC 打断 | transcript 中被中断的 turn |
| `toolReject` | 用户拒绝某个工具调用(permission deny) | transcript 中标记拒绝的 tool_result |
| `correction` | agent stop 后 60s 内用户追加含「不对 / 重来 / 错了 / wrong / redo / 違う / やり直し」等纠偏词(、英、日)的 prompt | stop → prompt_submit 事件模式 |
| `correction` | agent stop 后 60s 内用户追加含「不对 / 重来 / 错了 / wrong / redo / 違う / やり直し」等纠偏词(内置中、英、日,外加团队自定义词)的 prompt | stop → prompt_submit 事件模式 |

> 隐私:只统计**次数**,不落地任何 prompt 或 transcript 原文。

以空格分词的文字(英语、西班牙语等)中的纠偏词必须整词匹配,因此西班牙语 "segundo" 不会被算作 `undo`;中文、日文纠偏词仍按子串匹配。内置列表只覆盖中、英、日三种语言,其他语言的纠偏在团队于 `teamai.yaml` 添加自己的词之前不会被识别。团队词与内置列表合并,忽略大小写,遵循同样的匹配规则:

```yaml
sharing:
intervention:
correctionKeywords: [rehazlo, deshaz, "no era eso", "otra vez"]
```

匹配在 `UserPromptSubmit` hook 捕获 prompt 时完成,因此修改团队纠偏词后,下一次 `teamai pull` 之后的新 prompt 才会生效;之前记录的会话不会重新评估。

干预数据会随 `teamai pull` 自动聚合上报到团队 `stats/<user>.yaml`,并在 `teamai digest` 的「会话自主性」榜单中给出团队均值与人均干预率排行,可用于验证某个 skill / rule 上线后干预率是否下降。无 transcript 的工具(如 Cursor)会优雅降级,只统计 `correction`。

#### 对话量与 Token 用量
Expand Down Expand Up @@ -1528,6 +1538,8 @@ sharing:
enabled: false # 可选,为全团队去除 AI 工具提交尾注
contributeHint:
enabled: true # 可选,false = 高摩擦 session 结束后不再提示 /teamai-share-learnings
intervention:
correctionKeywords: [] # 可选,额外的纠偏词,与内置中/英/日列表合并
```

### config.yaml(本地配置)
Expand Down
102 changes: 102 additions & 0 deletions src/__tests__/dashboard-collector.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,71 @@ describe('parseHookEvent', () => {
expect(event!.tool).toBe('claude-internal');
});

it('flags a correction prompt on UserPromptSubmit', async () => {
const raw = JSON.stringify({ hook_event_name: 'UserPromptSubmit', session_id: 's', prompt: 'wrong, redo it' });
const event = await parseHookEvent(raw, 'claude');
expect(event!.correction).toBe(true);
});

it('does not flag a Latin keyword inside a longer word (issue #564)', async () => {
// "undo" / "redo" are common Spanish and Portuguese word endings.
for (const word of ['segundo', 'mundo', 'profundo', 'rotundo', 'redondo', 'enredo', 'oriundo']) {
const raw = JSON.stringify({ hook_event_name: 'UserPromptSubmit', session_id: 's', prompt: `dame el ${word} fichero` });
const event = await parseHookEvent(raw, 'claude');
expect(event!.correction, word).toBe(false);
}
});

it('matches Latin keywords as whole words, including multi-word ones', async () => {
for (const prompt of ['undo that', 'Undo.', "that's not it", "don't do that", 'wrong!']) {
const raw = JSON.stringify({ hook_event_name: 'UserPromptSubmit', session_id: 's', prompt });
const event = await parseHookEvent(raw, 'claude');
expect(event!.correction, prompt).toBe(true);
}
});

it('treats underscore as part of a word, so identifiers do not match', async () => {
const cases: Array<[string, boolean]> = [['run test_undo again', false], ['undo_it', false], ['undo it', true]];
for (const [prompt, expected] of cases) {
const raw = JSON.stringify({ hook_event_name: 'UserPromptSubmit', session_id: 's', prompt });
const event = await parseHookEvent(raw, 'claude');
expect(event!.correction, prompt).toBe(expected);
}
});

it('keeps substring matching for Chinese and Japanese keywords', async () => {
for (const prompt of ['这不对', '違うよ']) {
const raw = JSON.stringify({ hook_event_name: 'UserPromptSubmit', session_id: 's', prompt });
const event = await parseHookEvent(raw, 'claude');
expect(event!.correction, prompt).toBe(true);
}
});

it('merges team correctionKeywords with the built-in list', async () => {
const options = { correctionKeywords: ['rehazlo', 'no era eso', 'mal', '重做'] };
const cases: Array<[string, boolean]> = [
['esto está mal, rehazlo', true],
['No era eso', true],
['请重做一遍', true],
// Team Latin keywords also need a whole word: accented letters count as letters.
['no está malísimo', false],
['continúa con el siguiente paso', false],
];
for (const [prompt, expected] of cases) {
const raw = JSON.stringify({ hook_event_name: 'UserPromptSubmit', session_id: 's', prompt });
const event = await parseHookEvent(raw, 'claude', options);
expect(event!.correction, prompt).toBe(expected);
}
});

it('checks the full prompt, not only the 200-char summary', async () => {
const prompt = `${'x '.repeat(150)}wrong`;
const raw = JSON.stringify({ hook_event_name: 'UserPromptSubmit', session_id: 's', prompt });
const event = await parseHookEvent(raw, 'claude');
expect(event!.promptSummary!.length).toBe(200);
expect(event!.correction).toBe(true);
});

it('truncates long prompts to 200 chars', async () => {
const longPrompt = 'x'.repeat(500);
const raw = JSON.stringify({
Expand Down Expand Up @@ -743,6 +808,43 @@ describe('rebuildSessions interventions', () => {
expect(sessions[0].interventions.correction).toBe(0);
});

it('honors the correction flag written by the hook over the summary text', () => {
const t0 = new Date();
const later = new Date(t0.getTime() + 5_000).toISOString();
const flagged = rebuildSessions([
{ type: 'session_start', timestamp: t0.toISOString(), sessionId: 's1', tool: 'claude', cwd: '/p' },
{ type: 'stop', timestamp: t0.toISOString(), sessionId: 's1', tool: 'claude' },
// Team keyword matched at capture time; the summary alone would not match.
{ type: 'prompt_submit', timestamp: later, sessionId: 's1', tool: 'claude', promptSummary: 'esto está mal, rehazlo', correction: true },
]);
expect(flagged[0].interventions.correction).toBe(1);

const unflagged = rebuildSessions([
{ type: 'session_start', timestamp: t0.toISOString(), sessionId: 's1', tool: 'claude', cwd: '/p' },
{ type: 'stop', timestamp: t0.toISOString(), sessionId: 's1', tool: 'claude' },
{ type: 'prompt_submit', timestamp: later, sessionId: 's1', tool: 'claude', promptSummary: 'wrong, redo it', correction: false },
]);
expect(unflagged[0].interventions.correction).toBe(0);
});

it('falls back to whole-word matching on legacy events without the flag (issue #564)', () => {
const t0 = new Date();
const later = new Date(t0.getTime() + 5_000).toISOString();
const spanish = rebuildSessions([
{ type: 'session_start', timestamp: t0.toISOString(), sessionId: 's1', tool: 'claude', cwd: '/p' },
{ type: 'stop', timestamp: t0.toISOString(), sessionId: 's1', tool: 'claude' },
{ type: 'prompt_submit', timestamp: later, sessionId: 's1', tool: 'claude', promptSummary: 'dame el segundo fichero' },
]);
expect(spanish[0].interventions.correction).toBe(0);

const english = rebuildSessions([
{ type: 'session_start', timestamp: t0.toISOString(), sessionId: 's1', tool: 'claude', cwd: '/p' },
{ type: 'stop', timestamp: t0.toISOString(), sessionId: 's1', tool: 'claude' },
{ type: 'prompt_submit', timestamp: later, sessionId: 's1', tool: 'claude', promptSummary: 'undo that' },
]);
expect(english[0].interventions.correction).toBe(1);
});

it('aggregates all three intervention types together', () => {
const t0 = new Date();
const sessions = rebuildSessions([
Expand Down
32 changes: 32 additions & 0 deletions src/__tests__/hook-handlers.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -873,3 +873,35 @@ describe('post-tool-use dispatch — local-agent runs detached, never blocks hos
expect(mockParseHookEvent).toHaveBeenCalled();
});
});

describe('dashboard-report team correction keywords', () => {
const handler = () => buildHandlerRegistry().find(
(r) => r.event === 'prompt-submit' && r.handler.name === 'dashboard-report',
)!.handler;

beforeEach(() => {
mockParseHookEvent.mockClear();
mockAutoDetectInit.mockClear();
});

it('passes sharing.intervention.correctionKeywords to parseHookEvent on prompt hooks', async () => {
mockAutoDetectInit.mockResolvedValueOnce({
localConfig: { repo: { localPath: '/tmp', remote: '' }, username: 'test', scope: 'user' },
teamConfig: { team: 'test', repo: '', toolPaths: {}, sharing: { intervention: { correctionKeywords: ['rehazlo'] } } },
});
await handler().execute({ hook_event_name: 'UserPromptSubmit', session_id: 's', prompt: 'rehazlo' }, 'claude');
expect(mockParseHookEvent).toHaveBeenCalledWith(expect.any(String), 'claude', { correctionKeywords: ['rehazlo'] });
});

it('falls back to built-in keywords only when team config cannot be read', async () => {
mockAutoDetectInit.mockRejectedValueOnce(new Error('not initialized'));
await handler().execute({ hook_event_name: 'UserPromptSubmit', session_id: 's', prompt: 'wrong' }, 'claude');
expect(mockParseHookEvent).toHaveBeenCalledWith(expect.any(String), 'claude', { correctionKeywords: [] });
});

it('does not read team config for hooks without a prompt', async () => {
await handler().execute({ hook_event_name: 'Stop', session_id: 's' }, 'claude');
expect(mockAutoDetectInit).not.toHaveBeenCalled();
expect(mockParseHookEvent).toHaveBeenCalledWith(expect.any(String), 'claude', { correctionKeywords: [] });
});
});
19 changes: 19 additions & 0 deletions src/__tests__/types.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import {
MemberConfigSchema,
TeamaiConfigSchema,
SharingConfigSchema,
getInterventionSharing,
StateSchema,
LocalConfigSchema,
resolveLegacyProjectHookScope,
Expand Down Expand Up @@ -181,6 +182,24 @@ describe('SharingConfigSchema env', () => {
});
});

describe('SharingConfigSchema intervention', () => {
it('leaves intervention undefined when absent and defaults keywords to []', () => {
const result = SharingConfigSchema.parse({});
expect(result.intervention).toBeUndefined();
expect(getInterventionSharing({ sharing: result })).toEqual({ correctionKeywords: [] });
expect(getInterventionSharing({})).toEqual({ correctionKeywords: [] });
});

it('accepts team correctionKeywords', () => {
const result = SharingConfigSchema.parse({ intervention: { correctionKeywords: ['rehazlo', 'no era eso'] } });
expect(getInterventionSharing({ sharing: result }).correctionKeywords).toEqual(['rehazlo', 'no era eso']);
});

it('rejects non-string keywords', () => {
expect(() => SharingConfigSchema.parse({ intervention: { correctionKeywords: [1] } })).toThrow();
});
});

describe('StateSchema pushedEnvVars', () => {
it('should default pushedEnvVars to empty array', () => {
const result = StateSchema.parse({});
Expand Down
43 changes: 39 additions & 4 deletions src/dashboard-collector.ts
Original file line number Diff line number Diff line change
Expand Up @@ -810,11 +810,34 @@ export async function countInterventions(
return { interrupt, toolReject, toolError };
}

/** True when a prompt looks like a course-correction (vs. a fresh task). */
function isCorrectionPrompt(text?: string): boolean {
/** Scripts that do not separate words with spaces: substring matching is correct there. */
const UNSPACED_SCRIPT_RE = /[\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}\p{Script=Hangul}]/u;

/**
* Match a keyword as a whole word: no letter, digit or underscore may touch either
* end, so `undo` does not fire on "segundo" or on the identifier "test_undo".
*/
function wordBoundaryPattern(keyword: string): RegExp {
const escaped = keyword.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
return new RegExp(`(?<![\\p{L}\\p{N}_])${escaped}(?![\\p{L}\\p{N}_])`, 'u');
}

/** True when `lower` contains `keyword`, whole-word for spaced scripts, substring otherwise. */
function containsKeyword(lower: string, keyword: string): boolean {
const k = keyword.trim().toLowerCase();
if (!k) return false;
if (UNSPACED_SCRIPT_RE.test(k)) return lower.includes(k);
return wordBoundaryPattern(k).test(lower);
}

/**
* True when a prompt looks like a course-correction (vs. a fresh task).
* `extraKeywords` are the team's `sharing.intervention.correctionKeywords`.
*/
function isCorrectionPrompt(text?: string, extraKeywords: readonly string[] = []): boolean {
if (!text) return false;
const lower = text.toLowerCase();
return CORRECTION_KEYWORDS.some((k) => lower.includes(k));
return [...CORRECTION_KEYWORDS, ...extraKeywords].some((k) => containsKeyword(lower, k));
}

/**
Expand All @@ -841,6 +864,11 @@ function mapEventType(hookEventName: string): DashboardEventType | null {
}
}

export interface ParseHookEventOptions {
/** Team keywords (`sharing.intervention.correctionKeywords`) merged with the built-in list. */
correctionKeywords?: readonly string[];
}

/**
* Parse a hook STDIN JSON payload into a DashboardEvent.
* Returns null if the payload is invalid or irrelevant.
Expand All @@ -849,6 +877,7 @@ function mapEventType(hookEventName: string): DashboardEventType | null {
export async function parseHookEvent(
raw: string,
tool: string,
options?: ParseHookEventOptions,
): Promise<DashboardEvent | null> {
if (!raw.trim()) return null;

Expand Down Expand Up @@ -903,6 +932,10 @@ export async function parseHookEvent(
if (eventType === 'prompt_submit' && typeof hookData.prompt === 'string') {
// Keep first 200 chars of the prompt as summary
event.promptSummary = hookData.prompt.slice(0, 200);
// Decide "correction" here, over the full prompt, because only the hook knows
// which team (and so which extra keywords) the prompt belongs to. The
// machine-level events file mixes sessions from every team.
event.correction = isCorrectionPrompt(hookData.prompt, options?.correctionKeywords);
}

// Extract transcript path, AI output and intervention counts from Stop event
Expand Down Expand Up @@ -1339,7 +1372,9 @@ export function aggregateSessionMetrics(
const stopAt = lastStopAt.get(event.sessionId);
if (stopAt !== undefined) {
const gap = new Date(event.timestamp).getTime() - stopAt;
if (gap >= 0 && gap <= CORRECTION_WINDOW_MS && isCorrectionPrompt(event.promptSummary)) {
// Legacy events (no `correction` flag) fall back to the built-in list.
const isCorrection = event.correction ?? isCorrectionPrompt(event.promptSummary);
if (gap >= 0 && gap <= CORRECTION_WINDOW_MS && isCorrection) {
m.correction++;
}
// Each stop is consumed once — a later prompt is a new task, not a correction.
Expand Down
Loading