Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Changed
- Clear the complete current draft from a visible source-header button (also `Cmd/Ctrl+N`), stopping dictation and
retaining the insertion target and preferences. Pending startup, queued writes and late transcription results cannot
repopulate the cleared draft.
- Switching translation tiers refreshes the open composer's model requirements and cached translations; manually
edited translations are kept. A model switch waits for active requests instead of killing their server.
- Start live passage cleanup after a 2 s pause instead of 3 s. Clearly label live results as previews.
- Stopping recording atomically marks the draft as unfinished, cancels in-flight preview work, and prioritizes a
whole-text summary followed by final polish. Writing and copying wait for completion.
- Remove recursive final-polish retries that could multiply latency. Cancelled or stale results cannot overwrite
resumed recordings, source edits, or manually edited results; rejected polish is explicitly reported.

### Added
- Brief runtime model notices and on-demand details from the local/cloud chip. Notices distinguish requested models
from models actually loaded locally or named by a cloud response; deployment aliases are not presented as confirmed
backend identities.
- Editable whole-text polish requirements under Voice settings, with validation, persistent storage, and restore
default. Damaged settings use built-in requirements; unusable custom output retries default requirements once.
- A dedicated final-polish prompt sharing the existing local/cloud cleanup route. Long summaries cover all input
chunks, and final output may use paragraphs without disabling the other fidelity checks.

## [2.0.2] - 2026-11-02

Dictation now reads as connected text: it is cleaned passage by passage while you speak, and the whole dictation is
Expand Down Expand Up @@ -37,7 +58,7 @@ polished as one text when you stop.
- A cleanup answer that was cut off by the model's output limit no longer shows a red error that blocks writing; the
words are kept like any refused cleanup. The output limit leaves room for a model that copies the read-only context,
and a copied context is removed from the answer.
- Passages end after a 3 s pause or 200 characters (was 1.5 s / 120).
- Passages end after a 3 s pause or 200 characters.
- The composer window can be resized: drag its right edge, bottom edge or corner. The enlarge / restore button in the
title bar (next to close), `⌘E` / `Ctrl+E`, or a double click on an edge enlarges it to show as much text as the
screen allows and restores it. The panel then fills the window and, side by side, each column scrolls on its own.
Expand Down
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ release is tag `v1.4.2`.
too, shown next to them in the menus: `⌘1` / `⌘2` / `⌘3` pick the output, `⌘L` swaps the language, `⇧⌘C` copies,
`⌘N` starts a new draft, `⌘R` recent drafts, `⌘O` import, `⌘S` export, `⌘E` enlarges or restores the window, `⌘,` settings.
Drag the right edge, the bottom edge or the corner to resize the window, or use the enlarge button next to close.
The source header's **Clear** button stops recording and empties the entire current draft; `⌘N` / `Ctrl+N` does the same.
3. **Pick what is written**, with one control: *Clean up* (fillers, repetitions and homophone errors removed, meaning
untouched), *Translate to English / Chinese*, or *Original*. Your original words stay visible and can be restored for
any sentence.
Expand All @@ -26,6 +27,14 @@ Cleanup is checked by a fidelity gate: a rewrite that adds facts, drops negation
said is rejected and that sentence keeps your words. Cleanup can be switched off in Settings; the composer then offers
only *Translate* and *Original*.

While recording, cleanup is a live preview. After stopping, the whole text is summarized and polished; writing waits
until that processing ends. Customize the final polish requirements in **Settings → Voice**, or restore the default.
Invalid requirements fall back to the built-in ones; a rejected final result keeps the passage preview with a notice.

Model names appear briefly when recording starts or the active processing model changes. Click the **Local / Cloud**
chip to inspect what this window actually used. Speech uses X-ASR 480 ms plus Qwen3-ASR 0.6B; cleanup uses Qwen3.5 4B.
The fast/quality translation choice selects Hy-MT2 1.8B/7B and does not change the speech recognizer.

Imported recordings become a read-only timeline that goes through the same cleanup and translation, and can be
exported as SRT.

Expand Down
4 changes: 2 additions & 2 deletions docs/superpowers/specs/2026-10-29-voice-dictation-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,12 +119,12 @@ POC 用首个实时文本前的余量、VAD 和最后出字位置共同确定解
|---|---|---|
| 实时 ASR | X-ASR zh/en,480 ms,int8 | 首字反馈优先;160/960 ms 尚未完整对照 |
| 二遍 ASR | Qwen3-ASR-0.6B,int8 | 真人口音样本上综合较好;不是每种任务最优 |
| 整理 | Qwen3.5-4B,Q4_K_M | 本机可按段运行(一段约 200 字,约 4 s),停止后整篇润色(约 250 字 4–6 s);M2/16 GB 基线尚需实测 |
| 整理 | Qwen3.5-4B,Q4_K_M | 按段预览;停止后先摘要、再全文润色。约 250 字样本本机含冷启动约 10 s,长文更慢;M2/16 GB 基线尚需实测 |
| VAD | Silero,阈值 0.35,最短静音 250 ms | 三档阈值对照后的折中;并非所有数据集上单调改善 |
| 硬停顿 | 800 ms | 避免每次换气都断流 |
| 软切分触发 | 12,000 ms | 控制片段增长,但还需迁移时补无停顿策略 |
| 文本尾部强制成句 | 90 字符 | 防止无标点内容无限积累,不保证语义完整 |
| 成段停顿 | 3,000 ms | 说话人停下这么久,攒着的句子成为一段,立即整理 |
| 成段停顿 | 2,000 ms | 提供较快的逐段预览;停止后立即先摘要、再全文润色 |
| 成段长度 | 200 字 | 达到即在句末成段,长篇口述不必等到停顿 |
| 前文 | 约 300 字(整段,由近及远,至少上一段) | 小上下文降低成本;不得输出前文 |
| 首次出字回看余量 / 尾部证据余量 | 3,000 / 300 ms | 经验启发式,不是模型对齐精度保证 |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@

| 条件 | 值 | 理由 |
|---|---|---|
| 说话人停了 `PassagePauseMs` | 3000 ms | 想事情的停顿(1.5 s 时用户的口述几乎每句一段,仍然割裂);此时立即整理 |
| 说话人停了 `PassagePauseMs` | 2000 ms | 尽早提供逐段预览;不以这条边界作为最终润色的边界 |
| 当前段达到 `PassageMaxChars`,在句末成段 | 200 字 | 长篇口述不会一直等到停顿;整理一段约 4 s |
| 停止听写 | — | 刷出剩余内容 |

Expand All @@ -42,7 +42,7 @@

### 2.3 保真门禁(`FidelityGate`)

门禁仍然只拦“确实丢了信息”的情况。真实口述里它最常见的误拒是**删掉了虚词**:同一份截图文本,模型的整理基本都对,但“选择股票的话”“那么”“他们的这个工作”被当成“无故删除”,整段退回原话——这才是用户看到“整理没变化”的直接原因。现在补四条能解释的删除:
门禁是启发式检查,不是语义等价的证明,可能误拒或漏检。真实口述里常见的误拒是**删掉了虚词**:“选择股票的话”“那么”“他们的这个工作”被当成“无故删除”,整段退回原话。现在允许四类可解释的删除:

1. **口语虚词**:“的话、那么、然后、其实、这个、一下、呢、呀、吧、嘛……”整段被删算作语气词。
2. **轻微删减**:删掉不超过 3 个字、不含否定的零碎字(“他们的”“这边”),累计不超过原文的 8 %(至少 4 个字)。更大的删除仍拒绝,几处零碎删减加起来也会超预算。
Expand All @@ -59,14 +59,24 @@

说话时每段单独整理,只能在段内接话;用户在想事情时的停顿会把一句话切在段与段之间(“我想了解的是。”|“行业大牛是怎么……”)。实测(用户 12 段口述):逐段整理的结果读起来仍是 12 个孤立的短句,而**整篇一次交给模型**就得到连贯的段落。所以**停止录音后**,对整篇口述再做一遍:

- 等所有段都有了自己的整理结果,再把相邻的段按原文拼成一组(一组原文不超过 800 字;一般的口述就是一整组),整组交给整理模型,前文是这一组之前已整理的文字(约 300 字)。
- **停止即触发,不等待逐段任务排队完成**。取消尚在运行的逐段整理;`Draft.Recording` 从真变假时同步设置 `Polishing`,UI 通知尚未送达时,写入也已经被阻止。
- 先为完整原文生成主题/术语摘要,再润色。摘要只读,原文始终随润色请求提交,不能拿摘要代替原文。超出摘要输入预算的长文分块摘要后再合并,避免只读取前 2500 字。
- 每次润色最多 800 字原文;短口述整体提交,长口述按连续单位分组,所有组读取全文摘要,并携带前文约 300 字。最终润色使用独立的 `IDictationPolisher`,逐段预览仍使用 `ITextCleaner`;二者共享同一个整理角色、路由和本地模型。
- 整组结果过**同一道保真门禁**(对着原文)。通过:这一组段合并成**一个单位**,原文是它们拼起来的原文,整理结果是润色后的文本;用户仍可“用原话”回到原文。
- **整组被拒绝时对半拆开**,两半各自再试,直到只剩两段:一个听错的词只会让它所在的那一半失去润色,而不是整篇。模型回答被截断同样按这个办法缩小范围。服务不可用或已停止则放弃,说话时的整理结果照常使用。
- **写入要等润色做完**:润色期间草稿算“处理中”(`Draft.Polishing`),状态显示“正在整理…”,按回车会排队,润色一结束就写入;否则用户几乎总会写入润色前的文本。输出不是“整理”时(翻译、原话)不受影响。
- 不碰用户手改过的段、带排版的段(打字 / 粘贴来的文本)、已经润色过的段;同一组只尝试一次。没有经过录音的草稿不润色。
- 沿用原来的整理提示词。试过更强的“润色编辑”提示词(重新分段、加过渡),结果更书面,但会丢掉提问(“你觉得怎么样?”)和插入的信息点,被门禁拒绝得更多,所以没有采用。
- **不再递归拆分重试**。实测重复/噪声长文会因反复拒绝而串行调用多次模型,等待超过一分钟。默认要求的结果不通过检查时保留逐段结果;只有自定义要求无效时,额外重试一次默认要求。截断、不完整输出不得作为成功结果。
- **写入要等润色做完**:显示“正在润色全文… 此时写入会等润色完成”;按回车排队,结果完成且可用后才写入。录音中右侧标注“实时整理预览 · 停止后润色全文”,不把预览冒充最终结果;复制和导出同样遵循准备状态。关闭按钮只保留草稿,不会写入半成品。
- 单段录音也执行最终润色;重新录音后结合新增内容再次处理。旧请求受录音版本、原文版本、取消令牌约束,不能回写到新录音、源文本编辑或人工结果编辑中。
- 不覆盖人工编辑过的整理稿或译文,也不合并带排版/时间轴的单位。失败原因写不含原文的诊断日志;最终没有采用全文结果时,界面明确提示“已保留逐段结果”,不显示成功式提示。

实测(真实模型):你的 12 段口述(252 字)停止后 4 s 内合成一个连贯的段落并通过门禁;另一份带停顿的口述(218 字)合成两段,“Poker”按词表还原成“POC”,约 6 s。
实验不是 M2/16 GB 产品验收:本机 Qwen3.5 4B,252 字样本连贯成一段;包含两次推理及冷启动时约 10 s。1030 字重复/噪声压力样本,取消递归后最终阶段从约 79 s 降到约 16 s,但该样本最终结果被门禁拒绝,仍需逐段结果,不应声称润色质量已解决。摘要额外增加延迟,实测不总改善短文质量;这里遵循“先摘要再润色”的产品选择,保留明确的等待状态。

### 2.7 可编辑的全文润色要求

- **设置 → 语音与翻译 → 全文润色要求**:展示当前默认要求供直接编辑,失焦保存,支持一键恢复默认。只影响最终全文润色,不影响实时识别、逐段预览、翻译或隐私路由;下次唤起生效。
- 可编辑的是风格要求;输入格式、不得新增事实、保留提问/数字/条件等约束在固定提示词中,输出仍过保真检查。模型和检查均有局限,不能承诺任意自定义要求都能被安全执行。
- `PolishStyle` 是校验和默认值的唯一来源:10–1500 字符,允许正常换行/制表符,不允许控制字符。交互中无效的编辑显示错误、不覆盖最后有效配置;空白和“恢复默认”存为空字符串,以后内置默认升级可自动生效。
- 设置缺失、空白、过短、过长、损坏时解析为默认要求;整个 JSON 损坏沿用设置服务的默认恢复。格式合法但模型输出不可用的自定义要求,重试默认要求,成功后明确提示已切换默认;默认仍失败则保留预览并提示。
- 保留原文与“用原话”入口。摘要不写入结果,也不作为新的术语或事实来源。

### 2.6 逐段整理的两个失败模式

Expand Down Expand Up @@ -105,6 +115,16 @@
- 多显示器:限制按窗口所在屏幕的工作区和缩放计算。
- 钉住后窗口不再因内容高度变化而移动;放大或还原是唯一的复位方式(或下次唤出)。

### 3.4 清空与模型提示

- 原文栏右上角常驻“清空”文字按钮,复用 `NewDraftCommand` 和 `Cmd/Ctrl+N`,不新建第二套清空逻辑。它停止录音、清空当前原文/整理稿/译文/时间轴,取消待写入内容,保留目标、输出选择、偏好和历史列表。清空后焦点回原文框。
- 启动中的录音必须等启动/停止完成后再重置;清空期间显示“正在清空”,禁止重新开麦、写入或导入。停止操作共享进行中的任务。原生识别超时后线程可能仍在运行,因此识别会话捕获 `Draft.Epoch`,所有写操作拒绝旧代次,不能只依赖 UI 的取消标志。晚到的提交、转录和识别结果都不能写入新草稿。
- 模型提示是实际运行事实,不从设置值推断。`ModelUsage` 在语音实例创建、本地模型租约获取和云端服务响应边界报告;预热不冒充使用。录音开始、有效模型变化时提示约 5 秒,不长期占用正文空间;点击或悬停“本地/云端”可再次查看详情。
- 待准备模型和已实际加载模型分开;切换时显示新请求与上次实际模型。云端只回显请求别名时明确标注“服务端尚未回传模型名”,不把部署别名当成确认的后端模型。
- 模型职责分开:X-ASR 480 ms 实时识别 + Qwen3-ASR 0.6B 二遍校正;Qwen3.5 4B 整理/摘要/润色;Hy-MT2 1.8B/7B 翻译。“快速/质量”只影响翻译,不更换语音识别器。
- 查到两个切换问题:旧模型尚有请求时直接停止服务;打开的 Composer 保留旧模型需求和译文缓存。现在等待旧租约释放再切换,并刷新当前 Composer 的需求及非人工译文;源文本与人工译文保留。
- 已用真实本地服务执行 1.8B → 7B → 1.8B,三次响应分别报告对应 GGUF 文件,不是只核对配置枚举。Windows 原生交互仍需真机验收。

## 4. 验证与已知限制

已验证:
Expand All @@ -123,5 +143,5 @@
## 5. 开放问题

1. 大小要不要按应用记住(在 Slack 里紧凑,在编辑器里用大窗口)?成本很小:`AppChoice` 加一个可选的“放大”标记。
2. `PassagePauseMs`(3000 ms)和 `PassageMaxChars`(200 字)需要在真人口述上调。停顿越长,说话时右边出结果越晚,但段越完整;真正的连贯靠停止后的全文润色。
2. `PassagePauseMs`(2000 ms)和 `PassageMaxChars`(200 字)仍需真人口述验证;实时预览不再承担最终连贯性的承诺。
3. 门禁对“整段保留原话”的提示文字目前沿用 “已保留原话”;逐句降级后部分保留的情况不提示。
5 changes: 3 additions & 2 deletions src/LiveLingo.Core/Cleanup/CleanupScheduler.cs
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ private void OnChanged()

ObserveEdits(_time.GetUtcNow());

if (_inFlight is { } flight && !_session.Read(draft => draft.IsCurrent(flight.Ticket, UnitKind.Clean)))
if (_inFlight is { } flight && _session.Read(draft => draft.Polishing || !draft.IsCurrent(flight.Ticket, UnitKind.Clean)))
flight.Cancel.Cancel();

StartWorkerIfNeeded();
Expand Down Expand Up @@ -194,7 +194,8 @@ private async Task RunAsync()
/// <summary>The next unit to clean, or how long typing must stay still before the last unit is cleaned.</summary>
private ((JobTicket Ticket, CleanupRequest Request)? Next, TimeSpan? Wait) Find(Draft draft, DateTimeOffset now)
{
if (draft.Snapshot is not null || draft.Output != OutputKind.Clean)
// The polish of the whole dictation reads the original text and goes first; units it could not polish are cleaned after it.
if (draft.Snapshot is not null || draft.Output != OutputKind.Clean || draft.Polishing)
return (null, null);

var units = draft.Units;
Expand Down
30 changes: 30 additions & 0 deletions src/LiveLingo.Core/Cleanup/IDictationPolisher.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
namespace LiveLingo.Core.Cleanup;

/// <param name="Target">The original text of the passages being polished, in order.</param>
/// <param name="Preceding">Read-only: the already finished text before the target.</param>
/// <param name="Summary">Read-only summary of the whole dictation; empty when the polish sees the whole dictation itself.</param>
/// <param name="Glossary">Terms that must be written as given.</param>
/// <param name="Style">The polish requirements (<see cref="PolishStyle"/>).</param>
public sealed record PolishRequest(
string Target,
string Preceding,
string Summary,
IReadOnlyList<string> Glossary,
string Style);

/// <summary>
/// The polish of a finished dictation: a longer text than one passage, read in one piece. Unlike
/// <see cref="ITextCleaner"/> (which fixes one passage while the speaker is still talking) it may restructure
/// sentences across passage seams.
/// </summary>
public interface IDictationPolisher
{
/// <summary>A short summary of what the text is about and which terms occur in it, to be read with each part of a long text.</summary>
/// <exception cref="HttpRequestException">The inference endpoint could not serve the request.</exception>
/// <exception cref="InvalidDataException">The model output was incomplete.</exception>
Task<string> SummarizeAsync(string text, CancellationToken ct);

/// <exception cref="HttpRequestException">The inference endpoint could not serve the request.</exception>
/// <exception cref="InvalidDataException">The model output was incomplete.</exception>
Task<string> PolishAsync(PolishRequest request, CancellationToken ct);
}
Loading
Loading