Conversation
Reference local Whisper (Whisper.net + Whisper.net.Runtime) and NAudio for on-device audio transcription, and declare the microphone device capability needed for live microphone capture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Persist the user's chosen Whisper model (defaults to balanced multilingual base), trading transcription speed for accuracy and language coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implement on-device transcription backed by Whisper.net (whisper.cpp), running on the CPU packaged or unpackaged on x64/arm64: - File transcription decodes any Media Foundation audio to 16 kHz mono via NAudio and streams Whisper segments as they are recognized, cancellably. - Selectable model (tiny.en / base.en / base / small) via WhisperModelChoice, read from settings; the shared WhisperFactory reloads when the choice changes. - VAD-gated live transcription (LiveAudioTranscriber) from microphone or system-audio loopback: Silero VAD finds speech regions and only sends a region to Whisper once trailing silence marks the utterance complete, so silence is skipped and phrases cut on natural boundaries. - AudioDebugLog writes a timestamped diagnostic log for the transcription path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route audio files (CLI arg, File > Open, drag/drop) to on-device transcription, streaming each Whisper segment into the editor via a non-blocking, cancellable status bar that preserves already-transcribed text. Add a bottom-bar live-transcription toggle with a right-click menu to pick the capture source (microphone or system audio) and the transcription model, both restarting an active session on change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reference local Whisper (Whisper.net + Whisper.net.Runtime) and NAudio for on-device audio transcription, and declare the microphone device capability needed for live microphone capture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Persist the user's chosen Whisper model (defaults to balanced multilingual base), trading transcription speed for accuracy and language coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implement on-device transcription backed by Whisper.net (whisper.cpp), running on the CPU packaged or unpackaged on x64/arm64: - File transcription decodes any Media Foundation audio to 16 kHz mono via NAudio and streams Whisper segments as they are recognized, cancellably. - Selectable model (tiny.en / base.en / base / small) via WhisperModelChoice, read from settings; the shared WhisperFactory reloads when the choice changes. - VAD-gated live transcription (LiveAudioTranscriber) from microphone or system-audio loopback: Silero VAD finds speech regions and only sends a region to Whisper once trailing silence marks the utterance complete, so silence is skipped and phrases cut on natural boundaries. - AudioDebugLog writes a timestamped diagnostic log for the transcription path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route audio files (CLI arg, File > Open, drag/drop) to on-device transcription, streaming each Whisper segment into the editor via a non-blocking, cancellable status bar that preserves already-transcribed text. Add a bottom-bar live-transcription toggle with a right-click menu to pick the capture source (microphone or system audio) and the transcription model, both restarting an active session on change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Added EtwShowTranscribe setting to enable/disable audio transcription features. Introduced "Show Transcribe" toggle in BottomBarSettings and new menu items in EditTextWindow for transcription actions, shown only if supported and enabled. Synced UI and settings for transcription visibility, source, and model selection. Updated code generation version in Settings.Designer.cs.
Text-Grab.csproj listed Microsoft.WindowsAppSDK.AI/.Foundation/.Runtime/.WinUI twice at differing versions, producing NuGet warning NU1504. The newer set is not internally consistent (Runtime 2.4.0 expects InteractiveExperiences 2.1.6 but 2.1.3 is resolved), so keep the matched 2.2.x components. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The .NET 10 SDK no longer supports running Microsoft.Testing.Platform test projects through the VSTest target, which broke `dotnet test` in CI. Opt into the new runner via global.json and drop the VSTest-only packages (Microsoft.NET.Test.Sdk, xunit.runner.visualstudio, coverlet.collector). OutputType is now Exe as required by the xunit.v3 MTP entry point. xunit.v3 is pinned to 3.2.2 because Xunit.StaFact, used by every WpfFact test, has no build compatible with xunit.v3 4.0.0 yet. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With Microsoft.Testing.Platform, dotnet test no longer accepts a positional project path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Enabled capturing and transcribing both microphone and system audio at once via new LiveCaptureSource option and UI updates. Refactored LiveAudioTranscriber for multi-channel input, mixing, and buffer management. Improved VAD/transcription logic for mixed sources. Enhanced concurrency with semaphore locks. Updated UI feedback and error handling. Added quantization-aware Whisper model file handling.
Replaces WindowsAiUtilities/TextRewriter-based translation with new WinAiTranslator, managing on-device Phi Silica model and LAF unlock. Adds batched and streaming translation, improved error handling, and user feedback. Moves language heuristics to LanguageHeuristics.cs. Project now supports LAF tokens via build/env vars. Updates all translation entry points for new API. Cleans up old logic, adds LimitedAccessFeatureUtilities, and improves UI batching and resource management.
Enables secure injection of Windows AI Language Model (LAF) unlock tokens via LAF_TOKEN and LAF_PUBLISHER_ID environment variables, sourced from GitHub Actions secrets in Release.yml and buildDev.yml, and passed to MSBuild publish steps. Updates Text-Grab.csproj to default LafToken and LafPublisherId from environment variables for local and CI builds. Adds Configuring-LAF-Environment-Variables.md to document setup and usage, stressing token secrecy. Builds without secrets still succeed but omit on-device text AI features.
Refactored all Windows AI language model logic into WinAiLanguageModel for shared creation, availability checks, and inference queuing. Standardized error and result handling with WinAiFailure and WinAiGenerationResult. Updated WinAiTranslator, WindowsAiUtilities, and regex extraction to use the shared model and new result types. Improved error reporting in EditTextWindow and clarified documentation throughout.
Added "Summarize as Meeting Notes" to EditTextWindow with UI and handler. Implemented WinAiMeetingNotes utility for splitting, summarizing, and merging long text using Windows AI (Phi Silica). Added unit tests for text splitting. Refactored response cleaning to shared CleanResponse method in WinAiLanguageModel. Updated docs and comments. Improved error handling and user feedback for meeting notes generation.
When the out-of-process Windows AI runtime connection is lost (e.g., "The RPC server is unavailable"), the code now disposes and recreates the language model, retries the request once, and only fails if restart fails. This logic is centralized in `WinAiLanguageModel.GenerateAsync` and `RunWithModelAsync`, using a new `IsConnectionLost` helper. The unlock cache in `LimitedAccessFeatureUtilities` is reset on restart to avoid caching transient failures. All AI features now use the shared model and recovery logic. Public APIs allow manual model restarts. Exception handling distinguishes recoverable from unrecoverable errors, and error messages are clarified.
Wires the existing "New Window with Selected Text" action to Ctrl+N so it doesn't require the menu, and shows the shortcut in the menu item. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously these AI actions overwrote the source text in place. Now the source stays untouched and the generated result opens in a new Edit Text window, matching the existing "open in new window" pattern. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds better Local translation
TranscribeAudioFileAsync now accepts an optional hotWords string applied per-call via Whisper's WithPrompt/WithCarryInitialPrompt, plus GetAudioFileInfo (size/duration, no decode) and GetAudioFileFilter for an upcoming file-picker UI. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
IsPickerMode makes a pick always write into DestinationTextBox and close the window, regardless of EditWindowToggleButton, for callers that want QSL purely as a value picker rather than its ETW insert/clipboard flow. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New Capture menu item opens OpenMediaWindow: pick an audio file, add hot words for that transcription (typed directly or picked from Quick Simple Lookup via its new picker mode), and see size/duration/model info before starting. Video is a disabled placeholder for a future, unrelated feature. Hot words are transient per-transcription, not persisted. TranscribeAudioFilesAsync is now internal and takes an optional hotWords parameter to support this. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
LookupItem.ToCSVString writes rows as unquoted "ShortValue,LongValue", so a LongValue containing commas (e.g. a saved hot-words batch) split into more than two cells on reload. The parser rejoined those cells with a space, silently losing the commas. Rejoin with the same delimiter that was used to split when reading a CSV row, so the original value round-trips exactly; tab-delimited rows (typed entry, clipboard paste) keep their existing space-joined behavior. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a persisted NotifyOnTranscriptionComplete setting (default on) and a toggle in the Open Audio / Video window, so long-running file transcriptions can notify the user via toast when they finish instead of requiring the editor window to stay in focus.
Adds a persisted IncludeTimecodesInTranscription setting (default off) and a toggle in the Open Audio / Video window. When enabled, each Whisper segment is prefixed with its start time (e.g. [01:23]) and placed on its own line, using timing data Whisper.net already provides per segment.
GetTextFromTabledWordBorders only inserted a tab between columns when a capture spanned more than one detected row, so a single-row capture (e.g. grabbing one table row at a time) with genuinely distinct columns produced no separator at all, concatenating adjacent column values into one glued word. Column IDs are already established by the width-based clustering independent of row count, so drop the row-count gate and always tab-separate distinct columns. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TJcEHobwisx65fiBn57RhB
Table detection only ever read from wordBorders, but native-text PDF pages rendered line-level overlays instead of populating it, so Table mode silently did nothing for the common case of a selectable-text PDF (scanned/image PDFs already worked via the normal OCR path). Expose the native per-word rects PdfDocumentRenderer already computes internally and use them to build per-word borders when Table mode is active, switching back to line overlays for plain reading otherwise. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TJcEHobwisx65fiBn57RhB
The table outline drawn in Grab Frame's Table mode now has four corner handles that can be dragged to crop the region considered for table detection. Releasing a handle re-runs the table clustering algorithm using only the word borders whose center falls inside the repositioned boundary, instead of every word border in the frame. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TJcEHobwisx65fiBn57RhB
Recent text history had no display cap and could show up to 100 items (the on-disk retention limit). Recent grabs/PDFs relied only on storage retention to stay at 10; now the menu population enforces the cap directly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TJcEHobwisx65fiBn57RhB
Several soft/automatic paths (middle-mouse window drag, the Refresh/OCR Frame button, and the Freeze toggle) called ResetGrabFrame() or UnfreezeGrabFrame() unconditionally, silently wiping any word borders the user had corrected, moved, merged, split, or manually divided into table rows/columns. Window resize/move and the content-change watcher already skipped resetting a frozen frame, but nothing protected the middle-mouse path, and table-divider edits never froze the frame in the first place. Add a hasUnsavedWordEdits flag set at every real edit site and cleared by ResetGrabFrame(). Gate the three unconditional reset entry points behind a Discard/Cancel confirmation that only appears when there is something to lose, and record the existing word borders as an undoable removal (ResetGrabFrameWithUndo) so a confirmed or automatic reset can still be undone with Ctrl+Z. Manual table-divider edits now freeze the frame like word edits already did, so the existing frozen-frame guards on resize/content-change protect them too. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TJcEHobwisx65fiBn57RhB
hasUnsavedWordEdits only became true through UndoableWordChange, but typing in a word's edit textbox never called it: the textbox binds Text to Word/DisplayText live (TwoWay, UpdateSourceTrigger= PropertyChanged) with no commit step, so a manually retyped correction was invisible to the frame's dirty tracking and undo stack right up until this point. Capture the word's value on GotFocus and diff it against the current value on a new LostFocus handler, calling UndoableWordChange when it changed so typed corrections get the same undo/dirty-flag treatment as the existing auto-fix menu commands. As a second line of defense for paths that don't go through a focus change, also flag the frame dirty from the debounced TextChanged bubble (WordChanged) that already fires ~300ms into typing, ahead of any LostFocus commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TJcEHobwisx65fiBn57RhB
Editor cleanup pass: XAML attribute ordering and single-line bindings in GrabFrame.xaml, and redundant parentheses removed in GrabFrame.xaml.cs. No behavior change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012dZXjFSgaUKvv1YpDB8VhL
The corner-handle table region was wiped by every ResetGrabFrame (auto refresh, manual refresh, window move) and by PDF page navigation, so a table that repeats at the same spot on every page had to be re-cropped page after page. The override now survives those paths and is only dropped when the user clears table edits (Escape) or loads different content. Also: a page where the region contains no words falls back to the whole page for that analysis but keeps the region for the next page; the outline is drawn at the persisted region (not the tight bounds of the words inside) so it is visible where it will apply and the handles start their drag from where they are drawn; the region scales with a frozen frame's content; and dragging it no longer flags unsaved edits since a refresh no longer discards it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012dZXjFSgaUKvv1YpDB8VhL
Right-clicking the Grab button on a loaded multi-page PDF offers "Grab multiple pages...", which asks for a page range (defaulting to the current page through the last), all / odd-only / even-only pages, and whether to put a blank line between pages (off by default in Table mode so rows run together). Each page is run through the normal single-page pipeline — page render, word borders, table analysis with the persisted table boundary, active Grab Template — and the joined text opens in a fresh Edit Text Window, in spreadsheet mode when grabbed as a table. GrabExecuted is split into ResolveGrabOutputTextAsync / DeliverGrabbedText so the per-page text resolution is shared with the multi-page path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012dZXjFSgaUKvv1YpDB8VhL
Adds a checkable "Ignore repeated headers and footers" item to the Grab button's context menu (persisted as GrabFrameIgnoreRepeatedHeadersFooters, on by default). When on, a multi-page grab first reads every page in the range and snapshots each word border / native-PDF text line with its position, then drops the running headers and footers before assembling each page's text. OCR results are cached per page so scanned pages are only OCR'd once across the two passes. RepeatedPageElementDetector (Text-Grab.Core) flags an element when it sits in the top or bottom 20% of the page, an element at nearly the same spot on at least a third of the other pages has the same text — after collapsing case/whitespace and replacing digit runs, so "Page 3 of 12" matches "Page 4 of 12", plus an edit-distance fuzzy match for OCR noise — and it is not attached to content that is being kept. That last rule protects tables: a repeated column-header row has data rows directly under it, and a numeric cell that matches across pages shares its line with cells that differ; the check is transitive so a varying cell anchors its row, the row anchors the header above it, and so on. A running header or footer has a margin gap and nothing varying beside it, so it is still dropped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012dZXjFSgaUKvv1YpDB8VhL
Detects when Local AI tasks produce output identical to input and avoids opening new windows or overwriting text, showing a "No Changes" dialog instead. Refactored DeliverLocalAiResultAsync and PerformLocalAiTransformAsync to return a bool for change detection. Added LocalAiResultUtilities.IsUnchanged for normalized comparison and comprehensive unit tests for edge cases.
TheJoeFin
added
enhancement
Serialize creation, inference, recovery, and release under a shared lease while preserving orderly shutdown. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Serialize live transcription transitions and drain final dispatcher-delivered phrases before stopping or closing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Write in-place summary and meeting-note output to the active spreadsheet cell without changing other cells. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Merge image-region OCR with native PDF words and serialize stale-safe asynchronous overlay redraws. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
All WaitAsync calls with TestTimeout now also pass TestContext.Current.CancellationToken, allowing test operations to be canceled if the test context is canceled. This improves test reliability and ensures responsiveness to test aborts or timeouts.
Add selectable production and beta routines, collect consistently named release artifacts, fail on publish errors, and omit debug symbols from self-contained packages. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Web search, URL launch, and QR code now use spreadsheet cell selection when in Spreadsheet mode, and commands that operate on flat document text (isolate selection, delete all instances, insert on every line, unstack, split on selection) are disabled in that mode since they have no meaningful per-cell equivalent. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Lets users pick a default transcription model, download or delete individual models to manage disk space, and jump to the Languages page for OCR language packs. Adds DownloadModelAsync/DeleteModel helpers to AudioTranscriptionUtilities to support the new page. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tion Adds medium.en, medium, large-v3-turbo, and large-v3 as file-transcription model choices, and splits the transcription model setting in two: AudioTranscriptionModel for file/clip transcription (where a progress bar and cancel button make a slow model tolerable) and LiveTranscriptionModel for live sessions (restricted to WhisperModelInfo.LiveEligibleModels so a large model never gets loaded somewhere it can't keep up with speech). The live transcription flyouts (Capture menu and mic context menu) are now built dynamically from WhisperModelInfo instead of static XAML items with hand-synced check marks, with a "More models..." entry linking to the redesigned Models settings page. Picking an undownloaded live model now kicks off a background download with a completion toast. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously the shared WhisperFactory stayed cached in RAM indefinitely after a transcription finished, until a different model was selected. Now both TranscribeAudioFileAsync and LiveAudioTranscriber release the factory once they're done with it (on success, error, or cancellation), freeing the loaded model's memory; the next transcription just reloads it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
4.16
Audio and video transcription
Windows AI and Local AI
WinAiTranslator, prevent re-entrant Grab Frame translation, and show post-grab translation errors only for genuine failures.Grab Frame, tables, and PDFs
Edit Text, capture, and productivity
Ctrl+Nto open selected text in a new Edit Text window, and update the Markdown editing options/styles.Appearance, menus, and performance
Architecture, testing, and maintenance
Text-Grab.Coreand Windows-specific functionality intoText-Grab.Core.Windows; introduceTests.CoreandTests.Core.Windows, enforce the tier boundaries in CI, and migrate the relevant models, OCR, imaging, capture, clipboard, language, AI, speech, storage, utility, and pattern-processing code.RectangleFas the Core geometry type, add theITextGrabSettingsseam, remove dead ResultTable and WinUI-dependent code, and read the app version from the entry assembly.dotnet test --projectin CI; update Windows App SDK and PdfPig, remove duplicate Windows App SDK references, and refresh dependent API and UI styles.