Repository navigation
Add Glow and Wave voice agent visualizations to StreamChatAI - #4338
martinmitrevski wants to merge 4 commits into
Conversation
AIVoiceGlowView and AIVoiceWaveView show what a voice agent is doing (idle, connecting, listening, thinking, speaking). They port the web voice agent demo's shaders to Metal, compiled at runtime so building the SDK needs no Metal toolchain, with its per-frame easing, tap bounce and ripple, hover, live audio level and Reduce Motion support. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage |  |
Docstring coverage is 16.22% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 74 functions across 21 files. | Write docstrings for the functions missing them to satisfy the coverage threshold. |
-
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title clearly and concisely describes the primary change: adding Glow and Wave voice agent visualizations to StreamChatAI. Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request.
- Fix all pre-merge checks with AI
✨ Finishing Touches: 💡 1
-
📝 Generate docstrings: 💡
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
Generated by 🚫 Danger |
SDK Performance
|
| // Starts compiling the shaders, unless they are compiled or compiling. | ||
| func prepare() { | ||
| guard loading == nil else { return } | ||
| let device = UncheckedSendable(device) |
There was a problem hiding this comment.
Here MTLDevice is Sendable. Should be OK then. We should really avoid workarounds like UncheckedSendable because it loses the idea of what Swift 6 brings.
There was a problem hiding this comment.
yes, we should teach our skills not to do this
There was a problem hiding this comment.
Done in f775f74: the UncheckedSendable box and the detached task are gone, and the Metal objects go through as they are.
One catch: the Xcode 16.1 job in CI builds StreamChatAI too, and the iOS 18 SDKs don't mark MTLDevice, MTLLibrary or MTLRenderPipelineState Sendable yet (no NS_SWIFT_SENDABLE in their headers; every iOS 26 SDK has it). So for compilers before 6.2 Metal is imported with @preconcurrency, the same way DatabaseContainer imports CoreData. On Xcode 26+ it's a plain import Metal.
There was a problem hiding this comment.
Correction to the above: @preconcurrency wasn't enough. The Xcode 16.1 job failed because Swift 6.0 still rejects sending the main actor's MTLDevice to Metal's async APIs when the SDK doesn't mark it Sendable. In 95f81e0, Xcode 16 builds call the synchronous APIs on the main actor, so the device never leaves it, and Xcode 26+ keeps the async ones. There's no @preconcurrency or unchecked anything now. Only CI builds StreamChatAI with Xcode 16 (SPM needs Xcode 27 for it), and that job is green.
| on device: any MTLDevice | ||
| ) throws -> UncheckedSendable<[VoiceVisualizationStyle: any MTLRenderPipelineState]> { | ||
| let library = try device.makeLibrary(source: VoiceVisualizationShaders.source, options: nil) | ||
| var pipelines: [VoiceVisualizationStyle: any MTLRenderPipelineState] = [:] |
There was a problem hiding this comment.
Yes, the pipelines now come back as a plain [VoiceVisualizationStyle: any MTLRenderPipelineState] (f775f74).
| loading = Task { | ||
| do { | ||
| let compiled = try await Task.detached(priority: .userInitiated) { | ||
| try Self.compile(on: device.value) | ||
| }.value | ||
| pipelines = compiled.value | ||
| } catch { | ||
| log.error("Voice visualizations can't render", error: error) | ||
| } | ||
| } |
There was a problem hiding this comment.
Would it make sense to use the async version of the makeRenderPipelineState?
There was a problem hiding this comment.
Good call, done in f775f74. Both makeLibrary(source:options:) and makeRenderPipelineState(descriptor:) use the async variants now, so Metal compiles on its own threads and the main actor only awaits the result. That also removed the need for Task.detached.
| // Draws a voice visualization's shader over a whole render target. All visualizations share | ||
| // one. The shaders compile once, off the main thread, and nothing is drawn until they have. |
There was a problem hiding this comment.
Why not require a metal toolbox for compilation (it is just additional download)? Good for the performance of the SDK when it comes pre-compiled?
There was a problem hiding this comment.
I measured it before deciding. On an iPhone 15 Pro, compiling both shaders took 78–167 ms the first time the device saw this source, and 0 ms after that, because Metal caches libraries compiled from source on disk across launches. The compile runs asynchronously and the view fades in on its first frame, so the cost is roughly 0.1 s once per install, and never on the main thread.
Requiring the toolchain would cost everyone who builds StreamChatAI from source, which with SPM means every customer. Xcode 26 and later don't include the Metal compiler, and a .metal file in the target fails the build with cannot execute tool 'metal' due to missing Metal Toolchain until xcodebuild -downloadComponent MetalToolchain is run. That applies to every dev machine and CI runner, ours included. It seemed like a lot of friction to save ~0.1 s once per install.
If we want precompiled shaders anyway, there's a middle ground: commit prebuilt .metallibs (one for device, one for simulator) as resources, load them with makeLibrary(URL:), and regenerate them with a script when the shaders change, like we do for L10n. Customers wouldn't need the toolchain, and we'd maintain a binary artifact. Happy to do that if you prefer it.
Metal's devices, libraries and pipelines are Sendable in the Xcode 26 SDKs, so the shaders compile through the async makeLibrary and makeRenderPipelineState on the main actor, without the unchecked Sendable box or a detached task. Xcode 16's SDKs lack the annotations, so they import Metal with @preconcurrency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Xcode 16's SDKs do not mark MTLDevice Sendable, and Swift 6.0 rejects sending the main actor's device to Metal's async APIs even with a @preconcurrency import. Builds with those SDKs now call the synchronous APIs on the main actor; Xcode 26 and later keep the async ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Public Interface+ public enum AIVoiceAgentState: String, CaseIterable, Sendable
+
+ case idle
+ case connecting
+ case listening
+ case thinking
+ case speaking
+ public struct AIVoiceGlowView: View
+
+ public var body: some View
+
+
+ public init(state: AIVoiceAgentState,colors: [Color]? = nil,expressivity: Double = 1,audioLevel: Double? = nil)
+ public struct AIVoiceWaveView: View
+
+ public var body: some View
+
+
+ public init(state: AIVoiceAgentState,colors: [Color]? = nil,expressivity: Double = 1,audioLevel: Double? = nil)
@MainActor public final class Colors
-
+ public lazy var voiceVisualization: [UIColor]
-
+
- public nonisolated init(tokens: DesignSystemTokens = DesignSystemTokens())
+
+ public nonisolated init(tokens: DesignSystemTokens = DesignSystemTokens()) |
SDK Size
|
StreamChatAI XCSize
Show 14 more objects
|
|
This branch has not been deployed
3 participants
🔗 Issue Links
Resolves https://linear.app/stream/issue/IOS-2189/add-glow-and-wave-voice-agent-visualizations-to-streamchatai.
🎯 Goal
Give voice agents a visual presence. The voice agent demo defines seven visualization styles and the design bar for them. This brings the first two to iOS as reusable SwiftUI components: Glow, the default, and Wave. Reference implementation: GetStream/stream-voice-agent-demo.
📝 Summary
AIVoiceGlowViewandAIVoiceWaveViewin StreamChatAI, with the same parameters:state,colors,expressivityandaudioLevel.AIVoiceAgentState:idle,connecting,listening,thinking,speaking. Changing it eases from one motion into the next.AIAppearance.Colors.voiceVisualization, the default palette of one to five colors. It defaults to the accent.🛠 Implementation
mod,powof a negative base, and reversedsmoothstepedges. The simplex noise was checked numerically against a float32 run of the GLSL, within 0.0012..metalfile would break the build for SPM customers and CI. The Metal source lives in a Swift string instead (VoiceVisualizationShaders). It is compiled once, off the main thread, the first time a visualization appears. The view fades in over 0.3s when its first frame is ready.VoiceVisualizationDriverports the reference's per-frame easing, its three integrated clocks (time,flowSpin,orbSpin) and the expressivity scaling. The reference eases by a fixed share per frame, which settles twice as fast at 120 Hz. Here each share is converted to time, so motion is the same on any display. Long frames are clamped to 50 ms.ease-outover 340 ms. Touches outside the globe or line pass through to views behind. Hover usesUIHoverGestureRecognizer, so it only ever comes from a pointer.MTKViewkeeps working on iOS 15, gives control over the render loop, and outputs premultiplied alpha as the web does. The loop pauses off screen and in the background. Glow renders at 2x like the web; Wave renders at screen scale to keep its line crisp.UncheckedSendablebox. Older SDKs, including CI's Xcode 16.1, may not mark Metal protocolsSendable, and Metal documents these objects as thread-safe.🎨 Showcase
Glow, light and dark mode, in the Vision Agents Swift demo:
🧪 Manual Testing Notes
audioLevel(0 to 1) to drive the motion from live audio.These were tested in the Vision Agents Swift demo's Voice tab against a local router, on the simulator.
☑️ Contributor Checklist
docs-contentrepo🤖 Generated with Claude Code
Summary by CodeRabbit