Skip to content

Publish enriched Lody ACP usage and compaction activity #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
leynos wants to merge 1 commit into from
Draft

Publish enriched Lody ACP usage and compaction activity #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ vtcode update # self-update

## Providers

VT Code supports 24+ LLM providers out of the box, plus any custom API via `[[custom_providers]]`. Custom providers can declare their model capability with `context_window` (tokens), an optional `api_format` hint, and capability defaults such as `supports_tools` or `supports_vision`. `api_format` accepts `auto`, `openai-chat`, `openai-responses`, or `anthropic-messages`; when omitted VT Code preserves legacy autodetection, while an explicit value is honored and will not silently fall back. Per-model sparse profiles under `custom_providers.profiles."<model-id>"` can override these defaults for an already-allowed model — profiles do not add models to the picker. The configured `context_window` still drives UI context sizing, compaction thresholds, and preflight token checks.
VT Code supports 24+ LLM providers out of the box, plus any custom API via `[[custom_providers]]`. Custom providers can declare their model capability with `context_window` (tokens), opt-in pricing in USD per million tokens, an optional `api_format` hint, and capability defaults such as `supports_tools` or `supports_vision`. `api_format` accepts `auto`, `openai-chat`, `openai-responses`, or `anthropic-messages`; when omitted VT Code preserves legacy autodetection, while an explicit value is honored and will not silently fall back. Per-model sparse profiles under `custom_providers.profiles."<model-id>"` can override these defaults for an already-allowed model — profiles do not add models to the picker. The configured `context_window` still drives UI context sizing, compaction thresholds, and preflight token checks. In ACP usage updates, `costUSD` is emitted only when both input and output pricing are explicitly configured.

| Category | Providers |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
Expand All @@ -113,6 +113,13 @@ base_url = "https://llm.corp.example/v1"
api_key_env = "MYCORP_API_KEY"
model = "gpt-5-mini"
context_window = 256000 # optional; defaults to 128000 tokens

[custom_providers.pricing]
# USD per million tokens; costUSD requires both input and output rates.
input_per_million_usd = 0.15
output_per_million_usd = 0.50
# cache_read_per_million_usd = 0.03
# cache_write_per_million_usd = 0.00
```

`context_window` is the provider capability in tokens. The separate
Expand Down
42 changes: 35 additions & 7 deletions crates/codegen/vtcode-acp/src/zed/agent/compaction.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,18 +79,27 @@ impl ZedAgent {
return Ok(false);
}

let (thread, session_id, mut history, mut auto_compact_suppressed) = {
let (thread, acp_session_id, mut history, mut auto_compact_suppressed) = {
let data = session
.data
.lock()
.map_err(|error| anyhow::anyhow!("ACP session lock poisoned: {error}"))?;
(
data.thread.clone(),
data.session_id.to_string(),
data.thread.messages(),
data.auto_compact_suppressed,
)
(data.thread.clone(), data.session_id.clone(), data.thread.messages(), data.auto_compact_suppressed)
};
let activity = super::lody_activity::CompactionActivity::begin(&acp_session_id, prompt_tokens);
match activity.started_update() {
Ok(update) => {
if let Err(error) = self.send_update(&acp_session_id, update).await {
warn!(%error, session_id = %acp_session_id, "Failed to publish ACP compaction start update");
}
}
Err(error) => {
warn!(%error, session_id = %acp_session_id, "Failed to serialize ACP compaction start update")
}
}

let compaction_result: Result<bool> = async {
let session_id = acp_session_id.to_string();
let original_len = history.len();
let force_compaction = admission_budget.is_some_and(|budget| prompt_tokens >= budget)
&& configured_threshold.is_none_or(|threshold| prompt_tokens < threshold);
Expand Down Expand Up @@ -204,6 +213,25 @@ impl ZedAgent {
"Applied automatic ACP conversation compaction"
);
Ok(true)
}
.await;

let used_tokens_after = compaction_result
.as_ref()
.ok()
.map(|_| estimated_prompt_tokens(&self.resolved_messages(session), tools));
let failure_reason = compaction_result.as_ref().err().map(ToString::to_string);
match activity.finished_update(used_tokens_after, failure_reason.as_deref()) {
Ok(update) => {
if let Err(error) = self.send_update(&acp_session_id, update).await {
warn!(%error, session_id = %acp_session_id, "Failed to publish ACP compaction terminal update");
}
}
Err(error) => {
warn!(%error, session_id = %acp_session_id, "Failed to serialize ACP compaction terminal update");
}
}
compaction_result
}
}

Expand Down
129 changes: 104 additions & 25 deletions crates/codegen/vtcode-acp/src/zed/agent/handlers.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -298,7 +298,14 @@ struct ProviderErrorTelemetry<'a> {
}

fn provider_error_telemetry(error: &LLMError) -> ProviderErrorTelemetry<'_> {
let LLMError::Network { metadata: Some(metadata), .. } = error else {
let metadata = match error {
LLMError::Authentication { metadata, .. }
| LLMError::RateLimit { metadata }
| LLMError::InvalidRequest { metadata, .. }
| LLMError::Network { metadata, .. }
| LLMError::Provider { metadata, .. } => metadata.as_deref(),
};
let Some(metadata) = metadata else {
return ProviderErrorTelemetry::default();
};
ProviderErrorTelemetry {
Expand Down Expand Up @@ -448,6 +455,7 @@ async fn generate_with_retry(
request: LLMRequest,
runtime: &ProviderRequestRuntime,
cancellation: &super::super::types::SessionCancellation,
notice_target: Option<(&ZedAgent, &acp::SessionId)>,
) -> Result<LLMResponse, ProviderCallError> {
let policy = runtime.retry_policy();
let mut attempt_index = 0;
Expand Down Expand Up @@ -484,10 +492,18 @@ async fn generate_with_retry(
let decision = policy.decision_for_llm_error(&error, attempt_index);
telemetry.failed(runtime, attempt_index, retry_disposition(&decision), &error);
drop(permit);
let retry_delay = decision
.retryable
.then(|| decision.delay.unwrap_or_else(|| policy.delay_for_attempt(attempt_index)));
if let Some((agent, session_id)) = notice_target {
agent
.publish_rate_limit_notice(session_id, runtime.provider_name(), &error, retry_delay)
.await;
}
if !decision.retryable {
return Err(ProviderCallError::Failed(error.to_string()));
}
let delay = decision.delay.unwrap_or_else(|| policy.delay_for_attempt(attempt_index));
let delay = retry_delay.unwrap_or_else(|| policy.delay_for_attempt(attempt_index));
info!(
provider = runtime.provider_name(),
next_attempt = attempt_index + 2,
Expand Down Expand Up @@ -718,6 +734,7 @@ fn advertised_agent_capabilities(has_subagent_controller: bool, background_enabl
.list(acp::SessionListCapabilities::new())
.resume(acp::SessionResumeCapabilities::new());
super::lody_usage::add_lody_usage_capability(&mut capabilities);
super::lody_activity::add_lody_compaction_capability(&mut capabilities);
if has_subagent_controller {
super::lody::add_lody_subagent_management_capability(&mut capabilities, background_enabled);
}
Expand Down Expand Up @@ -1078,6 +1095,17 @@ async fn run_prompt(agent: Arc<ZedAgent>, args: PromptRequest) -> Result<PromptR
let decision = policy.decision_for_llm_error(&error, attempt_index);
telemetry.failed(&provider_runtime, attempt_index, retry_disposition(&decision), &error);
drop(permit);
let retry_delay = decision
.retryable
.then(|| decision.delay.unwrap_or_else(|| policy.delay_for_attempt(attempt_index)));
agent
.publish_rate_limit_notice(
&args.session_id,
provider_runtime.provider_name(),
&error,
retry_delay,
)
.await;
if !decision.retryable {
return Ok(finish_failed_provider_turn(
&agent,
Expand All @@ -1089,7 +1117,7 @@ async fn run_prompt(agent: Arc<ZedAgent>, args: PromptRequest) -> Result<PromptR
)
.await);
}
let delay = decision.delay.unwrap_or_else(|| policy.delay_for_attempt(attempt_index));
let delay = retry_delay.unwrap_or_else(|| policy.delay_for_attempt(attempt_index));
info!(
provider = provider_runtime.provider_name(),
next_attempt = attempt_index + 2,
Expand Down Expand Up @@ -1208,6 +1236,17 @@ async fn run_prompt(agent: Arc<ZedAgent>, args: PromptRequest) -> Result<PromptR
let decision = policy.decision_for_llm_error(&error, attempt_index);
telemetry.failed(&provider_runtime, attempt_index, retry_disposition(&decision), &error);
drop(permit);
let retry_delay = decision
.retryable
.then(|| decision.delay.unwrap_or_else(|| policy.delay_for_attempt(attempt_index)));
agent
.publish_rate_limit_notice(
&args.session_id,
provider_runtime.provider_name(),
&error,
retry_delay,
)
.await;
if !decision.retryable {
return Ok(finish_failed_provider_turn(
&agent,
Expand All @@ -1219,7 +1258,7 @@ async fn run_prompt(agent: Arc<ZedAgent>, args: PromptRequest) -> Result<PromptR
)
.await);
}
let delay = decision.delay.unwrap_or_else(|| policy.delay_for_attempt(attempt_index));
let delay = retry_delay.unwrap_or_else(|| policy.delay_for_attempt(attempt_index));
info!(
provider = provider_runtime.provider_name(),
next_attempt = attempt_index + 2,
Expand Down Expand Up @@ -1252,6 +1291,9 @@ async fn run_prompt(agent: Arc<ZedAgent>, args: PromptRequest) -> Result<PromptR
}
Err(error) => {
telemetry.failed(&provider_runtime, attempt_index, "partial_output_visible", &error);
agent
.publish_rate_limit_notice(&args.session_id, provider_runtime.provider_name(), &error, None)
.await;
return Ok(finish_failed_provider_turn(
&agent,
&session,
Expand Down Expand Up @@ -1306,7 +1348,7 @@ async fn run_prompt(agent: Arc<ZedAgent>, args: PromptRequest) -> Result<PromptR
}
}
telemetry.complete(&provider_runtime, &response, attempt_index, false);
agent.publish_lody_usage(&args.session_id, &session_model, &response);
agent.publish_lody_usage(&args.session_id, &session_provider_name, &session_model, &response);
if assistant_message.is_empty()
&& let Some(content) = response.content
{
Expand Down Expand Up @@ -1459,27 +1501,34 @@ async fn run_prompt(agent: Arc<ZedAgent>, args: PromptRequest) -> Result<PromptR
..Default::default()
};

let response =
match generate_with_retry(provider.as_ref(), request, &provider_runtime, &session.cancellation).await {
Ok(response) => response,
Err(ProviderCallError::Cancelled) => {
stop_reason = acp::StopReason::Cancelled;
break;
}
Err(ProviderCallError::Failed(error)) => {
return Ok(finish_failed_provider_turn(
&agent,
&session,
&args.session_id,
&assistant_message,
&assistant_reasoning,
&error,
)
.await);
}
};
let response = match generate_with_retry(
provider.as_ref(),
request,
&provider_runtime,
&session.cancellation,
Some((&agent, &args.session_id)),
)
.await
{
Ok(response) => response,
Err(ProviderCallError::Cancelled) => {
stop_reason = acp::StopReason::Cancelled;
break;
}
Err(ProviderCallError::Failed(error)) => {
return Ok(finish_failed_provider_turn(
&agent,
&session,
&args.session_id,
&assistant_message,
&assistant_reasoning,
&error,
)
.await);
}
};

agent.publish_lody_usage(&args.session_id, &session_model, &response);
agent.publish_lody_usage(&args.session_id, &session_provider_name, &session_model, &response);
if session.cancellation.is_cancelled() {
stop_reason = acp::StopReason::Cancelled;
break;
Expand Down Expand Up @@ -1688,6 +1737,8 @@ mod tests {
assert!(capabilities.session_capabilities.resume.is_some());
let lody = &capabilities.meta.expect("Lody capability metadata")["lody"];
assert_eq!(lody["usage"]["version"], 1);
assert_eq!(lody["compaction"]["version"], 1);
assert!(lody.get("rateLimits").is_none(), "no trustworthy quota source is configured");
assert!(lody.get("subagents").is_none());
}

Expand Down Expand Up @@ -1937,6 +1988,30 @@ Run the managed background fixture.
);
}

#[test]
fn provider_error_telemetry_exposes_rate_limit_diagnostics() {
let error = LLMError::RateLimit {
metadata: Some(LLMErrorMetadata::new(
"baseten",
Some(429),
Some("rate_limit_error".to_string()),
None,
None,
Some("17".to_string()),
Some("capacity temporarily exhausted".to_string()),
)),
};

assert_eq!(
provider_error_telemetry(&error),
ProviderErrorTelemetry {
code: Some("rate_limit_error"),
status: Some(429),
detail: Some("capacity temporarily exhausted"),
}
);
}

proptest! {
#[test]
fn streaming_eligibility_depends_only_on_provider_support_and_stop_hooks(
Expand Down Expand Up @@ -3399,6 +3474,7 @@ Run the managed background fixture.
LLMRequest::default(),
&timeout_runtime(),
&super::super::super::types::SessionCancellation::default(),
None,
)
.await;

Expand Down Expand Up @@ -3441,6 +3517,7 @@ Run the managed background fixture.
LLMRequest::default(),
&retry_runtime(),
&super::super::super::types::SessionCancellation::default(),
None,
)
.await
.expect("transient request should recover");
Expand All @@ -3458,6 +3535,7 @@ Run the managed background fixture.
LLMRequest::default(),
&retry_runtime(),
&super::super::super::types::SessionCancellation::default(),
None,
)
.await;

Expand All @@ -3481,6 +3559,7 @@ Run the managed background fixture.
LLMRequest::default(),
&retry_runtime(),
&super::super::super::types::SessionCancellation::default(),
None,
)
.await;

Expand Down
Loading
Loading