From 2fd9ee3001e34febae14586b3abd2c97d6f39ae6 Mon Sep 17 00:00:00 2001 From: Bryce Baril Date: Tue, 13 Oct 2026 15:22:42 -0700 Subject: [PATCH 1/3] fix(skills): ns-analyze-tracing builds the waterfall with tracing-detail The skill said tracing-detail isn't exposed over MCP and told agents to look a trace up with tracing's span_traceId filter. Both are wrong for current consoles: tracing-detail is an MCP tool that returns every span of a trace with pagination metadata, and tracing ignores span_traceId and returns the default page. Agents following the skill matched span_parentId values by hand, or gave up on full traces. - A trace ID alone goes to tracing-detail, paged while more remain. - Step 6 builds the parent/child waterfall from tracing-detail, and falls back to list evidence only when the tool isn't there (older consoles). - Parent/child claims are allowed when the hierarchy came from tracing-detail, not only from a supplied tree. - tracing rounds start down to the hour: check metadata.start, and for a sub-hour window pass an explicit start and filter rows by span_start. Fixes #72 Co-Authored-By: Claude Opus 5.5 --- skills/ns-analyze-tracing/SKILL.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/skills/ns-analyze-tracing/SKILL.md b/skills/ns-analyze-tracing/SKILL.md index 217c085..31c6acc 100644 --- a/skills/ns-analyze-tracing/SKILL.md +++ b/skills/ns-analyze-tracing/SKILL.md @@ -1,13 +1,13 @@ --- name: ns-analyze-tracing description: >- - Analyzes N|Solid/OpenTelemetry tracing evidence for request latency, HTTP errors, and service dependencies. Use when the user asks about slow endpoints, API timeouts, distributed traces/spans, N+1 queries, await chains, microservice latency, cascading failures, or trace IDs. Query N|Solid only when no trace data was provided. Do not assume full waterfall details unless supplied. + Analyzes N|Solid/OpenTelemetry tracing evidence for request latency, HTTP errors, and service dependencies. Use when the user asks about slow endpoints, API timeouts, distributed traces/spans, N+1 queries, await chains, microservice latency, cascading failures, or trace IDs. Query N|Solid only when no trace data was provided. Builds the span waterfall with `tracing-detail` once a trace ID is known. --- ### 1. Use Provided Trace Data First - Treat the prompt as primary evidence. Parse any provided trace ID, span list, app name, endpoint name, status code, duration, or error stack before calling tools. - If the user supplied a trace tree/export or any other host-provided trace data, treat it as authoritative: map that hierarchy directly and **do not fall through to live enrichment** (`information-dashboard`, `tracing`). The connected-services discovery (step 2) and live `tracing` queries (steps 3–4) are only for when no sufficient host data is present. -- If the user supplied only a trace ID (not a full tree), call `tracing` with `span_traceId` for list-level rows only; the MCP tool does not expose full waterfall details. +- If the user supplied only a trace ID (not a full tree), call `tracing-detail` with `span_traceId` to get every span of that trace. For a large trace, page with `page` and `showLimit` while the pagination metadata says more remain (`hasMore`). Do not call `tracing` with `span_traceId`: it ignores that filter and returns the default page. ### 2. Discover Connected Services Only When Needed - **Skip this step entirely if sufficient host-provided trace data is present** (step 1 already authoritatively covered it). @@ -19,6 +19,7 @@ description: >- ### 3. Find Slow Requests Only If No Trace Was Supplied - Call `tracing`. Use pipe duration syntax (e.g., `durations="1000|5000"` for 1–5s spans); do not use dash-range syntax. - Use `functionName` only when the exact server-side function name is known. +- `tracing` rounds `start` down to the hour. Check `metadata.start` in the response. For a window shorter than an hour, pass an explicit ISO `start` and keep only rows whose `span_start` falls inside the window. - Treat the response as a paginated, trace-ID-collapsed sample. The default page contains 25 traces, and the summary represents only the returned page; do not treat an absent app group as proof that the app has no tracing. ### 4. Find Failing Endpoints Only If No Trace Was Supplied @@ -37,12 +38,13 @@ description: >- ### 6. Triage the Trace - Use `tracing` results as collapsed trace-list evidence: slow/failing service, endpoint, status, duration, and `span_traceId`. -- Do not claim parent/child waterfall analysis from `tracing` alone; `tracing-detail` is not exposed as MCP. +- For the slow or failing trace, call `tracing-detail` with its `span_traceId` and build the parent/child waterfall from `span_id` and `span_parentId`: which child spans take the time, which fail, and which run one after another when they could run in parallel. +- If `tracing-detail` is not among the available tools (an older console), stay with the collapsed trace-list evidence and say that the waterfall was not available. - If the user supplied the full trace tree/export, analyze `span_parentId` vs child hierarchy directly. ### 7. Propose Architectural Fixes - Once you identify a bottleneck trace row or supplied trace tree, explain the strongest supported cause. -- Only discuss parent-child span relationships when the trace tree was supplied. +- Only discuss parent-child span relationships when the hierarchy came from `tracing-detail` or a supplied trace tree. - Propose topological changes like adding Redis caching, parallelizing independent `Promise.all` requests, or using message queues. ### 8. Present a Report @@ -53,7 +55,7 @@ description: >- - `## Findings` - `## Recommendations` - `## Validation Plan` when a fix is proposed -- Ground every claim in supplied trace data or MCP `tracing` output. State when only collapsed trace-list evidence is available. +- Ground every claim in supplied trace data or MCP `tracing` and `tracing-detail` output. State when only collapsed trace-list evidence is available. ### 9. Write the Report to Disk - Ask the user if they want to save the report to disk. @@ -67,5 +69,6 @@ description: >- ## Guardrails - NEVER call `global-filter` for service discovery — it returns ~18,000 tokens and fills the context window. Use `information-dashboard` only. - Do not search randomly; always filter using `durations` or status codes first to narrow down the dataset. -- A slow top-level span may be caused by a slow child span, but only assert that when full trace hierarchy is available. +- A slow top-level span may be caused by a slow child span, but only assert that when the full hierarchy is available, from `tracing-detail` or a supplied trace tree. +- Never pass `span_traceId` to `tracing` to look up one trace; it ignores the filter. Use `tracing-detail`. - Filter out expected long-polling or WebSocket connections when hunting for latency regressions. From 1f2c83cde5deaa6b0100daaf2590da82c51ba8d2 Mon Sep 17 00:00:00 2001 From: Bryce Baril Date: Wed, 14 Oct 2026 13:13:36 -0700 Subject: [PATCH 2/3] fix(skills): ns-analyze-tracing reads tracing windows in whole hours and sorts tracing-detail spans The console rounds a tracing start down to the hour, even an exact ISO time, and without one it covers the last 23 to 24 hours. Over MCP the tracing response is always summarized per app, so the skill's advice to keep only rows inside a shorter window could not be followed, and metadata.total counts the whole rounded window. The skill now says the window is whole hours and has the report state it from metadata.start and metadata.end. tracing-detail returns spans in no fixed order and pages without a sort, so the skill asks for the whole trace in one call (showLimit covering totalSpans) and sorts the spans by span_start before reading what ran in sequence. Its fallback now covers the tool erroring or returning no spans, as well as consoles before 6.5.2 that lack it, on both the trace-ID path and in triage. A supplied trace tree stays authoritative even when it includes its trace ID. Co-Authored-By: Claude Opus 5.5 --- skills/ns-analyze-tracing/SKILL.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/skills/ns-analyze-tracing/SKILL.md b/skills/ns-analyze-tracing/SKILL.md index 31c6acc..0a1f127 100644 --- a/skills/ns-analyze-tracing/SKILL.md +++ b/skills/ns-analyze-tracing/SKILL.md @@ -1,13 +1,14 @@ --- name: ns-analyze-tracing description: >- - Analyzes N|Solid/OpenTelemetry tracing evidence for request latency, HTTP errors, and service dependencies. Use when the user asks about slow endpoints, API timeouts, distributed traces/spans, N+1 queries, await chains, microservice latency, cascading failures, or trace IDs. Query N|Solid only when no trace data was provided. Builds the span waterfall with `tracing-detail` once a trace ID is known. + Analyzes N|Solid/OpenTelemetry tracing evidence for request latency, HTTP errors, and service dependencies. Use when the user asks about slow endpoints, API timeouts, distributed traces/spans, N+1 queries, await chains, microservice latency, cascading failures, or trace IDs. Query N|Solid only when no trace data was provided. Builds the span waterfall with `tracing-detail` when the console offers it. --- ### 1. Use Provided Trace Data First - Treat the prompt as primary evidence. Parse any provided trace ID, span list, app name, endpoint name, status code, duration, or error stack before calling tools. -- If the user supplied a trace tree/export or any other host-provided trace data, treat it as authoritative: map that hierarchy directly and **do not fall through to live enrichment** (`information-dashboard`, `tracing`). The connected-services discovery (step 2) and live `tracing` queries (steps 3–4) are only for when no sufficient host data is present. -- If the user supplied only a trace ID (not a full tree), call `tracing-detail` with `span_traceId` to get every span of that trace. For a large trace, page with `page` and `showLimit` while the pagination metadata says more remain (`hasMore`). Do not call `tracing` with `span_traceId`: it ignores that filter and returns the default page. +- If the user supplied a trace tree/export or any other host-provided trace data, treat it as authoritative: map that hierarchy directly and **do not fall through to live enrichment** (`information-dashboard`, `tracing`, `tracing-detail`), even when the export includes its trace ID. The connected-services discovery (step 2) and live `tracing` queries (steps 3–4) are only for when no sufficient host data is present. +- If the user supplied only a trace ID (not a full tree), call `tracing-detail` with `span_traceId` and `showLimit: 1000` to get the whole trace in one call. If the metadata's `totalSpans` is larger, call once more with `showLimit` at least `totalSpans` (at most 10000). Don't page with `page`: the spans come back in no fixed order, so pages can repeat or skip spans. Do not call `tracing` with `span_traceId`: it ignores that filter and returns the default page. +- If `tracing-detail` is not among the available tools (consoles before 6.5.2), returns an error (such as `elasticReader is not available`), or returns no spans (`totalSpans: 0`: the trace aged out or the ID is wrong), say the trace could not be read and ask the user for the trace data or a different trace ID. ### 2. Discover Connected Services Only When Needed - **Skip this step entirely if sufficient host-provided trace data is present** (step 1 already authoritatively covered it). @@ -19,7 +20,7 @@ description: >- ### 3. Find Slow Requests Only If No Trace Was Supplied - Call `tracing`. Use pipe duration syntax (e.g., `durations="1000|5000"` for 1–5s spans); do not use dash-range syntax. - Use `functionName` only when the exact server-side function name is known. -- `tracing` rounds `start` down to the hour. Check `metadata.start` in the response. For a window shorter than an hour, pass an explicit ISO `start` and keep only rows whose `span_start` falls inside the window. +- `tracing` works in whole hours. It rounds `start` down to the hour, even an exact ISO time, and without `start` it covers the last 23–24 hours (from 23 hours ago, rounded down). Its MCP response is always summarized per app (minimum, maximum, median, latest), so it can't be narrowed to a shorter window afterwards, and `metadata.total` counts the traces of the whole rounded window. For recent behavior, pass `start` at the current hour. Report the window the evidence covers, from `metadata.start` and `metadata.end`, and don't present a day of data as current. - Treat the response as a paginated, trace-ID-collapsed sample. The default page contains 25 traces, and the summary represents only the returned page; do not treat an absent app group as proof that the app has no tracing. ### 4. Find Failing Endpoints Only If No Trace Was Supplied @@ -33,13 +34,13 @@ description: >- - Re-query `tracing` once for each absent expected app, using the exact app name and preserving the original time range and filters (`durations`, status code, endpoint, and other applicable filters). Use the app-specific result to determine whether that app has matching traces. - If the app-specific query returns no matching traces, report only that the app has **no traces matching the requested filters and time range** — zero filtered rows do not prove the app has no tracing (e.g. an app with only 200 ms spans returns nothing under `durations="1000|5000"`). - To claim an app has no tracing at all, run one additional query with only the exact app name and the original time range (no `durations`, status code, or endpoint filters). Only a non-partial empty result from this unfiltered query justifies the "no tracing" claim; if it returns traces, report that tracing exists but none matched the filters. -- If either app-specific query is partial, or reports matching traces through `metadata.total`, report the tracing evidence and its incomplete coverage instead of claiming absence. +- If either app-specific query is partial, or reports matching traces through `metadata.total`, report the tracing evidence and its incomplete coverage instead of claiming absence. `metadata.total` covers the whole rounded window from step 3: it shows the app had matching traces in that window, not in a shorter one the user asked about. - Do not auto-page every global tracing result. These bounded app-specific checks are the fallback for reconciling the known app inventory. ### 6. Triage the Trace - Use `tracing` results as collapsed trace-list evidence: slow/failing service, endpoint, status, duration, and `span_traceId`. -- For the slow or failing trace, call `tracing-detail` with its `span_traceId` and build the parent/child waterfall from `span_id` and `span_parentId`: which child spans take the time, which fail, and which run one after another when they could run in parallel. -- If `tracing-detail` is not among the available tools (an older console), stay with the collapsed trace-list evidence and say that the waterfall was not available. +- If no trace tree was supplied, call `tracing-detail` for the slow or failing trace's `span_traceId`, as in step 1 (one call, `showLimit` covering the whole trace). Sort the spans by `span_start` first: they come back in no fixed order, and response order is not execution order. Then build the parent/child waterfall from `span_id` and `span_parentId`: which child spans take the time, which fail, and which run one after another (one starts after the previous ends) when they could run in parallel. +- If `tracing-detail` is not among the available tools (consoles before 6.5.2), returns an error, or returns no spans, stay with the collapsed trace-list evidence and say that the waterfall was not available. - If the user supplied the full trace tree/export, analyze `span_parentId` vs child hierarchy directly. ### 7. Propose Architectural Fixes @@ -71,4 +72,5 @@ description: >- - Do not search randomly; always filter using `durations` or status codes first to narrow down the dataset. - A slow top-level span may be caused by a slow child span, but only assert that when the full hierarchy is available, from `tracing-detail` or a supplied trace tree. - Never pass `span_traceId` to `tracing` to look up one trace; it ignores the filter. Use `tracing-detail`. +- Read `tracing-detail` spans in `span_start` order; the order they arrive in means nothing. - Filter out expected long-polling or WebSocket connections when hunting for latency regressions. From 6b782e5580414482cd9fcf0cda558ce453939c8f Mon Sep 17 00:00:00 2001 From: Bryce Baril Date: Mon, 19 Oct 2026 11:01:49 -0700 Subject: [PATCH 3/3] fix(skills): ns-analyze-tracing asks tracing-detail for 150 spans, and says when a trace is bigger tracing-detail returns every span whole, so one call with showLimit 1000 can pass the client's limit on tool output. Claude Code caps MCP output at 25,000 tokens by default, and a trace of 100 to 150 real spans already fills that. The big traces are the N+1 cases this skill is for. The skill now asks for 150 spans. A trace that fits comes back whole. For a bigger one it says the waterfall covers returnedSpans of totalSpans, in no particular order: enough to show repeated span names, not the critical path or totals. A guardrail repeats it. Co-Authored-By: Claude Opus 5.5 --- skills/ns-analyze-tracing/SKILL.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/skills/ns-analyze-tracing/SKILL.md b/skills/ns-analyze-tracing/SKILL.md index 0a1f127..e252d7e 100644 --- a/skills/ns-analyze-tracing/SKILL.md +++ b/skills/ns-analyze-tracing/SKILL.md @@ -7,7 +7,7 @@ description: >- ### 1. Use Provided Trace Data First - Treat the prompt as primary evidence. Parse any provided trace ID, span list, app name, endpoint name, status code, duration, or error stack before calling tools. - If the user supplied a trace tree/export or any other host-provided trace data, treat it as authoritative: map that hierarchy directly and **do not fall through to live enrichment** (`information-dashboard`, `tracing`, `tracing-detail`), even when the export includes its trace ID. The connected-services discovery (step 2) and live `tracing` queries (steps 3–4) are only for when no sufficient host data is present. -- If the user supplied only a trace ID (not a full tree), call `tracing-detail` with `span_traceId` and `showLimit: 1000` to get the whole trace in one call. If the metadata's `totalSpans` is larger, call once more with `showLimit` at least `totalSpans` (at most 10000). Don't page with `page`: the spans come back in no fixed order, so pages can repeat or skip spans. Do not call `tracing` with `span_traceId`: it ignores that filter and returns the default page. +- If the user supplied only a trace ID (not a full tree), call `tracing-detail` with `span_traceId` and `showLimit: 150`. Every span comes back whole, so a bigger page can pass the client's limit on tool output. If the metadata's `totalSpans` is at most 150, that is the whole trace. If it is larger, don't ask for more: say the waterfall covers `returnedSpans` of `totalSpans` spans, in no particular order. That still shows repeated span names (an N+1), but not the critical path or totals. Don't page with `page`: the spans come back in no fixed order, so pages can repeat or skip spans. Do not call `tracing` with `span_traceId`: it ignores that filter and returns the default page. - If `tracing-detail` is not among the available tools (consoles before 6.5.2), returns an error (such as `elasticReader is not available`), or returns no spans (`totalSpans: 0`: the trace aged out or the ID is wrong), say the trace could not be read and ask the user for the trace data or a different trace ID. ### 2. Discover Connected Services Only When Needed @@ -39,7 +39,7 @@ description: >- ### 6. Triage the Trace - Use `tracing` results as collapsed trace-list evidence: slow/failing service, endpoint, status, duration, and `span_traceId`. -- If no trace tree was supplied, call `tracing-detail` for the slow or failing trace's `span_traceId`, as in step 1 (one call, `showLimit` covering the whole trace). Sort the spans by `span_start` first: they come back in no fixed order, and response order is not execution order. Then build the parent/child waterfall from `span_id` and `span_parentId`: which child spans take the time, which fail, and which run one after another (one starts after the previous ends) when they could run in parallel. +- If no trace tree was supplied, call `tracing-detail` for the slow or failing trace's `span_traceId`, as in step 1 (one call with `showLimit: 150`; for a bigger trace, say the waterfall covers only part of it). Sort the spans by `span_start` first: they come back in no fixed order, and response order is not execution order. Then build the parent/child waterfall from `span_id` and `span_parentId`: which child spans take the time, which fail, and which run one after another (one starts after the previous ends) when they could run in parallel. - If `tracing-detail` is not among the available tools (consoles before 6.5.2), returns an error, or returns no spans, stay with the collapsed trace-list evidence and say that the waterfall was not available. - If the user supplied the full trace tree/export, analyze `span_parentId` vs child hierarchy directly. @@ -73,4 +73,5 @@ description: >- - A slow top-level span may be caused by a slow child span, but only assert that when the full hierarchy is available, from `tracing-detail` or a supplied trace tree. - Never pass `span_traceId` to `tracing` to look up one trace; it ignores the filter. Use `tracing-detail`. - Read `tracing-detail` spans in `span_start` order; the order they arrive in means nothing. +- When `returnedSpans` is below `totalSpans`, the waterfall is a sample: report repeated spans from it, never the critical path or a total. - Filter out expected long-polling or WebSocket connections when hunting for latency regressions.