fix(pipeline): tell the customer when the AI backend fails (CRM-236) - #9
Conversation
When the LLM provider degrades the turn exceeds the ceiling and the pipeline only logged and cleared state — nothing reached the chat. For the customer that is indistinguishable from a bot ignoring them, and it is worse than it looks: the tool's side effect may ALREADY be applied. In the live run the pipeline card moved at ~20s and the timeout fired at 30s, so the funnel advanced while the conversation stayed silent. Two changes. 1. AI_CALL_TIMEOUT_SECONDS default 30 -> 90. Measured against gemini-2.5-flash, the SAME trivial prompt answered in 0.74s / 0.79s / 1.36s / 10.34s / 20.40s — a 27x spread on the provider's tail. A tool-calling turn makes at least two of those round trips (decide the tool, then write the reply), so two bad tails alone exceed 30s with nothing wrong in the code. 90s covers that and still bounds a genuinely hung provider. Note this is NOT the fix on its own: raising a ceiling only moves it. The notice below is what protects the customer when the ceiling IS reached. 2. On timeout or error, dispatch a plain sentence to the conversation instead of silence. The provider's raw error never reaches the customer — it carries model names, quota ids and URLs (litellm.RateLimitError: ... limit: 20, model: gemini-2.5-flash) — it goes to the operator's log as `cause`. AI_FAILURE_NOTICE overrides the text; setting it empty keeps today's silence for operators who prefer it. Tests: 6 in pkg/pipeline/service/ai_failure_notice_test.go — the customer is told, the provider error never leaks, the text is overridable, empty disables it, a missing postback url is not a crash, and the default holds when the env is unset. go build + go vet clean; full suite green (pkg/... and internal/..., Redis-backed). Not addressed here, deliberately: the processor keeps working after the bot-runtime gives up, and the provider's 429/503 still surfaces as a generic 500 on the A2A route. Both are real and belong to the processor side.
Reviewer's GuideExtends AI pipeline robustness by increasing the AI call timeout and proactively notifying customers in chat when the AI backend times out or errors, with configurable and non‑leaky messaging, plus tests to validate the behavior. Sequence diagram for AI failure customer notificationsequenceDiagram
participant Pipeline
participant AIBackend
participant Chat
participant OperatorLog
Pipeline->>AIBackend: runAIStage
AIBackend-->>Pipeline: timeout or error
Pipeline->>Pipeline: clearStateWithLog
Pipeline->>OperatorLog: pipeline.ai.failure_notice.sending(cause)
Pipeline->>Chat: runDispatchStage(notice)
Chat-->>Pipeline: postback delivered
Flow diagram for configurable AI failure noticeflowchart TD
A[AI timeout or error] --> B[clearStateWithLog]
B --> C{AI_FAILURE_NOTICE configured?}
C -->|empty| D[Return silently]
C -->|unset| E[Use default notice]
C -->|non-empty| F[Use configured notice]
E --> G{postbackURL available?}
F --> G
G -->|no| H[Log no_postback and return]
G -->|yes| I[Log provider error as cause]
I --> J[runDispatchStage with customer-safe notice]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Hey - I've found 1 issue
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="pkg/pipeline/service/pipeline_service.go" line_range="422" />
<code_context>
+ // CRM-236: silence is indistinguishable from "the bot is ignoring you",
+ // and the tool's side effect may already be applied (the card moved at
+ // ~20s, the timeout fired at 30s). Tell the customer something.
+ s.sendAIFailureNotice(contactID, conversationID, cfg, postbackURL, err)
default:
slog.Error("pipeline.ai.error",
</code_context>
<issue_to_address>
**issue (bug_risk):** The failure notice is dispatched with a fresh background context after the AI stage has cleared its state, so a newer message can start a replacement pipeline before this dispatch completes. When the notice dispatch succeeds, `runDispatchStage` clears the state and deletes the entry for the same contact/conversation, thereby deleting or clearing the newer pipeline and causing its response to be lost.
**Triggers:** When a new customer message arrives after the AI failure has been recorded but before the fallback notice finishes dispatching.
**Suggested fix:** Revalidate ownership of the pipeline entry before dispatching and before cleanup, or route the notice through an ownership-aware dispatch path that cannot clear state belonging to a newer pipeline.
</issue_to_address>Sourcery assessment
Needs a human reviewer. 1 finding to address first, and if the new timeout or default failure message is the wrong product decision, every affected failed AI turn will wait up to 90 seconds and may send a customer-facing message; those messages cannot be undone by reverting the change. Reverting prevents future notices and restores the old timeout, but it cannot retract notifications already delivered.
Blocking findings: pkg/pipeline/service/pipeline_service.go:422
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
…-236 review)
Addresses the CRITICAL findings on the bot-runtime side.
1. The notice could delete the follow-up turn's entry.
sendAIFailureNotice dispatched through runDispatchStage, which owns the
turn's bookkeeping: its success path runs SetState(StageDone) -> ClearState
-> entries.Delete(pairKey). That Delete is what runDispatchStage's own
comments forbid ("A Delete here would race with the new event's Store and
could delete the replacement entry").
The race lands on the very scenario this feature targets — the customer who
waited and follows up. They send "oi?" while the notice is being dispatched,
startDebounce stores the new turn, then the notice finishes and deletes THAT
entry and clears its state. The new turn is orphaned: the next message cannot
cancel it, so two pipelines run concurrently on the same pair and the
customer gets a duplicated reply.
The notice now dispatches directly. There was nothing to book-keep anyway:
both call sites already run clearStateWithLog before asking for it.
2. The 5s budget truncated the notice and then lied about why.
cleanupCtx's 5s are documented for cleanup calls (ClearState, SetState). A
Dispatch is a different animal: it segments by TextSegmentationLimit and
sleeps DelayPerCharacter per rune between parts. With segmentation on, the
84-rune default became 2-3 parts and the delays alone exceeded 5s ->
ErrDispatchInterrupted -> the log said "New message arrived" when nothing had
arrived. Now bounded by noticeCtx (30s).
3. The 90s ceiling was still pinned at 30 inside this repo.
.env.example:8 and k8s/configmap.yaml:8 both set it explicitly, and by this
PR's own reasoning ("an explicit env beats the code default") the fix had no
effect where it runs — the ConfigMap is what actually serves staging/prod.
NOT fixed here: evolution-ecosystem/k8s/base/bot-runtime.yaml:44-45 pins 30
too, but that is the SaaS repo, outside these three PRs.
4. The default notice was hardcoded pt-BR (finding 9).
bot-runtime ships in community/self-hosted installs worldwide, so customers
of installations that never chose Portuguese were answered in it. Now English,
with AI_FAILURE_NOTICE documented in .env.example for localisation and the
empty value still restoring silence.
Tests: 10 (was 6). The new ones pin that the notice leaves the follow-up entry
and its state untouched, that its budget fits a segmented dispatch, and that the
default carries no pt-BR.
Negative proof: restoring the runDispatchStage call fails
"DoesNotTouchTheEntryOfTheNextTurn" with the orphaned-turn message.
gofmt: pipeline_service.go is already unformatted on develop (CRLF); left as-is
rather than reformatting the whole file into this diff.
11 — the doc comment of clearStateWithLog had been orphaned: the new constants
landed between it and its function, so godoc showed aiFailureNoticeEnv
documented as "clearStateWithLog calls ClearState...". Moved back.
12 — the test helper reimplemented strings.Contains in 9 lines with a closure.
13 — os.Unsetenv mutated the process env without restoring it, so every test
running after that one inherited the change. Restores via t.Cleanup now.
No behaviour change. build/vet clean, full suite green with a real Redis.
CI was red on TestE2E_PipelineIsolation, and the test was right to fail:
postback content = "We are having a temporary issue…", want "pair-b response"
It asserted callCount == 1 with the comment "only pair B should deliver". That
described the OLD behaviour, where pair A's AI error was swallowed — log, clear
state, nothing reaching the chat. Making that pair speak is the entire point of
this card, so the assertion had to move.
Updated rather than relaxed. The test is about isolation, and it now checks that
more strictly than before: both pairs deliver, and each must receive ITS OWN
message. A crossed delivery fails here even though the count would be right —
the previous version could not have caught that.
The notice is pinned through AI_FAILURE_NOTICE instead of reaching for the
package constant: it keeps test/e2e out of service's internals and exercises the
operator-facing env on the way.
On how this reached CI: I ran ./pkg/... and ./internal/... and called it "the
full suite". test/e2e was never in it. Same mistake as the migration parity spec
on CRM-210 — a hand-picked list reported as a green suite. Ran ./... this time.
…eview) The round-1 defect was runDispatchStage's success bookkeeping running for the failure notice: SetState(StageDone) -> ClearState -> entries.Delete(pairKey), against a pair that may already belong to a follow-up turn. The fix is in, but nothing pinned it: the existing test read Redis after the fact, and SetState followed by ClearState leaves nothing to read, so it passed with or without the regression. DoesNotTouchTheEntryOfTheNextTurn seeds the pair the way startDebounce leaves it (entry in the map, StageDebounce in Redis) and asserts both survive. Restoring the runDispatchStage call fails all three assertions. Also make DoesNotWriteTurnState fail on a read error instead of passing on it, and drop its unused rdb binding.
… 14) The measurement table, the incident timeline and the race walkthrough belong in the PR body, where they already are. In the source they were 11 lines above one call to getEnvIntOrDefault, 18 above one call to Dispatch, and 10 above a context.WithTimeout. What is kept is what the code cannot say: why 90 and not 30, why the notice must not go through runDispatchStage, why noticeCtx is not cleanupCtx. 102 added comment lines to 58, no behaviour touched.
* fix(EVO-2167): retry the AI Processor call on transient failures The bot-runtime made a single call to the AI Processor: any non-200 (401/5xx) or network error aborted the pipeline and the customer's message was dropped with no reply and no retry (only AI_CALL_TIMEOUT_SECONDS existed). A momentary blip — deploy, restart, DB hiccup — meant a permanently lost answer. - ai_adapter.go: extract a single attempt into doOnce() and wrap Call() in a retry loop with exponential backoff + jitter. Retryable: network errors and 429/500/502/503/504. NOT retried: 4xx (permanent), per-attempt timeout, pipeline cancellation. Body is built once and reused per attempt. - config.go: AI_CALL_MAX_RETRIES (default 2) and AI_CALL_RETRY_BASE_MS (default 200). - main.go: wire the new config into NewAIAdapter. - tests: 503->200 retry succeeds (2 calls); persistent 500 exhausts retries (1+2 calls); 400 not retried (1 call); network error then success (2 calls); maxRetries=0 disables retry. Existing tests updated to the new signature (0 retries). Complements EVO-2166: the processor now returns 503 (not a silent 401) on infra errors, so this retry covers the transient auth/infra case. Root cause of the incident is EVO-2141 (pool_pre_ping, already merged); this is defense in depth. Note: test/e2e/e2e_test.go was already incompatible with NewAIAdapter on develop (pre-existing, unrelated) and is left as-is; repo CI is docker-only (no go test lane). * fix(EVO-2167): harden retry — total-time cap + per-attempt timeout test Review follow-ups on the AI Processor retry path: - Add an overall time-budget backstop so the retry loop is provably bounded ((attempts+1) x per-attempt timeout + summed max backoff); the +1 slack keeps a per-attempt timeout surfacing as ErrAITimeout instead of being swallowed by the backstop. AC "teto de tempo total". - Add TestCall_TimeoutIsNotRetried: a per-attempt timeout must return ErrAITimeout and must NOT be retried with retries enabled. AC #5 "timeout por tentativa". - Document the idempotency contract on the retry path (502/504/network replay can re-run an already-processed turn; customer still gets one reply; dedupe of the duplicate server-side turn is the AI Processor's job, tracked in EVO-2166). * feat(EVO-2180): forward incoming media to the AI Processor as A2A file parts The bot only ever sent a text part, so images/audio the customer sent never reached the AI (the agent replied "No content to process"). Accept attachments on the inbound event, carry them through the debounce window, download each and send it as a base64 A2A file part. Part of EVO-2178 (image end-to-end). - pipeline/model: MessageEvent.Attachments + Attachment{URL,ContentType,FileType}. - pipeline/repository: AppendAttachments/GetAttachments on a parallel Redis list (bot_runtime:attach:{contact}:{conv}), aggregated like the text buffer and cleared together in ClearState (no stale media leaks into the next turn). - debounce/service: Start/Reset accept attachments; GetAttachments added. - pipeline/service: thread event.Attachments through start/skip/reset/advance -> the A2ARequest (read fresh from Redis at stage launch, like the buffer). - ai/model: A2ARequest.Attachments; JSONRPCPart.File + JSONRPCFile{Name,MimeType,Bytes} (tags match the processor's extract_files_from_message). - ai/service/ai_adapter: download each attachment once (before Marshal, reused across retries) with a 15 MiB cap; base64-encode; append a file part. A download failure is logged and skipped so the text-only message always survives. - tests: adapter forwards a file part with decodable base64 + download-failure sends text only; repo AppendAttachments/GetAttachments roundtrip + ClearState clears the attach key. Full suite green (go build/vet/test ./pkg/... ./internal/...). Note: test/e2e was already incompatible with NewAIAdapter on develop (pre-existing); repo CI is docker-only. * fix(EVO-2180): bound media forwarding and validate what is forwarded Review follow-ups on the incoming-media path. The per-file cap was the only bound, so the failure modes it did not cover fell back on the customer losing the whole reply instead of just the media. - Shared byte budget (20 MiB) across every attachment of the call. The debounce window aggregates the media of all its messages, so a photo burst built a body of len(attachments) x 15 MiB; base64 pushed that past the gateway's client_max_body_size and the resulting 413 is not retryable, killing the text reply too. Probe: 20 x 2 MiB went from a 53 MiB request to 26 MiB. - Dedicated download timeouts. Downloads run before the AI call and outside its retry ceiling, but reused AI_CALL_TIMEOUT_SECONDS (30s) per attachment, so an unreachable media host stalled the turn by 30s x len(attachments) with no bound. Now 10s per download and 30s for the whole set. - Resolve the mime type from the bytes in hand: the response Content-Type wins, then the CRM's declared type, then the URL extension. The processor feeds this straight into Blob(mime_type=...), so an HTML error/login page answered with 200 was being forwarded as a valid image, and a missing content_type became application/octet-stream. Both are now dropped or resolved. - A Redis failure on the attachment buffer no longer aborts the turn: media is best-effort everywhere else in this path, and dropping the text reply over it contradicted the card's own acceptance criterion. Tests: the event -> debounce -> Redis -> A2ARequest seam had no coverage (the debounce mock always returned nil attachments), so a refactor could silently drop the media; two pipeline tests now pin it, including aggregation across the debounce window. Adapter tests cover the byte budget, the time budget, HTML responses, oversize files and the mime resolution table. Also repairs test/e2e, which has not compiled since EVO-2167 changed NewAIAdapter/NewDispatchEngine — which is why `go vet ./...` and `go test ./...` could not be run at all. Two assertions had drifted: the message signature moved to a prefix on the first segment in EVO-558, and the state-leak check raced the cleanup goroutine it was asserting on. go build ./... && go vet ./... && go test ./... green, e2e included. * fix(EVO-2178): validate incoming media URLs before fetching them Review follow-up to EVO-2180. Attachment URLs arrive inside the /events payload and the adapter fetched them verbatim, so the endpoint doubled as a read primitive aimed by its caller: the bytes of any URL reachable from this service were base64-encoded into the A2A call, whose destination (outgoing_url) comes from the same payload. Reproduced end to end against a local metadata-style endpoint, both directly and through a 302. - checkMediaURL pins the scheme to http/https and requires the host to be one the CRM is known to serve blobs from: the postback URL's host (already mandatory in MessageEvent.Validate, so no new config for the default topology) plus whatever MEDIA_HOST_ALLOWLIST names, for deployments serving blobs off an S3/MinIO/CDN host. Unauthorized media is skipped and logged; the text reply is unaffected, like every other media failure here. - The download client re-runs that check on every redirect hop, so an authorized host cannot walk the fetch onto an internal address. - BOT_RUNTIME_SECRET becomes required. It was read with os.Getenv, and SecretMiddleware compares the header against it, so an empty value authenticated every caller that simply omitted the header. - The media buffer key gets a TTL. ClearState remains the normal cleanup; the TTL only stops a turn that dies before reaching it from leaving media URLs in Redis forever. - Attachment download failures now log the HTTP status: the common production case is a 404 from a signed link that expired while the queue was backed up, and it read identically to an unreachable host. - Adds .github/workflows/ci.yml. Nothing ran the Go suite on a PR, which is how test/e2e stayed non-compiling from EVO-558 until EVO-2180. * style(EVO-2180): trim the comments to what is not obvious from the code * fix(EVO-2178): take the media host allowlist from config, not from the event Fixes a regression I introduced in #6. The allowlist was anchored on the host of the event's postback_url, which never matches the host the CRM actually signs media URLs with: postback_url comes from BOT_RUNTIME_POSTBACK_BASE_URL (internal DNS, "evo-crm" in the shipped compose) while the URL is built from ACTIVE_STORAGE_URL, falling back to BACKEND_URL — which production requires to be a public host. So on develop every attachment was rejected as blocked_url and the agent stopped seeing images: the EVO-2178 bug, back. Anchoring on the event was also the wrong shape for the guard. Whoever sends the event chooses every field in it, including the one being used to decide what that same event may reach, so the check constrained nobody it needed to. Reading MEDIA_HOST_ALLOWLIST only puts the decision with the operator, where it cannot be chosen by the caller. A2ARequest.PostbackURL is dropped again. The scheme check and the per-redirect re-check are unchanged. This makes the variable required wherever media is expected: unset means no attachment is fetched. The deploy surfaces are wired up in the umbrella PR; k8s/configmap.yaml and k8s/deployment.yaml carry it here. * Merge pull request #9 from evolution-foundation/fix/CRM-236-degraded-provider-feedback fix(pipeline): tell the customer when the AI backend fails (CRM-236) --------- Co-authored-by: Matheus Pastorini <matheus.pastorini@etus.com.br> Co-authored-by: Matheus Pastorini <pastorinimatheus@gmail.com>
Problema
Quando o provedor de LLM degrada, o turno estoura o teto e o pipeline apenas logava e limpava o estado — nada chegava ao chat. Para o cliente é indistinguível de um bot que o está ignorando.
E é pior do que parece: o efeito colateral da ferramenta já foi aplicado. Na execução ao vivo o card do funil moveu aos ~20s e o timeout disparou aos 30s — o funil andou e a conversa ficou muda.
Medição (a premissa original do card estava errada)
O card dizia "30s não cobre turno com tool-calling". Falso. Turno completo, em condições normais:
O que oscila é o provedor. Mesma chamada trivial ("responda apenas: ok") ao
gemini-2.5-flash, cinco vezes seguidas:Variação de 27× na mesma chamada. Um turno com ferramenta faz ao menos duas dessas idas (decidir a tool, depois redigir a resposta), então duas caudas ruins sozinhas passam de 30s sem nada de errado no código.
E sob cota estourada o litellm ainda retenta internamente:
Mudanças
1.
AI_CALL_TIMEOUT_SECONDS30 → 90. Cobre duas chamadas na cauda mais o round trip do CRM, e ainda limita um provedor genuinamente travado.Isto não é a correção sozinha: elevar um teto só o desloca. É a mensagem abaixo que protege o cliente quando o teto é atingido.
2. Aviso no chat em vez de silêncio. No timeout ou erro, o pipeline despacha uma frase para a conversa. O erro cru do provedor nunca chega ao cliente — ele carrega nome de modelo, id de cota e URLs — e vai para o log do operador como
cause:AI_FAILURE_NOTICEsobrescreve o texto; defini-la vazia mantém o silêncio de hoje, para quem preferir.Testes
pkg/pipeline/service/ai_failure_notice_test.go— 6 testes: o cliente é avisado; o erro do provedor nunca vaza (verifica quegemini,Quota,RateLimitError,googleapisnão aparecem na mensagem); o texto é sobrescrevível; vazio desativa; postback ausente não quebra; o default vale quando a env não existe.Deliberadamente fora deste PR
Dois defeitos reais que pertencem ao lado do processor, não ao bot-runtime:
500 INTERNAL_ERRORgenérico na rota A2A, então nem o operador vê "cota excedida" sem ler o log do container.Ambos estão registrados no card. Fiz aqui o que resolve o sintoma para o cliente; aquilo exige mexer no caminho de erro do processor e merece PR próprio.
Trade-off assumido
Um teto de 90s significa que um provedor travado segura o turno por mais tempo antes de o cliente receber o aviso. É o preço de não cortar turnos legítimos com cauda alta — e o aviso garante que, mesmo no pior caso, a conversa não termina em silêncio.
Summary by Sourcery
Ensure customers receive a safe, actionable response when AI processing times out or fails instead of being left in silence.
New Features:
Bug Fixes:
Enhancements:
Tests:
Descoberto ao verificar o CRM-236 na stack rodando: depois de rebuildar com este fix, o container ainda reportava
AI_CALL_TIMEOUT_SECONDS=30.Todo compose entregue setava a env explicitamente em 30, e env explícita ganha do default do código. Elevar o default para 90 aqui não muda nada onde importa.
Corrigido em PR separado no repo raiz — evo-crm-community#175 (
docker-compose.swarm.yamleinternal/review/docker-compose.yml). Mergear os dois juntos..env.example:224carrega o mesmo30e é arquivo protegido no ambiente onde trabalhei, então precisa de um mantenedor — sem isso, toda instalação nova nasce com o teto antigo.Ordem de merge — PRs irmãos do CRM-236
#9 (este) e #175 devem subir juntos; o #52 é independente e pode ir em qualquer ordem.
evo-bot-runtimeevo-ai-processor-communityevo-crm-communityVerificação ao vivo do aviso (item C do card)
Feita após este PR: bot-runtime rebuildado, teto forçado a 1s, evento real injetado numa conversa.
A conversa foi de 7 para 8 mensagens e a nova é o aviso,
message_type=1(outgoing), persistida no CRM. O cliente vê texto em vez de silêncio.Summary by Sourcery
Ensure customers receive a clear response when AI processing fails or times out instead of being left in silence.
New Features:
Bug Fixes:
Enhancements:
Deployment:
Tests:
Review — críticos 2, 3 e item 9
🔴 2 — o aviso podia apagar o turno seguinte
Confirmado, e o cenário é justamente o que a feature mira.
sendAIFailureNoticedespachava viarunDispatchStage, que é dono do bookkeeping do turno: o caminho de sucesso rodaSetState(StageDone)→ClearState→entries.Delete(pairKey). EsseDeleteé o que os comentários da própria função proíbem: "A Delete here would race with the new event's Store and could delete the replacement entry."O cliente espera, desiste, manda "oi?" durante o despacho do aviso →
startDebounceguarda o turno novo → o aviso termina e deleta essa entry, órfã o turno novo, e sobem dois pipelines no mesmo par: resposta duplicada.Agora o aviso despacha direto. Não havia o que book-keepar de todo jeito: os dois call sites já rodam
clearStateWithLogantes.Prova negativa: restaurando a chamada, falha
DoesNotTouchTheEntryOfTheNextTurncom a mensagem do turno órfão.🟡 6 — fechou junto
cleanupCtxsão 5s documentados para cleanup (ClearState,SetState). UmDispatchsegmenta porTextSegmentationLimite dormeDelayPerCharacterpor runa entre as partes: com segmentação ligada, o aviso de 84 runas virava 2-3 partes e só os delays estouravam os 5s →ErrDispatchInterrupted→ o log dizia "New message arrived" sem que mensagem nenhuma tivesse chegado. AgoranoticeCtx()(30s).🔴 3 — o teto seguia anulado dentro deste repo
.env.example:8ek8s/configmap.yaml:8corrigidos para 90. O critério é do próprio PR: se env explícita ganha do default, deixar 30 no ConfigMap que serve staging/prod anula o fix onde ele mais importa.Não tocado:
evolution-ecosystem/k8s/base/bot-runtime.yaml:44-45também fixa 30, mas é o SaaS, fora destes três PRs.🟡 9 — aviso default em pt-BR hardcoded
O bot-runtime roda em instalações community/self-hosted no mundo todo, e clientes de instalações que nunca escolheram português eram respondidos nele. Agora inglês, com
AI_FAILURE_NOTICEdocumentada em.env.examplepara localização — e string vazia ainda restaura o silêncio. Há teste que falha se o pt-BR voltar.🟢 11, 12, 13
Doc comment devolvido ao
clearStateWithLog(meu bloco novo tinha caído entre ele e a função, então o godoc deaiFailureNoticeEnvcomeçava com "clearStateWithLog calls ClearState…"),containstrocado porstrings.Contains, eos.Unsetenvagora restaura viat.Cleanup.Testes
10 (eram 6): entry do turno seguinte intacta, estado não reescrito, orçamento do dispatch, default sem pt-BR.
go build/go vetlimpos, suíte completa verde com Redis real.gofmt:pipeline_service.gojá está fora de formato em develop (CRLF); deixado como está para não jogar o arquivo inteiro neste diff.Em aberto, dito por mim
10 (nada exercita
sendAIFailureNoticeviarunAIStage) e 14 (AI_FAILURE_NOTICElido poros.LookupEnvem vez de passar pelointernal/config).