Async executions can stay stuck in running with missing final logs

[PlaneInABottle]

Summary

In production, we hit a webhook-triggered workflow execution that remained stuck in workflow_execution_logs.status = 'running' while the related async job stayed in async_jobs.status = 'processing'. The workflow had clearly started, some internal requests were executed, but the run never reached a terminal state and there was not enough persisted evidence to determine where it got stuck.

What we observed

• Webhook request returned 200 successfully.
• Related workflow_execution_logs row remained:
  • status = 'running'
  • ended_at = NULL
  • no final trace/failure envelope persisted
• Related async_jobs row remained:
  • status = 'processing'
  • no completed_at
  • no error persisted
• There was no paused execution row, so this was not a wait/human-in-the-loop pause case.
• Render logs showed the ingress webhook request plus multiple internal Bun requests (function/tool/memory/custom-tool calls), so the execution had definitely progressed beyond ingress.
• But request logs alone were not enough to reconstruct the last successful block or failure point.

Likely root causes

Based on code-path analysis, there seem to be two main races:

1. Detached local async execution after enqueue

   • For non-Trigger.dev backends, jobs are enqueued and then still executed in-process via a detached async path.
   • If the process is interrupted/recycled, jobs can be left in processing.

2. Detached execution-log finalization

   • Execution completion/error logging appears to be finalized in a fire-and-forget path.
   • That allows workflow_execution_logs to remain running if the process dies before final persistence.

Why this is painful operationally

When this happens, operators cannot reliably answer:

• which webhook request maps to which job and execution
• whether the execution never really started vs started and stalled
• which block started last
• which block completed last
• whether execution completed but log finalization failed

Suggestions

Immediate reliability fixes

• Replace raw detached local execution in async routes with a route-safe post-response mechanism.
• Await execution finalization/log completion before marking jobs complete.
• Apply the same reliability fix consistently across webhook/workflow/schedule async entry points.

Forensic observability fixes

• Persist additive execution progress markers in workflow_execution_logs.execution_data, especially:
  • lastStartedBlock
  • lastCompletedBlock
• Preserve start-time execution metadata instead of overwriting it on completion.
• Persist a structured completion/fallback status so log-completion-failed is detectable without string matching.

Correlation-chain fixes

• Carry a stable chain across ingress → queue → execution log:
  • ingressRequestId
  • jobId
  • executionId
• Expose jobId -> executionId in job-status reads.
• Make it possible to traverse:
  • requestId -> jobId -> executionId -> execution log

Cleanup / operator improvements

• Reconcile stale async jobs and stale execution logs together instead of treating them as unrelated cleanup domains.
• Distinguish cleanup classifications such as:
  • never started
  • started but no block started
  • stalled in block
  • stalled after block
  • log completion failed

Longer-term architecture

• Consider moving Redis/database backends toward a true worker/claim model rather than request-owned execution after enqueue.

Why I’m filing this

This issue is not just about a single stuck run; it exposed a broader gap where:

• async execution can be abandoned
• terminal log state can be lost
• postmortem correlation is weak

If useful, I can also open a follow-up issue/PR proposal with a phased implementation plan for:

1. reliability fixes,
2. forensic observability,
3. correlation chain,
4. stale cleanup classification,
5. longer-term worker model.

[guoyangzhen]

Technical Analysis

After reviewing the codebase, I found the race condition occurs at three locations using the same fire-and-forget pattern:

Affected Code Paths

1. apps/sim/lib/webhooks/processor.ts (line ~537)
2. apps/sim/app/api/schedules/execute/route.ts (line ~137)
3. apps/sim/app/api/workflows/[id]/execute/route.ts (line ~167)

All three use the same pattern:

void (async () => {
  try {
    await jobQueue.startJob(jobId)
    const output = await executeXxxJob(payload)
    await jobQueue.completeJob(jobId, output)
  } catch (error) {
    try { await jobQueue.markJobFailed(jobId, errorMessage) }
    catch (markFailedError) { ... }
  }
})()

Root Cause

If the process dies (OOM, SIGTERM, container eviction) during executeXxxJob():

• async_jobs.status stays processing
• workflow_execution_logs.status stays running
• No finally block guarantees state cleanup

Existing Mitigation

The cleanup-stale-executions cron already handles stuck jobs after STALE_THRESHOLD_MS = maxAsyncTimeout + 5min. However, this is reactive, not preventive.

Recommended Fix (Phased)

Phase 1: Correlation Chain
Add job_id column to workflow_execution_logs to enable direct job↔execution tracing. Update all three entry points to pass jobId through LoggingSession.

Phase 2: Progress Markers
Persist lastStartedBlock and lastCompletedBlock in executionData during executeWorkflowCore(). Enables post-mortem analysis even when execution stalls.

Phase 3: Structured Cleanup Classification
Extend the cleanup cron to classify stale executions: never_started, started_no_block, stalled_in_block, log_finalization_failed.

Happy to contribute a PR for Phase 1 if the maintainers are interested.

[guoyangzhen]

Phase 1 Implementation: jobId Correlation Chain

Overview

Add a job_id foreign key column to workflow_execution_logs that references async_jobs.id. This creates the critical missing link between the fire-and-forget job system and the execution logging system.

Changes Required

1. Schema Migration (packages/db/schema.ts)

Add jobId column to workflowExecutionLogs after executionId (line ~292):

jobId: text('job_id').references(() => asyncJobs.id, { onDelete: 'set null' }),

Add index in the table constraints:

jobIdIdx: index('workflow_execution_logs_job_id_idx').on(table.jobId),

2. Payload Types

apps/sim/background/webhook-execution.ts (line ~98):

export type WebhookExecutionPayload = {
  // ... existing fields ...
  jobId?: string  // NEW
}

apps/sim/background/workflow-execution.ts (line ~17):

export type WorkflowExecutionPayload = {
  // ... existing fields ...
  jobId?: string  // NEW
}

3. ExecutionLogger (apps/sim/lib/logs/execution/logger.ts)

Add jobId to startWorkflowExecution params (line ~54):

async startWorkflowExecution(params: {
  workflowId: string
  workspaceId: string
  executionId: string
  jobId?: string  // NEW
  trigger: ExecutionTrigger
  // ...
})

Add to INSERT values (line ~129):

jobId: params.jobId ?? null,

4. Entry Points — Pass jobId

webhooks/processor.ts (line ~1166):

const payloadWithJobId = { ...payload, jobId }
void (async () => {
  await jobQueue.startJob(jobId)
  const output = await executeWebhookJob(payloadWithJobId)
  await jobQueue.completeJob(jobId, output)
})()

schedules/execute/route.ts (line ~81): Same pattern.

workflows/[id]/execute/route.ts (line ~190): Same pattern.

5. Background Jobs — Thread jobId to Logger

In each execute*Job function, pass jobId to the logging session:

const preprocessResult = await preprocessExecution({
  // ... existing params ...
  jobId: payload.jobId,  // NEW
})

Verification Query

After deployment, the cleanup cron can now correlate:

SELECT 
  el.execution_id,
  el.status as exec_status,
  aj.id as job_id,
  aj.status as job_status,
  el.started_at
FROM workflow_execution_logs el
LEFT JOIN async_jobs aj ON el.job_id = aj.id
WHERE el.status = 'running' 
  AND el.started_at < NOW() - INTERVAL '10 minutes'

Risk Assessment

• Low risk: jobId is nullable, no breaking changes
• Backward compatible: Existing executions without jobId continue to work
• No downtime: Schema change is additive only

[PlaneInABottle]

@guoyangzhen I have already implemented phase 1, #3564 please check

[guoyangzhen]

@PlaneInABottle Nice work on the diagnostics foundation! 👍

Looking at #3564, you've implemented progress markers (lastStartedBlock, lastCompletedBlock, finalizationPath) in the executionData JSONB — which I had proposed as Phase 2.

My Phase 1 proposal was actually different: adding a job_id foreign key column to workflow_execution_logs that references async_jobs.id. This creates the missing correlation chain between the fire-and-forget job system and execution logs.

So our approaches are complementary:

• Your PR fix(logs): add durable execution diagnostics foundation #3564: Progress markers (where did execution get to)
• My Phase 1 proposal: jobId correlation (which async job owns this execution)

Both are needed for the full fix. I'll hold off on implementing the jobId piece until #3564 merges to avoid conflicts.

Would you be interested in collaborating on the jobId correlation piece after your PR lands?