Skip to content

feat(ingest): per-run ingest report and PR callback / ingest：按运行统计的报告与 PR 回调 #1495

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Oseltamivir merged 2 commits into from
Oct 25, 2026
Merged

feat(ingest): per-run ingest report and PR callback / ingest：按运行统计的报告与 PR 回调 #1495

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b01dc87
feat(ingest): per-run ingest report and InferenceX callback
Oseltamivir Oct 25, 2026
c942745
Drop the ingest-report artifact upload; keep the job summary and PR c…
Oseltamivir Oct 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions .github/workflows/ingest-results.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,7 @@ jobs:
UNMAPPED_ENTITIES_OUTPUT: ${{ github.workspace }}/unmapped-entities.json
POWER_PUBLICATION_MANIFEST: ${{ github.workspace }}/power-publication.json
DATABASE_WRITE_RECEIPT: ${{ github.workspace }}/database-writes.json
INGEST_REPORT_OUTPUT: ${{ github.workspace }}/ingest-report.json
run: |
: "${INGEST_REPO:?BENCHMARK_REPO repository variable is not set}"
bun run admin:db:ingest:ci
Expand Down Expand Up @@ -376,3 +377,47 @@ jobs:
{
"text": ":rotating_light: *Ingest results workflow failed*\nRun ID: ${{ steps.artifacts.outputs.source-run-id || github.event.client_payload.source-run-id || github.event.client_payload.run-id || inputs.run-id }}\n${{ steps.powerx.outcome == 'failure' && 'PowerX publication check failed\n' || '' }}${{ steps.database_writes.outputs.summary_json && format('```{0}```\n', steps.database_writes.outputs.summary_json) || '' }}<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run>"
}

# Report the outcome to InferenceX, whose callback comments on the PR that
# published the changelog entry. Only merge ingests arrive by dispatch;
# staging and manual runs have no merged PR to report to.
- name: Report ingest to InferenceX
if: always() && github.event_name == 'repository_dispatch' && (github.event.action == 'ingest-results' || github.event.action == 'ingest-agentic-results')
continue-on-error: true
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
MERGE_RUN_ID: ${{ github.event.client_payload.merge-run-id || github.event.client_payload.run-id }}
SOURCE_RUN_ID: ${{ steps.artifacts.outputs.source-run-id ||
github.event.client_payload.source-run-id || github.event.client_payload.run-id }}
INGEST_SUCCEEDED: ${{ job.status == 'success' }}
INGEST_REPORT: ${{ github.workspace }}/ingest-report.json
BENCHMARK_REPO: ${{ vars.BENCHMARK_REPO }}
with:
github-token: ${{ secrets.PAT }}
script: |
const fs = require('fs');
if (!process.env.BENCHMARK_REPO) {
throw new Error('BENCHMARK_REPO repository variable is not set');
}
let report = null;
try {
report = JSON.parse(fs.readFileSync(process.env.INGEST_REPORT, 'utf8'));
} catch {
core.warning('Ingest report not found; reporting status only');
}
const [owner, repo] = process.env.BENCHMARK_REPO.split('/');
await github.rest.repos.createDispatchEvent({
owner,
repo,
event_type: 'ingest-results-completed',
client_payload: {
'source-repository': `${context.repo.owner}/${context.repo.repo}`,
'merge-run-id': process.env.MERGE_RUN_ID,
'source-run-id': process.env.SOURCE_RUN_ID,
'app-run-id': String(context.runId),
'app-run-attempt': String(context.runAttempt),
'database-target': process.env.REQUESTED_DATABASE_TARGET,
'ingest-succeeded': process.env.INGEST_SUCCEEDED,
report: report && { counts: report.counts, detail: report.detail, unmapped: report.unmapped },
},
});
13 changes: 13 additions & 0 deletions docs/data-pipeline.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -654,6 +654,19 @@ and API values; ingest success alone does not establish complete coverage. To un
repair, restore the snapshotted rows or original bytes under the same identities.
Do not delete a shared series to repair one point.

### Per-run ingest report

`admin:db:ingest:ci` writes `INGEST_REPORT_OUTPUT` (JSON, `packages/db/src/etl/ingest-run-report.ts`)
and appends the same counts to the job summary, for success and failure alike. It counts what this
run wrote, unlike `admin:db:verify`, which prints whole-database totals. Rows are `benchmark`
(benchmark_results rows), `eval` (eval_results rows), `trace` (agentic benchmark rows linked to a
trace_replay blob) and `power` (gpu_metrics series; `unmapped` counts power result rows the mapper
could not place). Columns are `new`, `duplicate`, `skipped` (failed-run rows, rows held by per-job
files, purged points, missing siblings, telemetry not collected), `unmapped` and `failed` (database
write or telemetry errors). For
dispatched merge ingests it also sends `ingest-results-completed` to `BENCHMARK_REPO`, whose
`ingest-results-callback` workflow comments the counts on the merged PR.

### PowerX publication receipts

The normal CI importer writes `POWER_PUBLICATION_MANIFEST` when configured. Each
Expand Down
26 changes: 13 additions & 13 deletions docs/deployment-configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,19 +94,19 @@ before doing any other work. `stage-results` also checks `PAT` up front.
`backfill-server-log-files`, and `recompute-agentic-metrics` check `DATABASE_WRITE_URL`
before writing to the production database.

| Secret | Required | Used for | Workflows |
| --------------------------------- | -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `SLACK_WEBHOOK_URL` | yes | Incoming webhook for ingest failure and unmapped-entity alerts. | `ingest-results`, `ingest-operatorx`, `stage-results` (passed to `ingest-results`) |
| `DATABASE_WRITE_URL` | yes, for writes to the production database | Production database write connection. | `apply-run-overrides`, `ingest-results` on the `production` target, `backfill-server-log-files` and `recompute-agentic-metrics` without `neon-branch` |
| `DATABASE_STAGING_WRITE_URL` | yes, for the `staging` target | Write connection for the Neon `staging` branch. | `ingest-results`, `stage-results` |
| `DATABASE_OPERATORX_WRITE_URL` | yes, by `ingest-operatorx` | OperatorX database write connection for migrations and ingest. | `ingest-operatorx` |
| `INFX_MAIN_PAT` | yes | Reads Actions runs and artifacts in `BENCHMARK_REPO`. | `ingest-results`, `ingest-operatorx`, `stage-results`, `backfill-server-log-files`, `recompute-agentic-metrics` with `restore-missing-trace` |
| `PAT` | yes, by `stage-results` | Sends the `stage-results-started` and `stage-results-completed` repository dispatch events to `BENCHMARK_REPO`. | `stage-results` |
| `NEON_API_KEY` | yes, for Neon branch operations | Resolves named Neon child branches and restores or syncs the `staging` branch. | `ingest-results` on the `neon-branch` target, `backfill-server-log-files` and `recompute-agentic-metrics` with `neon-branch`, `stage-results` with `reset-staging-database`, `sync-staging-database` |
| `NEON_PROJECT_ID` | with `NEON_API_KEY` | Neon project that holds those branches. | Same as `NEON_API_KEY` |
| `VERCEL_INVALIDATE_SECRET` | yes, for cache invalidation outside the `staging` target | Bearer token for `/api/v1/invalidate`. It must equal the app's runtime `INVALIDATE_SECRET`. | `apply-run-overrides`, `backfill-server-log-files`, `cache-refresh`, `ingest-operatorx`, `ingest-results`, `recompute-agentic-metrics` |
| `VERCEL_PROTECTION_BYPASS_SECRET` | optional | Deployment Protection bypass sent with requests to `APP_URL` and by the cache scripts. | `apply-run-overrides`, `backfill-server-log-files`, `cache-refresh`, `h3-media`, `ingest-operatorx`, `ingest-results` on the `production` target, `recompute-agentic-metrics`, `warm-after-deploy` |
| `VERCEL_STAGING_BYPASS_SECRET` | yes, for the `staging` and `neon-branch` targets | Vercel automation bypass for the staging and preview deployments. | `ingest-results`, `stage-results`, `sync-staging-database` |
| Secret | Required | Used for | Workflows |
| --------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `SLACK_WEBHOOK_URL` | yes | Incoming webhook for ingest failure and unmapped-entity alerts. | `ingest-results`, `ingest-operatorx`, `stage-results` (passed to `ingest-results`) |
| `DATABASE_WRITE_URL` | yes, for writes to the production database | Production database write connection. | `apply-run-overrides`, `ingest-results` on the `production` target, `backfill-server-log-files` and `recompute-agentic-metrics` without `neon-branch` |
| `DATABASE_STAGING_WRITE_URL` | yes, for the `staging` target | Write connection for the Neon `staging` branch. | `ingest-results`, `stage-results` |
| `DATABASE_OPERATORX_WRITE_URL` | yes, by `ingest-operatorx` | OperatorX database write connection for migrations and ingest. | `ingest-operatorx` |
| `INFX_MAIN_PAT` | yes | Reads Actions runs and artifacts in `BENCHMARK_REPO`. | `ingest-results`, `ingest-operatorx`, `stage-results`, `backfill-server-log-files`, `recompute-agentic-metrics` with `restore-missing-trace` |
| `PAT` | yes, by `stage-results`; optional for `ingest-results` | Sends the `stage-results-started`, `stage-results-completed` and `ingest-results-completed` repository dispatch events to `BENCHMARK_REPO`. | `stage-results`, `ingest-results` (dispatched merge ingests only; a failed report never fails the ingest) |
| `NEON_API_KEY` | yes, for Neon branch operations | Resolves named Neon child branches and restores or syncs the `staging` branch. | `ingest-results` on the `neon-branch` target, `backfill-server-log-files` and `recompute-agentic-metrics` with `neon-branch`, `stage-results` with `reset-staging-database`, `sync-staging-database` |
| `NEON_PROJECT_ID` | with `NEON_API_KEY` | Neon project that holds those branches. | Same as `NEON_API_KEY` |
| `VERCEL_INVALIDATE_SECRET` | yes, for cache invalidation outside the `staging` target | Bearer token for `/api/v1/invalidate`. It must equal the app's runtime `INVALIDATE_SECRET`. | `apply-run-overrides`, `backfill-server-log-files`, `cache-refresh`, `ingest-operatorx`, `ingest-results`, `recompute-agentic-metrics` |
| `VERCEL_PROTECTION_BYPASS_SECRET` | optional | Deployment Protection bypass sent with requests to `APP_URL` and by the cache scripts. | `apply-run-overrides`, `backfill-server-log-files`, `cache-refresh`, `h3-media`, `ingest-operatorx`, `ingest-results` on the `production` target, `recompute-agentic-metrics`, `warm-after-deploy` |
| `VERCEL_STAGING_BYPASS_SECRET` | yes, for the `staging` and `neon-branch` targets | Vercel automation bypass for the staging and preview deployments. | `ingest-results`, `stage-results`, `sync-staging-database` |

Values reach scripts only through step `env:`, never by inline expression in a `run:`
block.
Expand Down
44 changes: 44 additions & 0 deletions packages/db/src/etl/ingest-run-report.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
import { describe, expect, it } from 'vitest';

import {
createIngestRunReport,
INGEST_REPORT_CATEGORIES,
INGEST_REPORT_OUTCOMES,
renderIngestRunReportMarkdown,
} from './ingest-run-report';

describe('ingest run report', () => {
it('starts every category and outcome at zero', () => {
const report = createIngestRunReport(40141873883, 1);
for (const category of INGEST_REPORT_CATEGORIES) {
for (const outcome of INGEST_REPORT_OUTCOMES) {
expect(report.counts[category][outcome]).toBe(0);
}
}
expect(report.status).toBe('succeeded');
});

it('renders one table row per category with counts in outcome order', () => {
const report = createIngestRunReport(40141873883, 2);
report.counts.benchmark.new = 3;
report.counts.benchmark.duplicate = 1;
report.counts.trace.skipped = 2;
report.prLinks = ['https://github.com/SemiAnalysisAI/InferenceX/pull/4456'];
const markdown = renderIngestRunReportMarkdown(report);
expect(markdown).toContain('run 40141873883 attempt 2');
expect(markdown).toContain('| benchmark | 3 | 1 | 0 | 0 | 0 |');
expect(markdown).toContain('| trace | 0 | 0 | 2 | 0 | 0 |');
expect(markdown).toContain('PR: https://github.com/SemiAnalysisAI/InferenceX/pull/4456');
});

it('keeps failure text and unmapped names inside one table-safe line', () => {
const report = createIngestRunReport(1, 1);
report.status = 'failed';
report.error = 'boom | `x`\nnext';
report.unmapped.models = ['a|b'];
const markdown = renderIngestRunReportMarkdown(report);
expect(markdown).toContain('Status: **failed**');
expect(markdown).toContain("Error: `boom 'x' next`");
expect(markdown).toContain('- unmapped models: a b');
});
});
Loading
Loading