Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@
# DATABASE_DRIVER=neon
# DATABASE_SSL=true

# GitHub PAT — required for CollectiveX lazy ingest (sweep artifact reads); also used
# for framework releases, star counts, and unofficial-run artifact reads
# GitHub PAT — used for framework releases, star counts, unofficial-run artifact reads,
# and (optional, raises the rate limit) the CollectiveX committed-results ingest
# Create at: https://github.com/settings/personal-access-tokens
# GITHUB_TOKEN=
# Extra benchmark source repos (owner/repo, comma-separated) searched after
Expand All @@ -31,8 +31,10 @@
# Directory contains <runId>.json with { run, benchmarks, evaluations }.
# INFERENCEX_LOCAL_ARTIFACT_DIR=/absolute/path/to/local-artifacts

# CollectiveX database (separate Neon instance). The app routes read and write through
# DATABASE_COLLECTIVEX_WRITE_URL; they do not read DATABASE_COLLECTIVEX_READONLY_URL.
# CollectiveX database (separate Neon instance), filled from InferenceX's committed
# collectivex/results/ by the ingest-collectivex workflow or admin:db:ingest:collectivex.
# The app routes read and write through DATABASE_COLLECTIVEX_WRITE_URL; they do not read
# DATABASE_COLLECTIVEX_READONLY_URL.
# DATABASE_COLLECTIVEX_READONLY_URL=
# DATABASE_COLLECTIVEX_WRITE_URL=

Expand Down Expand Up @@ -69,8 +71,8 @@
# ║ Secrets and database administration (DO NOT SHARE) ║
# ║ ║
# ║ DB scripts (migrate, ingest, reset, etc) read these. The app also reads ║
# ║ INVALIDATE_SECRET, FEEDBACK_SECRET, and COLLECTIVEX_ADMIN_SECRET (and ║
# ║ DATABASE_WRITE_URL for feedback) at runtime; set them in Vercel too. ║
# ║ INVALIDATE_SECRET and FEEDBACK_SECRET (and DATABASE_WRITE_URL for ║
# ║ feedback) at runtime; set them in Vercel too. ║
# ╚══════════════════════════════════════════════════════════════════════════╝

# DATABASE_WRITE_URL=
Expand All @@ -84,7 +86,6 @@
# argument or run URL names the repository.
# INGEST_REPO=
# FEEDBACK_SECRET=
# COLLECTIVEX_ADMIN_SECRET=

# OperatorX durable raw-artifact cache (primary, write-capable connection).
# Apply admin:db:migrate:operatorx before enabling the hidden OperatorX tab.
Expand Down
159 changes: 159 additions & 0 deletions .github/workflows/ingest-collectivex.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
name: Ingest CollectiveX Results

on:
repository_dispatch:
types: [ingest-collectivex]
workflow_dispatch:
inputs:
sha:
description: InferenceX commit (40 hex) whose collectivex/results/ to ingest
required: false
ref:
description: InferenceX branch or tag, instead of sha
required: false
backfill:
description: Ingest every collectivex/results/ commit on main instead
type: boolean
default: false
prune:
description: With backfill, then delete every stored run the backfill did not store
type: boolean
default: false
delete_run:
description: Delete this one stored run_id instead (no other target)
required: false

permissions: {}

# One group for every ingest, prune and delete: runs queue behind each other instead of
# writing at once (the database also serializes writers with an advisory lock). GitHub
# keeps a single pending run per group, so a burst of dispatches can supersede a pending
# one; a backfill then stores anything it skipped.
concurrency:
group: ${{ github.workflow }}
cancel-in-progress: false

jobs:
ingest:
name: Ingest CollectiveX results
timeout-minutes: 60
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Check inputs and secrets
env:
APP_URL: ${{ vars.APP_URL }}
VERCEL_INVALIDATE_SECRET: ${{ secrets.VERCEL_INVALIDATE_SECRET }}
DATABASE_COLLECTIVEX_WRITE_URL: ${{ secrets.DATABASE_COLLECTIVEX_WRITE_URL }}
GH_TOKEN: ${{ secrets.INFX_MAIN_PAT }}
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
SHA: ${{ github.event.client_payload.sha || inputs.sha }}
REF: ${{ inputs.ref }}
BACKFILL: ${{ inputs.backfill }}
PRUNE: ${{ inputs.prune }}
DELETE_RUN: ${{ inputs.delete_run }}
run: |
: "${SLACK_WEBHOOK_URL:?SLACK_WEBHOOK_URL secret is not set}"
: "${APP_URL:?APP_URL repository variable is not set}"
: "${VERCEL_INVALIDATE_SECRET:?VERCEL_INVALIDATE_SECRET secret is not set}"
: "${DATABASE_COLLECTIVEX_WRITE_URL:?DATABASE_COLLECTIVEX_WRITE_URL secret is not set}"
: "${GH_TOKEN:?INFX_MAIN_PAT secret is not set}"
selected=0
[ -n "$SHA" ] && selected=$((selected + 1))
[ -n "$REF" ] && selected=$((selected + 1))
[ "$BACKFILL" = true ] && selected=$((selected + 1))
[ -n "$DELETE_RUN" ] && selected=$((selected + 1))
if [ "$selected" -ne 1 ]; then
echo "give exactly one of sha, ref, backfill, or delete_run" >&2
exit 1
fi
if [ "$PRUNE" = true ] && [ "$BACKFILL" != true ]; then
echo "prune needs backfill" >&2
exit 1
fi
if [ -n "$DELETE_RUN" ] && ! [[ "$DELETE_RUN" =~ ^[1-9][0-9]{0,9}$ ]]; then
echo "delete_run must be a stored run_id" >&2
exit 1
fi
if [ -n "$SHA" ] && ! [[ "$SHA" =~ ^[0-9a-f]{40}$ ]]; then
echo "sha must be 40 lowercase hex characters" >&2
exit 1
fi
if [ -n "$REF" ] && { ! [[ "$REF" =~ ^[A-Za-z0-9_][A-Za-z0-9_./-]{0,254}$ ]] || [[ "$REF" == *..* ]]; }; then
echo "ref is not a valid branch or tag name" >&2
exit 1
fi

- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Setup Bun
id: setup-bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version-file: package.json
- name: Cache Bun packages
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.bun/install/cache
key: ${{ runner.os }}-bun-${{ runner.arch }}-${{
steps.setup-bun.outputs.bun-version }}-${{ hashFiles('bun.lock') }}
restore-keys: |
${{ runner.os }}-bun-${{ runner.arch }}-${{ steps.setup-bun.outputs.bun-version }}-
- name: Install dependencies
run: bun install --frozen-lockfile
env:
CYPRESS_INSTALL_BINARY: '0'

- name: Run migrations
env:
DATABASE_COLLECTIVEX_WRITE_URL: ${{ secrets.DATABASE_COLLECTIVEX_WRITE_URL }}
run: bun run --cwd packages/db src/migrate-collectivex.ts --yes

- name: Ingest results
env:
DATABASE_COLLECTIVEX_WRITE_URL: ${{ secrets.DATABASE_COLLECTIVEX_WRITE_URL }}
GITHUB_TOKEN: ${{ secrets.INFX_MAIN_PAT }}
SHA: ${{ github.event.client_payload.sha || inputs.sha }}
REF: ${{ inputs.ref }}
BACKFILL: ${{ inputs.backfill }}
PRUNE: ${{ inputs.prune }}
DELETE_RUN: ${{ inputs.delete_run }}
run: |
if [ -n "$DELETE_RUN" ]; then
target=(--delete-run "$DELETE_RUN")
elif [ "$BACKFILL" = true ] && [ "$PRUNE" = true ]; then
target=(--backfill --prune)
elif [ "$BACKFILL" = true ]; then
target=(--backfill)
elif [ -n "$SHA" ]; then
target=(--sha "$SHA")
else
target=(--ref "$REF")
fi
bun run --cwd packages/db src/ingest-collectivex.ts "${target[@]}" --yes

- name: Invalidate CollectiveX cache
env:
APP_URL: ${{ vars.APP_URL }}
VERCEL_INVALIDATE_SECRET: ${{ secrets.VERCEL_INVALIDATE_SECRET }}
VERCEL_PROTECTION_BYPASS_SECRET: ${{ secrets.VERCEL_PROTECTION_BYPASS_SECRET }}
run: |
: "${APP_URL:?APP_URL repository variable is not set}"
headers=(-H "Authorization: Bearer $VERCEL_INVALIDATE_SECRET")
if [ -n "$VERCEL_PROTECTION_BYPASS_SECRET" ]; then
headers+=(-H "x-vercel-protection-bypass: $VERCEL_PROTECTION_BYPASS_SECRET")
fi
curl -sSf -X POST "${APP_URL%/}/api/v1/invalidate?scope=collectivex" "${headers[@]}"

- name: Notify Slack on failure
if: failure()
uses: slackapi/slack-github-action@dcb1066f776dd043e64d0e8ba94ca15cc7e1875d # v4.0.0
with:
webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
webhook-type: incoming-webhook
payload: |
{
"text": ":rotating_light: *Ingest CollectiveX workflow failed*\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run>"
}
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Raw-data API routes return raw DB rows with zero transformation, validation, or
Some routes serve a purpose that raw DB rows cannot satisfy:

- Dashboard projections under `/api/v1/views/*` use shared pure UI transformations server-side and publish their selectors and assumptions. Keep raw-data contracts separate from these projections.
- CollectiveX assembles stored sweep documents through its shared reader.
- CollectiveX assembles its stored committed-result documents through its shared reader.
- OperatorX uses versioned ingest-time comparison projections for its page-owned API. The reader merges compact timing rows across runs while preserving source revisions, statuses, and timeline references; raw documents remain available for drilldowns. An optional revision-keyed Blob cache shares the merged comparison across instances; partial reads are excluded and OperatorX invalidation purges its deployment subtree. See [OperatorX](./operatorx.md#compact-comparison-reads).
- `tco-feed` performs server-side frontier interpolation for spreadsheet consumers that cannot run the TypeScript transforms. Its assumptions (tier weights, workload mix, α) enter only as explicit query params with documented defaults, so a published sheet's URL fully records its methodology.
- `pareto` returns frontier and hinterland coordinates over stored benchmark rows with explicit axis preferences. It reuses the chart dominance algorithm without UI filters. See the [Pareto frontier and hinterland reference](../packages/skills/skills/inferencex-api/references/pareto.md).
Expand Down
Loading
Loading