Skip to content

refactor: single Go binary for all block-runner types - #65

Draft
grubmeshi wants to merge 2 commits into
mainfrom
refactor/single-go-binary/consolidated
Draft

refactor: single Go binary for all block-runner types#65
grubmeshi wants to merge 2 commits into
mainfrom
refactor/single-go-binary/consolidated

Conversation

@grubmeshi

@grubmeshi grubmeshi commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Single squashed commit: replaces the JVM/Gradle multi-module tree with one Go module at the repo root. Based directly on current main.

What

  • cmd/<persona> fit entrypoints (tf/manual/gitlab/azdevops/github) + cmd/bbrunner (controller / forced single persona / out-of-cluster all-types in-process superset that can replace meshfed-release's multiplexing-block-runner).
  • internal/*: shared core (meshapi, crypto, config, report, mgmt, dispatch, k8sjob) + one dispatch.RunHandler package per persona. Layering enforced by depguard + an AST logging-stack test; single log/slog stack.
  • Contracts preserved byte-for-byte (claim/register/status wire shapes, k8s Job contract, image names/ports, metric names); new series + env/YAML aliases are additive and warn.
  • Green: task lint 0 issues, task test -race, all 11 domain/shared packages ≥90 % coverage, golangci-lint v2, Go-only CI.

Docs

docs/ARCHITECTURE.md (maintained record), docs/DEPRECATIONS.md, CROSS_REPO_TODO.md, and FOLLOW_UP.md — the forward plan: the live-acceptance gate (required before merge), the tf in-process default + Manager deletion (with the deferred shared-report adoption and ABORTED-on-shutdown), and the cross-repo meshfed-release hand-offs.

History

The full phase-by-phase history + the planning archive (run logs, detail plans, errata) are preserved on backup/consolidated-full-history. Supersedes the earlier stacked PRs #51#64.

Before merge (FOLLOW_UP.md §A)

Live local-dev-stack acceptance (TERRAFORM + MANUAL + kind controller smoke) and confirmation that meshStack's coordinator accepts the decrypt-failure FAILED wire change.

🤖 Generated with Claude Code

Comment thread README.md Outdated
Comment thread README.md Outdated
Comment thread README.md Outdated
Comment thread README.md Outdated
Comment thread README.md Outdated
Comment thread README.md Outdated
Comment thread docs/ARCHITECTURE.md Outdated
Comment thread docs/ARCHITECTURE.md Outdated
Comment thread docs/ARCHITECTURE.md Outdated
Comment thread internal/azdevops/client.go Outdated
Comment thread internal/azdevops/client.go Outdated
Comment thread docs/ARCHITECTURE.md Outdated
Comment thread docs/ARCHITECTURE.md Outdated
Comment thread docs/ARCHITECTURE.md Outdated
Comment thread docs/ARCHITECTURE.md Outdated
Comment thread docs/ARCHITECTURE.md Outdated
Comment thread docs/ARCHITECTURE.md Outdated
Comment thread docs/ARCHITECTURE.md Outdated
Comment thread docs/ARCHITECTURE.md Outdated
Comment thread docs/ARCHITECTURE.md Outdated
Comment thread docs/ARCHITECTURE.md Outdated
Comment thread docs/DEPRECATIONS.md Outdated
@grubmeshi

Copy link
Copy Markdown
Collaborator Author

Review feedback addressed (4 follow-up commits)

Thanks for the review — here's how each point was addressed. New commits on this branch:

Commit Scope
1235c56 README: terminology + table + dispatch-description fix
89411ce ARCHITECTURE: present-tense rewrite + observability + future-direction
771fdd3 DEPRECATIONS: per-surface sections, no timeline framing
0cc9867 refactor: shared internal/valuestring helper

Terminology — drop "persona", use "runner type"/"type"

Aligned all docs (README + ARCHITECTURE) on the meshStack-established "runner type"/"type", with ALL = the run-controller. The code-wide rename (~50 files, the persona type + personaTf constants + the bbrunner <type> subcommand naming) is mechanical but large, so it's captured as a scoped follow-up with an effort estimate in ARCHITECTURE.md §8 — best done as its own reviewable commit rather than ballooning this diff.

README

  • Table restructured: five fit runners first, run-controller (ALL) separated and last; column renamed to Type + cmd/ entrypoint.
  • Fixed the single-run/dispatch paragraph that read as tf-only — clarified single-run is shared by every type and the controller dispatches a Job for any non-ALL type by selecting that type's image. Verified in code: internal/k8sjob/kubernetes.go picks the image from the operator-supplied implementations[run.Type] map — no per-type branch.
  • Removed the "what it's not" negations from the repo-structure section.

ARCHITECTURE

  • Removed all historic/refactor narrative (no more "ported from Kotlin", the old argv[0]/mux approach, "P2.1 closed", the poll-interval Kotlin @Scheduled detail, "ported runners"). It now reads as a present-tense design record.
  • Relaxed the strict no-subpackages rule (a domain package like internal/tf may grow subpackages as it earns them).
  • Package map is now illustrative + domain-driven-shaping, aligned with the actual depguard prometheus grouping.
  • New §6 Observability: documents the pull model, the real gap (Job-dispatched single-run pods serve no listener → no scrape path for per-run metrics), and a push-gateway design with a ~2–3d effort estimate (prometheus/client_golang/prometheus/push is already available, no go.mod bump). Also notes the mgmtobservability / port rename idea — mgmt already takes a plain address, so it's a <1d mechanical rename, deferred to keep this diff focused.
  • §8 Future direction replaces the follow-up/history dump: each item is a design sketch + effort estimate (lean run-controller via build tags ~3–5d incl. an init() registration seam; generic JSON HTTP client ~2–4d; etc.).
  • Dropped the cross-repo TODO references.

DEPRECATIONS

Dropped the removal-timeline framing (reduced to a one-line policy) and converted the alias table into one content-focused section per surface.

HTTP client dedup

Landed the safe first step: the duplicated scalar value-stringification (azdevops.renderValue / gitlab.valueString, functionally identical) is now shared as internal/valuestring.Render. The larger generic client direction (shared no-redirect client builder + lifting meshapi's retry transport into a shared package) is documented as a design + ~2–4d estimate in §8 — the per-type request/error bodies deliberately stay per-package since each pins a frozen, byte-exact wire/error-message contract (and adding retry/timeout where a client has none today is a behaviour change to schedule deliberately).

All gates green: task lint (0 issues), task test (-race), task coverage (every package ≥ 90%, incl. the new internal/valuestring).

@grubmeshi

Copy link
Copy Markdown
Collaborator Author

Correction (re: value-stringification). My note above said github "does not stringify arbitrary values and does not use it" — that was inaccurate. github does have a stringifier (internal/github/inputs.go valueToString); it's just a deliberately different frozen Kotlin twin, so it stays per-package:

  • nil"" (vs valuestring.Render's "null"), and composites → fmt.Sprint Go format (vs compact JSON). Both are wire-visible.
  • github applies it only to known scalars (the Mode-B tokens/endpoint and the sensitive-input decrypt path — regular user inputs are never dispatch inputs there), so fmt.Sprint/"" is the Kotlin-faithful choice for github specifically.

Only azdevops and gitlab were byte-identical and were the correct pair to fold onto the shared valuestring.Render. Documented the exception on valueToString in 1381693.

grubmeshi added a commit that referenced this pull request Jul 14, 2026
…ncident research

Investigate the gitlab/github/azdevops external-API call surface and design a
retry/timeout policy that never double-triggers a pipeline:

- PLAN_DETAIL_HTTP_RETRY.md: full call catalog (idempotency per call); the three
  trigger calls are non-idempotent POSTs (gitlab /trigger/pipeline, github
  /dispatches, azdevops /runs) -> fail hard, retry only on provably-pre-side-effect
  rejections (transient-auth 401/403, rate-limit 429). Sourced incident research
  (GitHub intermittent 401 + secondary rate limits; GitLab 401-permanent + 401->403
  IP-block trap; Azure DevOps 203/HTML sign-in quirk on bad PAT) with a cross-API
  classifier table. Timeout is the safer half: gitlab+github use NoRedirectClient(0)
  = no timeout; github also lacks NewRequestWithContext (not cancellable) -> its
  timeout is fixable via the client builder but ctx-cancellation is a per-call edit.
- PLAN.md: widen T2 to include azdevops, reframe "Next" tickets as sequential on the
  branch (PR #65 is not merging soon, so no post-merge worktree fan-out).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
grubmeshi added a commit that referenced this pull request Jul 14, 2026
…direction (X8)

Investigate gitlab polling feasibility; decide to keep the async design.

- gitlab is deliberately non-polling (D9): its GitlabImplementation wire DTO carries
  only pipelineTriggerToken, which is trigger-only and cannot read the API, so status
  arrives entirely pipeline-side via the callback URLs it hands the pipeline. Decision:
  keep it, do not add polling (would need a whole extra read_api credential in the wire).
- Fix PLAN_DETAIL_ABORT_FLAG.md: it wrongly grouped gitlab with the pollers. gitlab has
  no poll loop -> like manual, T1 abort is a no-op window; only github+azdevops poll.
- CROSS_REPO_TODO X8 (new, reusing the dropped env-guard number): the option to move
  github+azdevops to gitlab's async-handover model. The dispatcher already supports a
  fast-returning Execute (gitlab proves it); the real cost is cross-repo (instrument the
  workflow/pipeline templates to call back). Wins: no poll loop / read cred, T1-abort
  moot, shorter k8s Jobs, smaller T2 retry surface. Risk: trust the external CI to report.
- Reframe T1-T3 as sequential-on-branch (PR #65 not merging soon).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@grubmeshi
grubmeshi force-pushed the refactor/single-go-binary/consolidated branch from 580e147 to 075801d Compare July 15, 2026 21:31
grubmeshi and others added 2 commits July 16, 2026 23:43
…unner type

One Go module builds a single cmd/bbrunner binary whose linked runner types are chosen at
compile time via build tags: no tags = the in-process superset (all types), -tags type_<x> =
a lean single-type image (tf/manual/gitlab/azdevops/github), -tags k8s = the run-controller
(Kubernetes-Job dispatcher, no handlers). The former per-type cmd/* mains are gone; each type
self-registers via init() in a tag-gated cmd/bbrunner/<type>.go.

- Run mode auto-detected from RUN_JSON_FILE_PATH alone (mounted run file => single-run, else
  poll); the EXECUTION_MODE / SPRING_PROFILES_ACTIVE single-run toggles are removed. The
  dispatcher auto-detects in-cluster (k8sjob) vs out-of-cluster (in-process superset).
- internal/ domain packages (tf/manual/gitlab/azdevops/github) over shared ports (meshapi,
  report, config, dispatch, secret, observability, httpclient, runmode); depguard enforces
  domain-must-not-import-adapters.
- One generic meshapi HTTP facade (a single http.Client.Do site) with the shared retry/timeout
  policy; the hand-rolled CI clients route through it.
- Decryption lifted to the claim boundary (internal/rundecrypt) so typed handlers are
  key-oblivious; a shared handover sanitizer strips the implementation object from forwarded
  run objects.
- Backend-agnostic dispatch.Loop + InProcess/Kubernetes dispatchers; a claimed run of an
  unlinked type fails fast as FAILED. Graceful shutdown reports ABORTED after a configurable
  grace.
- internal/config is the single YAML+config owner; per-package 90% coverage gate.
- The tf pre-run script receives the decrypted meshfed run object verbatim (cr.RawJson), not a
  re-serialized DTO subset, so every backend field (incl. spec.trigger) reaches it.
- One parameterized containers/http-runner/Dockerfile for the four HTTP-only images; tf and
  run-controller keep their own; the dev flake pins docker-buildx (bake consolidation planned).

State-describing docs live with the code: docs/ARCHITECTURE.md, docs/DEPRECATIONS.md, and the
AGENTS.md / CLAUDE.md conventions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PLAN.md — the in-repo follow-up plan and the branch/rebase discipline for this
two-commit PR. CROSS_REPO_TODO.md — work that needs another repo or a live
meshStack/cluster. These are the second of the PR's two permanent commits.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@grubmeshi
grubmeshi force-pushed the refactor/single-go-binary/consolidated branch from 075801d to d74c7bd Compare July 16, 2026 21:43
@grubmeshi grubmeshi changed the title refactor: single Go binary for all block-runner personas refactor: single Go binary for all block-runner types Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant