Repository navigation
Onchain Analytics System #40
Description
Context
GoodDollar's onchain data was scattered across subgraphs, Google Apps Script pipelines, block explorers, and loose Dune dashboards -- no centralized, reliable view of protocol activity existed. The idea emerged in early 2026 to build a proper analytics system.
Journey so far:
- Hypersync discovery -- Hadar pointed to Hypersync as an indexing solution; studied it, built the first ingestion pipeline
- BigQuery as the warehouse -- researched storage options; chose BigQuery for cost, public dataset capability, and SQL familiarity
- MVP: XDC invites pipeline -- first end-to-end pipeline (Hypersync to BigQuery to Looker Studio dashboard), validating the architecture
- dbt migration -- 10 models across staging/semantic/marts, 57 tests, full documentation and governance framework
- Production deployment -- all models live in BigQuery, dbt docs site operational
Goal
A production-grade onchain analytics system: centralized, automated, documented, and expandable -- so any team member can query protocol activity without asking the data team.
Scope
- Hypersync ingestion pipeline (MVP)
- BigQuery warehouse setup
- XDC invites dashboard (architecture validation)
- dbt migration -- 10 models, staging to semantic to marts
- Documentation and governance framework (analytics contract, business glossary, freshness macros)
- Pipeline hardening -- error handling, retries, alerting
- Scheduled execution -- cron/Cloud Scheduler
- Expansion: additional contract events (phased -- each entity gets its own sub-issue as scoped)
- Centralized metric dictionary (dbt-native, evolves with expansion)
Current Status
Production-deployed with 10 models and 57 passing tests. Pipeline hardening complete (PR #51). Data freshness blocked on Envio XDC indexer outage -- reported, awaiting fix. Hadar greenlit the approach after a demo (Jul 6).
Value
Moves GoodDollar from "ask Thales to pull the data" to "query it yourself." Foundation for every dashboard, metric, and reporting cadence downstream.
References
- Project folder:
projects/onchain-analytics/ - dbt docs: generated locally via
dbt docs generate - dbt models:
projects/onchain-analytics/gd_dbt/
Activity
thalescb commentedon Jul 9, 2026
CI fix shipped (PR #46): dbt-docs workflow updated for the onchain-analytics folder rename. Docs now deploy to /onchain-analytics/ subfolder, making room for other tools (utm-builder) on the same GitHub Pages site. Live at https://gooddollar.github.io/data-team/onchain-analytics/
thalescb commentedon Aug 3, 2026
Pipeline hardening shipped in PR #51. Hardened v5 pipeline replaces the prototype: staging+MERGE write path (idempotent), block-aligned chunking, Slack alerting on failure/staleness, structured logging and exit codes, PipelineRuns observability table. GitHub Actions cron workflow included (01:00 UTC daily). Blocked on Envio -- their XDC HyperSync indexer is stuck at block 105,265,600 (~400K blocks behind the actual chain tip). Reported to their team. Once they fix it, pipeline will auto-catch-up and scheduling can be activated.
thalescb commentedon Aug 4, 2026
Status Update (Aug 4, 2026)
Pipeline v5 hardened and shipped (PR #51). Daily cron at 01:00 UTC configured (needs service account key for CI). Slack alerting and freshness monitoring live.
XDC/Fuse ingestion ON HOLD: HyperSync dropped XDC support entirely. Hadar is pinging the XDC team about subsidizing infra costs. Decision: wait for their answer before committing to full RPC pivot. For Fuse, existing subgraphs can provide backfill data. Celo ingestion working fine.
Backfill strategy pivot: Issues #52/#53 (HyperSync backfill) are superseded -- HyperSync is gone. New plan depends on XDC team response.
thalescb commentedon Sep 25, 2026
Three PRs open that rebuild the ingestion substrate ahead of the expansion item in this epic's scope.
The system currently captures two contracts on one chain into decoded, per-contract tables, which means every new contract needs a schema change. The inventory work found the real surface is 148 contracts and 354 implementation eras across Celo, XDC, Fuse and Ethereum, so that pattern does not scale and a log missed because nobody had defined a column for it is unrecoverable.
Merge order is #70, then #69, then #71: the pipeline reads the seed at startup.
Nothing has been run against the production dataset. No historical backfill is included or authorised; that is the next piece of work, and the Fuse chain in particular needs a decision on paid archive access before it can be planned.
thalescb commentedon Sep 29, 2026
Three PRs merged today: the L0 v4 ingestion contract (#69), the contract registry seed expanded to all four chains (#70), and the pipeline rewritten against that contract (#71).
#72 is now open on top of them -- the test foundation, reference data and safety controls the pipeline needed before it can be run against production.
docs/release-scope.md: Celo, XDC and Ethereum. Fuse is out, with its measured reference data retained rather than deleted.Next: reconciling ingested claim counts and amounts against the contracts' own daily ledgers on both chains, then a bounded ingestion of the recent window. Scheduled execution stays off until the pipeline has run end to end at least once.
thalescb commentedon Sep 29, 2026
Ingestion pipeline hardening merged — #72
masteris now ate493247. Twelve commits, each independently green, covering correctness andwrite-safety work on the ingestion pipeline.
The defects that had real consequences, and are now closed:
measured run left 149 rows across 102 distinct keys, with every coverage row marked complete.
reached that result, and all three exited zero.
time. XDC matches to the wei on 255 of 255 days.
forced a writer identity to hold table-delete permission on the raw layer.
State of the test suite: 543 tests passing across 25 files, plus a suite of 21 tests that pin
the specific defects above. All 21 were red when this work began and all 21 are green now, none
skipped. Statement coverage is 80.19%, up from 57.55%.
Nothing was written to production and no write credential exists yet. The identity used
throughout has
tables.create,tables.update,tables.updateData,tables.deleteanddatasets.deletedenied on the raw dataset, re-verified after every change.Next step, and it needs a decision rather than code: the first production write requires a
separate writer identity scoped to a staging dataset that does not exist yet
(
BlockchainEvents_Staging). Five preconditions are defined and must each be independentlyverifiable before that credential is issued. That work is deliberately not in this PR.
thalescb commentedon Oct 6, 2026
PR #74 adds a current start-here guide and corrects the onchain analytics status documentation; production ingestion and scheduling remain separate open work.
thalescb commentedon Oct 6, 2026
PR #74 is merged to master. The current onboarding/status guide is now available at projects/onchain-analytics/docs/START_HERE.md; production ingestion and scheduling remain separate open work.
thalescb commentedon Oct 8, 2026
PR #75 is merged and the interactive current/target lineage map is live at https://gooddollar.github.io/data-team/onchain-analytics/map/. Selecting a model shows its grain, purpose, dependencies, status, and caveats; live models link to dbt Docs.