Repository navigation
feat(receipt): one receipt for each learning update, with a Nickel contract (E0-09) #1161
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
2bf46b9
859995b
0ad4eb2
54dfae6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -699,6 +699,48 @@ defmodule Hypatia.FleetDispatcher do | |
| # place for it is this function. | ||
| defp trusted_verification(_finding), do: nil | ||
|
|
||
| # The receipt of a neural learning update (E0-09, owner ruling D2). The evidence | ||
| # is the qualified outcome that was handed to the coordinator; there is no row | ||
| # in a log, so the receipt names the digest of that evidence in canonical form. | ||
| # It never raises: a receipt that cannot be written is logged as receipt_pending. | ||
| # There is no row to rebuild it from, so nothing repairs it later. | ||
| defp write_neural_receipt(pattern, repo, outcome, verification) do | ||
| data_root = Path.expand(Hypatia.Paths.verisimdb_data()) | ||
|
|
||
| evidence = %{ | ||
| "file" => ".", | ||
| "outcome" => outcome, | ||
| "recipe_id" => pattern, | ||
| "repo" => repo, | ||
| "verification" => verification | ||
| } | ||
|
|
||
| event_id = Hypatia.EventId.generate() | ||
|
|
||
| with {:ok, bytes} <- Hypatia.Receipt.encode(evidence), | ||
| row = | ||
| Map.merge(evidence, %{ | ||
| "event_id" => event_id, | ||
| "event_id_source" => "hypatia", | ||
| "source_kind" => "neural_path" | ||
| }), | ||
| ctx = %{ | ||
| kind: :neural_path, | ||
| log: "neural/outcome-queue", | ||
| line: "0", | ||
| row_sha256: :crypto.hash(:sha256, bytes) |> Base.encode16(case: :lower) | ||
| }, | ||
| :ok <- Hypatia.Receipt.write(data_root, row, ctx: ctx) do | ||
| :ok | ||
| else | ||
| {:error, reason} -> | ||
| Logger.warning("receipt_pending event_id=#{event_id} reason=#{inspect(reason)}") | ||
| end | ||
|
Comment on lines
+735
to
+738
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When neural queueing succeeds but the receipt append fails transiently, this handler records only a log message and discards the event ID and evidence. Unlike outcome-log rows, the neural event has no durable row that Useful? React with 👍 / 👎. |
||
| rescue | ||
| error -> | ||
| Logger.warning("receipt_pending reason=#{Exception.message(error)}") | ||
| end | ||
|
|
||
| # :queued records a volatile source-bound submission, never completed learning. | ||
| @doc false | ||
| def record_neural_outcome(finding, outcome, server \\ Hypatia.Neural.Coordinator) do | ||
|
|
@@ -766,6 +808,7 @@ defmodule Hypatia.FleetDispatcher do | |
| "verification" => verification | ||
| }) do | ||
| record_neural_outcome(finding, outcome_str, server) | ||
| write_neural_receipt(to_string(pattern), to_string(repo), outcome_str, verification) | ||
|
Comment on lines
810
to
+811
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When the coordinator is unavailable or returns an error, Useful? React with 👍 / 👎. |
||
| else | ||
| Logger.info( | ||
| "Learning loop: outcome for pattern '#{pattern}' in #{repo} not learned: " <> | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| # SPDX-License-Identifier: MPL-2.0 | ||
| # Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk> | ||
|
|
||
| defmodule Hypatia.EventId do | ||
| @moduledoc """ | ||
| The identity of an event: an ADR-008 version 8 identifier under profile T | ||
| (the time-ordered profile), written by Hypatia itself (owner ruling R5-A of | ||
| 2026-11-15, `docs/design/e0-receipt.adoc`). It makes no call to an external | ||
| command, so there is no tool that can be missing. | ||
|
|
||
| The layout is the one ADR-008 fixes (`metadatastician/standards`, | ||
| `docs/decisions/ADR-008-uuid-v8-estate-standard.adoc`): | ||
|
|
||
| 48 bits unix_ts_ms big-endian Unix epoch milliseconds | ||
| 4 bits ver 0b1000 (8) | ||
| 12 bits rand_a a counter that makes ids increase within a millisecond | ||
| 2 bits var 0b10 | ||
| 62 bits rand_b from a cryptographic random source | ||
|
|
||
| The text is lowercase `8-4-4-4-12`. The field that holds an `event_id` | ||
| declares profile T. The timestamp is never an audited event time and never a | ||
| basis for authorisation (ADR-008). | ||
|
|
||
| The counter is shared by the whole node (an `:atomics` word that holds the | ||
| millisecond and the counter together), so ids increase within one process and | ||
| are distinct across processes. If the clock goes backwards, ids keep | ||
| increasing: the counter carries into the millisecond instead. | ||
| """ | ||
|
|
||
| import Bitwise | ||
|
|
||
| @key {__MODULE__, :state} | ||
| @text ~r/\A[0-9a-f]{8}-[0-9a-f]{4}-8[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}\z/ | ||
|
|
||
| @doc "Mint an id: lowercase `8-4-4-4-12` text, version 8, variant `10`, profile T." | ||
| @spec generate() :: String.t() | ||
| def generate do | ||
| packed = next_packed() | ||
| ms = packed >>> 12 | ||
| counter = packed &&& 0xFFF | ||
| <<rand_b::62, _::2>> = :crypto.strong_rand_bytes(8) | ||
|
|
||
| format(<<ms::48, 8::4, counter::12, 2::2, rand_b::62>>) | ||
| end | ||
|
|
||
| @doc "The 16 bytes of an id in the text form." | ||
| @spec to_binary(String.t()) :: <<_::128>> | ||
| def to_binary(text) when is_binary(text) do | ||
| text |> String.replace("-", "") |> Base.decode16!(case: :lower) | ||
| end | ||
|
|
||
| @doc "Is this a valid `event_id`: a string in the text form, version 8, variant `10`?" | ||
| @spec valid?(term()) :: boolean() | ||
| def valid?(text) when is_binary(text), do: Regex.match?(@text, text) | ||
| def valid?(_other), do: false | ||
|
|
||
| # The shared word holds `(milliseconds << 12) | counter`. A new millisecond | ||
| # resets the counter; the same or an earlier one adds 1, which carries into the | ||
| # millisecond when the 12 bits overflow, so the value only ever increases. | ||
| defp next_packed do | ||
| ref = state() | ||
| now = System.system_time(:millisecond) <<< 12 | ||
| advance(ref, now) | ||
| end | ||
|
|
||
| defp advance(ref, now) do | ||
| current = :atomics.get(ref, 1) | ||
| next = if now > current, do: now, else: current + 1 | ||
|
|
||
| case :atomics.compare_exchange(ref, 1, current, next) do | ||
| :ok -> next | ||
| _changed -> advance(ref, now) | ||
| end | ||
| end | ||
|
|
||
| defp state do | ||
| case :persistent_term.get(@key, nil) do | ||
| nil -> | ||
| ref = :atomics.new(1, signed: false) | ||
| :persistent_term.put(@key, ref) | ||
| :persistent_term.get(@key) | ||
|
Comment on lines
+77
to
+81
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
On the first concurrent calls to Useful? React with 👍 / 👎. |
||
|
|
||
| ref -> | ||
| ref | ||
| end | ||
| end | ||
|
|
||
| defp format( | ||
| <<a::binary-size(4), b::binary-size(2), c::binary-size(2), d::binary-size(2), | ||
| e::binary-size(6)>> | ||
| ) do | ||
| [a, b, c, d, e] | ||
| |> Enum.map_join("-", &Base.encode16(&1, case: :lower)) | ||
| end | ||
| end | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The coordinator trains MoE, LSM, and the sequence model from the entire
findingmap, including fields such asfix_type, confidence, and any other features, but this receipt hashes only pattern, repo, outcome, and verification. Two updates with different model inputs can therefore have the samerow_sha256, and changes to the fields that actually influence training are invisible to the receipt, so it does not name the exact evidence on which the neural update rested.Useful? React with 👍 / 👎.