Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ PatternRegistry ──> TriangleRouter ──> FleetDispatcher ──> DispatchM

[cols="1,3"]
|===
| `lib/` | 170 Elixir source files — pipeline, rules, VCL, neural, safety, web, merge orchestration
| `lib/` | 172 Elixir source files — pipeline, rules, VCL, neural, safety, web, merge orchestration
| `lib/rules/` | 44 source files: rule modules and their helpers; see link:docs/rules/catalogue.adoc[the catalogue]
| `lib/neural/` | 8 networks + blackboard, coordinator, persistence, rebalancer
| `lib/vcl/` | VCL query layer over verisim-data: parser, file executor, cache, federation
Expand Down
43 changes: 43 additions & 0 deletions lib/fleet_dispatcher.ex
Original file line number Diff line number Diff line change
Expand Up @@ -699,6 +699,48 @@ defmodule Hypatia.FleetDispatcher do
# place for it is this function.
defp trusted_verification(_finding), do: nil

# The receipt of a neural learning update (E0-09, owner ruling D2). The evidence
# is the qualified outcome that was handed to the coordinator; there is no row
# in a log, so the receipt names the digest of that evidence in canonical form.
# It never raises: a receipt that cannot be written is logged as receipt_pending.
# There is no row to rebuild it from, so nothing repairs it later.
defp write_neural_receipt(pattern, repo, outcome, verification) do
data_root = Path.expand(Hypatia.Paths.verisimdb_data())

evidence = %{
"file" => ".",
"outcome" => outcome,
"recipe_id" => pattern,
"repo" => repo,
"verification" => verification
}
Comment on lines +710 to +716

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Hash the complete neural training input

The coordinator trains MoE, LSM, and the sequence model from the entire finding map, including fields such as fix_type, confidence, and any other features, but this receipt hashes only pattern, repo, outcome, and verification. Two updates with different model inputs can therefore have the same row_sha256, and changes to the fields that actually influence training are invisible to the receipt, so it does not name the exact evidence on which the neural update rested.

Useful? React with 👍 / 👎.


event_id = Hypatia.EventId.generate()

with {:ok, bytes} <- Hypatia.Receipt.encode(evidence),
row =
Map.merge(evidence, %{
"event_id" => event_id,
"event_id_source" => "hypatia",
"source_kind" => "neural_path"
}),
ctx = %{
kind: :neural_path,
log: "neural/outcome-queue",
line: "0",
row_sha256: :crypto.hash(:sha256, bytes) |> Base.encode16(case: :lower)
},
:ok <- Hypatia.Receipt.write(data_root, row, ctx: ctx) do
:ok
else
{:error, reason} ->
Logger.warning("receipt_pending event_id=#{event_id} reason=#{inspect(reason)}")
end
Comment on lines +735 to +738

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Retain failed neural receipts for later repair

When neural queueing succeeds but the receipt append fails transiently, this handler records only a log message and discards the event ID and evidence. Unlike outcome-log rows, the neural event has no durable row that Receipt.repair/2 can discover, so no later cycle can ever create the missing receipt and the promised one-receipt-per-learning-update invariant is permanently broken.

Useful? React with 👍 / 👎.

rescue
error ->
Logger.warning("receipt_pending reason=#{Exception.message(error)}")
end

# :queued records a volatile source-bound submission, never completed learning.
@doc false
def record_neural_outcome(finding, outcome, server \\ Hypatia.Neural.Coordinator) do
Expand Down Expand Up @@ -766,6 +808,7 @@ defmodule Hypatia.FleetDispatcher do
"verification" => verification
}) do
record_neural_outcome(finding, outcome_str, server)
write_neural_receipt(to_string(pattern), to_string(repo), outcome_str, verification)
Comment on lines 810 to +811

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Write the receipt only when neural queueing succeeds

When the coordinator is unavailable or returns an error, record_neural_outcome/3 converts that failure to {:error, ...}, but this return value is ignored and the next line still emits a receipt. In that scenario no update was queued, yet the receipt records a credited_success or credited_failure and therefore falsely claims that the event changed the learner; branch on :queued before writing it.

Useful? React with 👍 / 👎.

else
Logger.info(
"Learning loop: outcome for pattern '#{pattern}' in #{repo} not learned: " <>
Expand Down
95 changes: 95 additions & 0 deletions lib/hypatia/event_id.ex
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# SPDX-License-Identifier: MPL-2.0
# Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>

defmodule Hypatia.EventId do
@moduledoc """
The identity of an event: an ADR-008 version 8 identifier under profile T
(the time-ordered profile), written by Hypatia itself (owner ruling R5-A of
2026-11-15, `docs/design/e0-receipt.adoc`). It makes no call to an external
command, so there is no tool that can be missing.

The layout is the one ADR-008 fixes (`metadatastician/standards`,
`docs/decisions/ADR-008-uuid-v8-estate-standard.adoc`):

48 bits unix_ts_ms big-endian Unix epoch milliseconds
4 bits ver 0b1000 (8)
12 bits rand_a a counter that makes ids increase within a millisecond
2 bits var 0b10
62 bits rand_b from a cryptographic random source

The text is lowercase `8-4-4-4-12`. The field that holds an `event_id`
declares profile T. The timestamp is never an audited event time and never a
basis for authorisation (ADR-008).

The counter is shared by the whole node (an `:atomics` word that holds the
millisecond and the counter together), so ids increase within one process and
are distinct across processes. If the clock goes backwards, ids keep
increasing: the counter carries into the millisecond instead.
"""

import Bitwise

@key {__MODULE__, :state}
@text ~r/\A[0-9a-f]{8}-[0-9a-f]{4}-8[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}\z/

@doc "Mint an id: lowercase `8-4-4-4-12` text, version 8, variant `10`, profile T."
@spec generate() :: String.t()
def generate do
packed = next_packed()
ms = packed >>> 12
counter = packed &&& 0xFFF
<<rand_b::62, _::2>> = :crypto.strong_rand_bytes(8)

format(<<ms::48, 8::4, counter::12, 2::2, rand_b::62>>)
end

@doc "The 16 bytes of an id in the text form."
@spec to_binary(String.t()) :: <<_::128>>
def to_binary(text) when is_binary(text) do
text |> String.replace("-", "") |> Base.decode16!(case: :lower)
end

@doc "Is this a valid `event_id`: a string in the text form, version 8, variant `10`?"
@spec valid?(term()) :: boolean()
def valid?(text) when is_binary(text), do: Regex.match?(@text, text)
def valid?(_other), do: false

# The shared word holds `(milliseconds << 12) | counter`. A new millisecond
# resets the counter; the same or an earlier one adds 1, which carries into the
# millisecond when the 12 bits overflow, so the value only ever increases.
defp next_packed do
ref = state()
now = System.system_time(:millisecond) <<< 12
advance(ref, now)
end

defp advance(ref, now) do
current = :atomics.get(ref, 1)
next = if now > current, do: now, else: current + 1

case :atomics.compare_exchange(ref, 1, current, next) do
:ok -> next
_changed -> advance(ref, now)
end
end

defp state do
case :persistent_term.get(@key, nil) do
nil ->
ref = :atomics.new(1, signed: false)
:persistent_term.put(@key, ref)
:persistent_term.get(@key)
Comment on lines +77 to +81

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Initialize the shared event counter atomically

On the first concurrent calls to generate/0, multiple processes can all observe a missing persistent term, create separate atomics references, and overwrite each other's reference. A process that generated its first ID from the displaced reference can then generate its next ID from the replacement counter starting at zero, so IDs within that process are no longer guaranteed to increase as the profile and module documentation require; the one-time initialization needs synchronization rather than an unconditional put.

Useful? React with 👍 / 👎.


ref ->
ref
end
end

defp format(
<<a::binary-size(4), b::binary-size(2), c::binary-size(2), d::binary-size(2),
e::binary-size(6)>>
) do
[a, b, c, d, e]
|> Enum.map_join("-", &Base.encode16(&1, case: :lower))
end
end
Loading
Loading