ariadne: app-wide HTTP(s) cache in the state tree (§13.9, PR2)

docs/roadmap/CVCGL-UI-DSL-ROADMAP.md

@@ -3290,6 +3290,29 @@ instead of a full transfer.
   demand) the distributed state system already sketches — the cache is a first, local consumer of
   that path.
 
+> **LANDED (PR2 of the §13.9 work) — the state-tree cache.** `ariadne/uri_http_cache.cpp` +
+> `register_cached_http_uri_handler(cvc::app&)` (inc/cvc/ariadne/uri_http_cache.h). Entries at
+> `sys.net.http_cache.entries.<key>` (`key = cvc::sha256_hex(normalized-URL)`): body on the entry
+> node's `data()` (raw `std::string` → shows as `bytes` in state_exec, serves via `state://…?data`),
+> metadata (status/etag/last_modified/content_type/effective_url/fetched_at/freshness_ttl) as child
+> value nodes. Freshness (Cache-Control `max-age`, anchored at `now - Age`; else `0`-with-validator
+> heuristic; `no-cache`→revalidate; `no-store`→bypass) serves with no network; a stale/miss does a
+> conditional GET (`If-None-Match`/`If-Modified-Since`) — 304 refreshes + serves cached, 200 replaces.
+> Retention rides node `expireAt` (>= freshness) + `sweepExpired()` on access; a successful store
+> invalidates. Concurrency: one process-wide `cache_mutex` serializes all `entries`-subtree work
+> (sweep+read+write+invalidate) — held only briefly, released across the network send — because
+> `findDescendant`/`operator()` hand back a bare pointer a concurrent `sweepExpired()` can free;
+> single-flight coalesces concurrent same-URL fetches. A credentialed request (provider headers or
+> URL userinfo) BYPASSES the cache (no cross-principal reuse in a process-global cache). Reviewed
+> adversarially (15 findings fixed). Tests: `uri_http_cache_test` (12, offline via the PR1
+> `set_http_client` fake). **Deferred to PR3:** the `sys.net.http_cache.policy.*` knobs + a
+> max-entries/max-bytes **LRU budget** (today bounded only by per-entry 64 MiB + retention, so a flood
+> of distinct fresh URLs grows unbounded), full `Vary`/`private`, `Expires`-based freshness, and
+> richer URL normalization (percent-encoding, query-key order). **The transport stays synchronous
+> (§13.8's `fetch_sync`); an async/awaitable `(http-get)` intrinsic — chunked, off-thread on the
+> compute/I/O pool, parking the state_exec process via the `compute_async → post_message → msg-recv`
+> path and returning `bytes` — is a natural later PR the cache rides unchanged.**
+
 ### 13.10 The WRITE side — a `store`/PUT capability + URI-aware `state::save`/`restore`
 
 > **Status — core LANDED** (`cvc::ariadne`): `store(uri, content, base)` +

inc/cvc/ariadne/uri_http_cache.h

@@ -0,0 +1,42 @@
+#ifndef CVC_ARIADNE_URI_HTTP_CACHE_H
+#define CVC_ARIADNE_URI_HTTP_CACHE_H
+
+// Ariadne — an app-wide caching http(s):// URI handler (roadmap §13.9). It replaces the plain
+// http(s) READ handler with one backed by the cvc::app's state tree: repeated import:/load:/source:
+// of the same URL is served from a cache under `sys.net.http_cache.entries.<key>` instead of
+// re-fetching. Freshness (Cache-Control max-age / Expires) serves with NO network; a stale entry
+// does a conditional GET (If-None-Match / If-Modified-Since) — a 304 revalidates cheaply, a 200
+// replaces. Retention (>= freshness) rides the state tree's own node-expiry (expireAt +
+// sweepExpired, swept on access), and concurrent requests for the same URL coalesce
+// (single-flight).
+//
+// It builds on the plain handler's transport (cvc::net) and policy (uri_http.cpp): the auth-header
+// provider, the response cap, and the redirect-safety + >=400 mapping are shared. A CREDENTIALED
+// request (the host provider returns headers) BYPASSES the cache entirely — a process-global cache
+// must not serve one principal's authorized body to another; full Vary/`private` handling is later
+// work. The WRITE side stays the plain PUT/POST store, but a successful store INVALIDATES the
+// cached entry for that URL.
+//
+// Like state://, the cached handler needs the app for its state tree, so this is the app-bound
+// variant of register_http_uri_handler(): a host calls it during setup (before load_*). It holds
+// the app's root state BY POINTER — call unregister_cached_http_uri_handler() before the app is
+// destroyed. A no-op (leaves the scheme unregistered) when no cvc::net backend is compiled.
+
+namespace cvc {
+class app;
+namespace ariadne {
+
+// Register the caching http/https READ handler and an invalidating PUT/POST WRITE handler against
+// `app`'s root state (cvc::state::instance(app)). Replaces any existing http/https handlers. No-op
+// if no HTTP backend is compiled (cvc::net::have_http_backend() == false).
+void register_cached_http_uri_handler(cvc::app &app);
+
+// Tear down the cached http/https read + write handlers (leaves the schemes unregistered; a host
+// that wants the plain handler back calls register_http_uri_handler()). Call before the app/root is
+// destroyed. The cached data under sys.net.http_cache stays in the state tree.
+void unregister_cached_http_uri_handler();
+
+} // namespace ariadne
+} // namespace cvc
+
+#endif // CVC_ARIADNE_URI_HTTP_CACHE_H

src/cvc/CMakeLists.txt

@@ -1118,7 +1118,7 @@ list(APPEND INCLUDE_FILES ../../inc/cvc/net/http_client.h)
 # the retained Widget tree, the Runtime + reconcile boundary, and direct
 # cvc::state binding, walking a pluggable Backend. It touches NO VTK/ImGui/GL —
 # the ImGui-over-VTK backend lives in cvcGL (src/cvcGL/ariadne). Roadmap §16.1.
-list(APPEND SOURCE_FILES ariadne/ariadne.cpp ariadne/loader.cpp ariadne/uri.cpp ariadne/uri_state.cpp ariadne/uri_http.cpp ariadne/state_io.cpp ariadne/ftxui_backend.cpp)
+list(APPEND SOURCE_FILES ariadne/ariadne.cpp ariadne/loader.cpp ariadne/uri.cpp ariadne/uri_state.cpp ariadne/uri_http.cpp ariadne/uri_http_cache.cpp ariadne/state_io.cpp ariadne/ftxui_backend.cpp)
 list(APPEND INCLUDE_FILES
   ../../inc/cvc/ariadne/widget.h
   ../../inc/cvc/ariadne/backend.h
@@ -1128,6 +1128,7 @@ list(APPEND INCLUDE_FILES
   ../../inc/cvc/ariadne/uri.h
   ../../inc/cvc/ariadne/uri_state.h
   ../../inc/cvc/ariadne/uri_http.h
+  ../../inc/cvc/ariadne/uri_http_cache.h
   ../../inc/cvc/ariadne/state_io.h
   ../../inc/cvc/ariadne/scene.h
   ../../inc/cvc/ariadne/value.h

src/cvc/ariadne/uri_http.cpp

@@ -2,15 +2,14 @@
 //
 // A THIN ADAPTER over cvc::net (inc/cvc/net/http_client.h). The transport — libcurl on native,
 // Emscripten fetch on wasm — lives behind cvc::net, so this handler works identically on both
-// targets. What stays HERE is the Ariadne-specific policy the transport must not bake in:
-//   - the host auth-header/method provider (credentials come from the host, never the .ari doc),
-//   - the response-size cap,
-//   - the redirect-safety rules (a credentialed read does not follow; a write never follows),
-//   - the store Content-Type default,
-//   - mapping a cvc::net::HttpResponse (which reports a >= 400 as ok=true + status) to a
-//     UriResult/StoreResult, re-imposing ">= 400 is an error" to preserve resolve()/store() output.
-// It is opt-in: a host must call register_http_uri_handler(), which registers only when a real net
-// backend is compiled — otherwise the scheme is left unresolved, exactly as before.
+// targets. The Ariadne-specific policy the transport must not bake in — the host auth-header/method
+// provider, the response-size cap, the redirect-safety rules, the store Content-Type default, and
+// the HttpResponse -> UriResult/StoreResult mapping — lives in the `detail` helpers here
+// (uri_http_detail.h) so the §13.9 caching handler (uri_http_cache.cpp) reuses exactly the same
+// policy. The handler is opt-in: a host must call register_http_uri_handler(), which registers only
+// when a real net backend is compiled — otherwise the scheme is left unresolved, as before.
+
+#include "uri_http_detail.h"
 
 #include <cctype>
 #include <cstddef>
@@ -23,12 +22,8 @@
 
 namespace cvc {
 namespace ariadne {
-namespace {
 
-// A remote fragment/library is DSL text or a modest asset; a response past this is refused rather
-// than buffered whole (an availability guard, mirroring the file handler's cap). Also bounds the
-// (discarded) response body of a store.
-constexpr std::size_t kMaxHttpBytes = 64u * 1024u * 1024u; // 64 MiB
+namespace {
 
 // A store body of unknown media type gets a neutral Content-Type so a strict server does not
 // form-parse or 415 it (a host that knows better passes its own via the provider).
@@ -44,15 +39,23 @@ HttpOptionsProvider &provider_cell() {
   static HttpOptionsProvider p;
   return p;
 }
-HttpOptionsProvider current_provider() {
-  std::lock_guard<std::mutex> lock(provider_mutex());
-  return provider_cell();
+
+} // namespace
+
+namespace detail {
+
+HttpRequestOptions consult_http_provider(const std::string &url, bool for_write) {
+  HttpOptionsProvider p;
+  {
+    std::lock_guard<std::mutex> lock(provider_mutex());
+    p = provider_cell();
+  }
+  if (p)
+    return p(url, for_write);
+  return HttpRequestOptions{};
 }
 
-// True if `lines` already carries a header whose name (the text before the first ':') equals `name`
-// case-insensitively. Also matches libcurl's suppression form (`Name:` with an empty value), so an
-// explicit suppression by the provider is respected rather than overridden by a default.
-bool has_header(const std::vector<std::string> &lines, const std::string &name) {
+bool http_header_present(const std::vector<std::string> &lines, const std::string &name) {
   for (const std::string &line : lines) {
     if (line.find(':') != name.size())
       continue; // name-length must match exactly (":" right after the name)
@@ -69,54 +72,37 @@ bool has_header(const std::vector<std::string> &lines, const std::string &name)
   return false;
 }
 
-UriResult http_fetch(const Uri &u, const std::string & /*base*/) {
-  // Consult the provider FIRST. No provider -> default GET, no extra headers. A throwing provider
-  // (host code) unwinds to resolve()'s barrier.
-  HttpRequestOptions opts;
-  if (const HttpOptionsProvider p = current_provider())
-    opts = p(u.raw, /*for_write=*/false);
-
-  cvc::net::HttpRequest req;
-  req.url = u.raw;
-  req.max_bytes = kMaxHttpBytes;
-  req.headers = opts.headers;
-  // libcurl re-sends custom headers (X-Api-Key, …) VERBATIM on every redirect hop and only strips
-  // Authorization/Cookie/Proxy-Authorization on a cross-origin redirect. So follow a redirect only
-  // when NO provider headers are attached (nothing to leak across an origin the untrusted .ari
-  // document chose); a credentialed read stops at the redirect instead of carrying the token
-  // onward.
-  const bool follow = opts.headers.empty();
-  req.follow_redirects = follow;
-  // A method override on a read is a bodyless custom verb (default stays GET).
-  if (!opts.method.empty() && opts.method != "GET")
-    req.method = opts.method;
-
-  const cvc::net::HttpResponse r = cvc::net::send(req);
+UriResult map_http_fetch_result(const std::string &raw_url, bool follow,
+                                const cvc::net::HttpResponse &r) {
   if (!r.ok)
     return {false, std::string(), std::string(),
-            "ari: http fetch of '" + u.raw + "' failed: " + r.error};
+            "ari: http fetch of '" + raw_url + "' failed: " + r.error};
+  // A 304 reaching the plain mapper is unexpected (only the cache sends conditional requests, and
+  // it serves the cached body on its own path); a bodyless 304 must never surface as an empty-body
+  // OK.
+  if (r.status == 304)
+    return {false, std::string(), std::string(),
+            "ari: http fetch of '" + raw_url +
+                "' returned 304 Not Modified with no cached entry to revalidate"};
   // The facade does not fail a >= 400 (so a cache can see a 304); the resolver treats it as an
   // error.
   if (r.status >= 400)
     return {false, std::string(), std::string(),
-            "ari: http fetch of '" + u.raw + "' failed: HTTP status " + std::to_string(r.status)};
+            "ari: http fetch of '" + raw_url + "' failed: HTTP status " + std::to_string(r.status)};
   // Not following (credentialed): a 3xx would otherwise be returned as OK with the redirect PAGE as
   // the body. Surface it instead of silently handing back the wrong bytes.
   if (!follow && r.status >= 300 && r.status < 400)
     return {false, std::string(), std::string(),
-            "ari: http fetch of '" + u.raw + "' returned redirect status " +
+            "ari: http fetch of '" + raw_url + "' returned redirect status " +
                 std::to_string(r.status) + " to '" + r.canonical_url +
                 "'; a credentialed read does not follow redirects (the auth header would cross an "
                 "origin). Resolve the target directly."};
   return {true, r.body, r.canonical_url, std::string()};
 }
 
-// §13.10 the write analogue: PUT (default) or POST/… the `content` bytes to the URL. Method + auth
-// headers come from the same provider as the reader (for_write = true).
-StoreResult http_store(const Uri &u, const std::string &content, const std::string & /*base*/) {
-  HttpRequestOptions opts;
-  if (const HttpOptionsProvider p = current_provider())
-    opts = p(u.raw, /*for_write=*/true);
+StoreResult http_store_uncached(const Uri &u, const std::string &content,
+                                const std::string & /*base*/) {
+  const HttpRequestOptions opts = consult_http_provider(u.raw, /*for_write=*/true);
   const std::string method = opts.method.empty() ? std::string("PUT") : opts.method;
 
   cvc::net::HttpRequest req;
@@ -128,10 +114,9 @@ StoreResult http_store(const Uri &u, const std::string &content, const std::stri
   // to a host the untrusted .ari document chose. A 3xx on a store is an error the host must
   // resolve.
   req.follow_redirects = false;
-  // Default a neutral Content-Type when the provider supplied none (its own wins) — otherwise a
-  // server might form-parse opaque store bytes.
+  // Default a neutral Content-Type when the provider supplied none (its own wins).
   std::vector<std::string> header_lines = opts.headers;
-  if (!has_header(header_lines, "Content-Type"))
+  if (!http_header_present(header_lines, "Content-Type"))
     header_lines.push_back(kDefaultStoreContentType);
   req.headers = std::move(header_lines);
 
@@ -151,6 +136,31 @@ StoreResult http_store(const Uri &u, const std::string &content, const std::stri
   return {true, r.canonical_url, std::string()};
 }
 
+} // namespace detail
+
+namespace {
+
+UriResult http_fetch(const Uri &u, const std::string & /*base*/) {
+  const HttpRequestOptions opts = detail::consult_http_provider(u.raw, /*for_write=*/false);
+  cvc::net::HttpRequest req;
+  req.url = u.raw;
+  req.max_bytes = detail::kMaxHttpBytes;
+  req.headers = opts.headers;
+  // Follow a redirect only when uncredentialed (nothing to leak across an origin the untrusted .ari
+  // document chose); a credentialed read stops at the redirect instead of carrying the token
+  // onward.
+  const bool follow = opts.headers.empty();
+  req.follow_redirects = follow;
+  // A method override on a read is a bodyless custom verb (default stays GET).
+  if (!opts.method.empty() && opts.method != "GET")
+    req.method = opts.method;
+  return detail::map_http_fetch_result(u.raw, follow, cvc::net::send(req));
+}
+
+StoreResult http_store(const Uri &u, const std::string &content, const std::string &base) {
+  return detail::http_store_uncached(u, content, base);
+}
+
 } // namespace
 
 void set_http_options_provider(HttpOptionsProvider provider) {