fix(sync): replicate malformed frontmatter losslessly by default

docs/lossless-opaque-sync-acceptance.md

@@ -0,0 +1,87 @@
+# Default lossless opaque-document sync
+
+## Contract
+
+Readable UTF-8 Markdown is replicated exactly even when its leading YAML is
+malformed, duplicated, null, scalar, or a sequence. Structural diagnostics are
+nonblocking; validation does not repair or normalize documents. No option,
+public TypeScript shape, protocol field, or persisted-state layout is added.
+
+Unreadable files and invalid UTF-8 still fence work. Conflicts and receive-only
+local divergence still require an explicit decision. A readable malformed local
+edit is no longer mistaken for corrupt bytes that a receive-only mirror may
+replace automatically. Existing authority scope/configuration checks remain.
+
+Both Node and embedding adapters must preserve a UTF-8 BOM. Node's decoder now
+retains it; the new real-filesystem regression caught the previous BOM loss.
+The YAML parser's default duplicate-key rejection is retained and tested.
+Structural diagnostics and projection parsing now share Rust-compatible
+leading-fence recognition, including BOM-prefixed mappings/body-only records,
+non-leading fences, and the authority's closing-delimiter/body boundary.
+
+## Consumer audit
+
+- Portable and Node directory mirrors share the inspector. Exact `document`
+  bytes were already separate from the structured projection; opaque records
+  already have a valid representation without changing generic/public types.
+- CLI synchronization executes the engine plan, not `local_issues.length`.
+  Diagnostic `attention` can remain after successful synchronization.
+- Server/reference-authority consumers and every Connect TypeScript workspace
+  typecheck with this change. Application SDK, editor, devkit, testing and other
+  non-desktop workspace tests pass.
+- Obsidian's apply button already uses the plan's blocking count, but its
+  diagnostic prose incorrectly described every warning as a pause. A separate
+  consumer branch updates that prose, tests nonblocking reviews, and preserves
+  BOMs in its Vault adapter. Its release pin must not move to an unpublished
+  npm version. Immutable artifacts from commit `87e28391233c` pass all 96
+  Obsidian unit tests, TypeScript/build and unchanged mobile budgets, plus ten
+  strict exact round trips through the Obsidian adapter and reference authority.
+  The same strict check fails against released beta.91 with seven YAML blockers.
+  Its released dependencies and generated bundle were restored afterward.
+- `tasknotes-app` only pins `connect-sync` as an immutable SDK override; no direct
+  directory-mirror or `local_issues` consumer was found in its application code.
+- Desktop renderer status is owned by the Rust mirror, not this JavaScript SDK
+  inspector. This change does not claim to update that separate implementation.
+
+There is no persisted local-issue gate to migrate. Existing plan-only checkpoints
+are reused; unsupported older engine layouts remain rejected as before.
+
+## Automated evidence
+
+Node 24.19.0 / pnpm 11.15.1:
+
+- `pnpm typecheck`: all workspaces pass. The guarded consumer-artifact packaging
+  command also completed a full `pnpm -r build`.
+- `pnpm --filter @mdbase-dev/connect-sync test`: 196 tests pass.
+- `pnpm --filter '!@mdbase/connect-desktop' -r test`: passes, including server
+  (582 passed, 35 intentionally skipped) and editor (473 passed).
+- `pnpm check:mirror:mobile`: passes unchanged budgets, 187,664 raw / 55,970
+  gzip bytes; no Node-only references in the portable bundle.
+
+New/updated regressions cover exact uploads and second-mirror downloads,
+BOM/CRLF/no-final-newline preservation, updates/moves/deletes, both conflict
+choices, lost-reply replay after restart, existing checkpoint reuse, stale
+reviews, receive-only divergence, invalid UTF-8 and failed rereads.
+
+The canonical Rust runtime's five `frontmatter::parser` tests and
+`schema_invalid_record_remains_semantically_projectable` test also pass with
+`cargo test --locked --lib`; these are source/runtime evidence, not a hosted
+round trip or full Connect Rust-workspace qualification.
+
+Authorities in these SDK tests are `MemoryAuthority`, not deployed hosted
+infrastructure. The Node round-trip uses real temporary filesystem directories;
+the portable fixtures use in-memory adapters.
+
+## Remaining qualification
+
+- Full `pnpm test` reached the desktop tests, where 124 passed and one failed
+  while importing Electron: its binary installation could not create an
+  already-existing `dist/locales` directory. Non-desktop suites were run
+  separately rather than represented as a full-workspace pass.
+- `cargo fmt --all --check` and `cargo test --locked --workspace` cannot resolve
+  the required sibling `../mdbase-rs` in this isolated workspace. Rust/system
+  end-to-end qualification remains outstanding; no Rust sources changed.
+- This is not a published SDK or deployed-hosted acceptance result. The prior
+  LAB loopback-port ownership failure and disposable-Obsidian-vault prerequisite
+  have not been bypassed. A real Obsidian → hosted → second-mirror round trip
+  remains required before claiming live acceptance.

docs/release-contract.md

@@ -55,9 +55,9 @@ components.
 `.github/previous-release.env` identifies the immediate published predecessor.
 Update its annotated tag, full commit and immutable server/provider digests as
 part of release preparation; ordinary upgrade qualification still requires it to
-be the unique newest non-draft GitHub release. Following beta100 publication,
-qualification uses beta100 (`6bd9420690295d3c5f65e68edcbf8e3e55612614`) and its
-exact signed server/provider images. Advancing this fixture preserves the
+be the unique newest non-draft GitHub release. Following beta102 publication,
+qualification uses beta102 (`ea65ae4f13d0e0621fe9654618e13b505cdebf21`) and its
+exact signed server/provider images from verified publication run `35059159880`. Advancing this fixture preserves the
 unique-newest-release check; the retained beta95 and beta94 lanes do not move. This fixture refresh does not itself publish or deploy a release.
 
 Historical regressions and candidate qualification are separate. The beta94

packages/sync/README.md

@@ -12,7 +12,22 @@ unclosed `---` fence are preserved byte-for-byte as body content. An explicitly
 empty frontmatter block is also accepted. Malformed, scalar, null, or list
 frontmatter is synchronized byte-for-byte as opaque Markdown with `{}`
 persisted fields; structured queries and field operations ignore its invalid
-frontmatter until the document is repaired.
+frontmatter until the document is repaired. Leading-fence and BOM recognition
+matches the Rust authority: a body-only record's structured body projection
+omits one leading BOM, but the exact `document` always retains it. Opaque
+fallback bodies retain the whole document. This is the default for directory
+mirrors, not an opt-in mode. Record schema validation is separate from exact
+replication: missing required fields, wrong field types, and enum violations
+do not by themselves prevent synchronization. Authority scope, configuration,
+conflict, and safety checks still apply.
+
+Local frontmatter diagnostics remain visible in `local_issues` and can make
+status `attention` even after a successful sync. Consumers must use the reviewed
+plan's `issues[].blocking` / `summary.blocking_issues` to decide whether applying
+that plan is allowed, not the diagnostic code or `local_issues.length`.
+Unreadable files and invalid UTF-8 still fence writes and checkpoint advancement.
+Receive-only mirrors also fence local divergence, including readable malformed
+edits; malformed YAML is not permission to overwrite those edits from authority.
 
 Application replicas resolve a stale record explicitly with
 `resolveConflict(recordId, "local" | "remote")`. Keeping the local version
@@ -92,7 +107,9 @@ await mirror.sync();
 ```
 
 The filesystem adapter deals only in collection-relative POSIX paths and
-ordinary UTF-8 strings. The state store must live in device-local application
+ordinary UTF-8 strings. Text reads must preserve every byte representable by
+UTF-8, including a leading BOM (use `TextDecoder` with `fatal: true` and
+`ignoreBOM: true`, which retains the BOM character). The state store must live in device-local application
 state, not inside the mirrored collection. A host should provide a lease that
 excludes concurrent mirror owners for the same vault; the default memory lease
 only protects overlapping calls in one JavaScript process.

packages/sync/src/mirror-format.ts

@@ -18,14 +18,15 @@ export type LocalRecordStructuralOutcome = {
   outcome: "parsed" | "invalid_yaml" | "non_mapping_frontmatter";
 };
 
-/** Strict structural check for local mirror records; authority parsing remains opaque-compatible. */
+/** Structural diagnostics only. Readable documents synchronize exactly, including opaque frontmatter. */
 export function classifyLocalRecord(document: string): LocalRecordStructuralOutcome {
-  const yaml = leadingFrontmatterYaml(document);
-  if (yaml === null) return { outcome: "parsed" };
+  const block = leadingFrontmatter(document);
+  if (block === null) return { outcome: "parsed" };
+  const { yaml } = block;
 
   let frontmatter: unknown;
   try {
-    frontmatter = parse(yaml, { mapAsMap: true, uniqueKeys: true });
+    frontmatter = parse(yaml, { mapAsMap: true });
   } catch {
     return { outcome: "invalid_yaml" };
   }
@@ -37,7 +38,7 @@ export function classifyLocalRecord(document: string): LocalRecordStructuralOutc
 }
 
 /** Mirrors mdbase-rs parse_document_without_bom delimiter recognition. */
-function leadingFrontmatterYaml(document: string): string | null {
+function leadingFrontmatter(document: string): { yaml: string; body: string } | null {
   const content = document.startsWith("\uFEFF") ? document.slice(1) : document;
   const openingEnd = content.indexOf("\n");
   if (openingEnd < 0 || content.slice(0, openingEnd).trimEnd() !== "---") return null;
@@ -46,7 +47,10 @@ function leadingFrontmatterYaml(document: string): string | null {
     let lineEnd = content.indexOf("\n", lineStart);
     if (lineEnd < 0) lineEnd = content.length;
     if (content.slice(lineStart, lineEnd).trimEnd() === "---") {
-      return content.slice(yamlStart, lineStart);
+      return {
+        yaml: content.slice(yamlStart, lineStart),
+        body: content.slice(lineStart + 3).replace(/^\r?\n/, "")
+      };
     }
     lineStart = lineEnd + 1;
   }
@@ -148,24 +152,24 @@ export function fastRecordDocumentMatches(document: string, record: SyncRecord):
 }
 
 export function parseMarkdown(document: string, _path: string): { frontmatter: JsonObject; body: string } {
-  const match = document.match(/^---[ \t]*\r?\n([\s\S]*?)^---[ \t]*(?:\r?\n|$)([\s\S]*)$/m);
-  if (!match) {
-    return { frontmatter: {}, body: document };
+  const block = leadingFrontmatter(document);
+  if (block === null) {
+    return { frontmatter: {}, body: document.startsWith("\uFEFF") ? document.slice(1) : document };
   }
   let frontmatter: unknown;
   try {
-    frontmatter = parse(match[1]!, { mapAsMap: true });
+    frontmatter = parse(block.yaml, { mapAsMap: true });
   } catch {
     return { frontmatter: {}, body: document };
   }
-  if (frontmatter === null && match[1]!.trim() === "") {
-    return { frontmatter: {}, body: match[2] ?? "" };
+  if (frontmatter === null && block.yaml.trim() === "") {
+    return { frontmatter: {}, body: block.body };
   }
   const projection = jsonProjection(frontmatter, new Set());
   if (projection === INVALID_JSON_PROJECTION || !isJsonObject(projection)) {
     return { frontmatter: {}, body: document };
   }
-  return { frontmatter: projection, body: match[2] ?? "" };
+  return { frontmatter: projection, body: block.body };
 }
 
 function jsonProjection(

packages/sync/src/mirror-state.ts

@@ -31,6 +31,7 @@ export interface MirrorFileEntry {
   file: CollectionFileDescriptor;
 }
 
+/** Diagnostics, not an apply gate. Use the inspected plan's blocking issues. */
 export interface MirrorLocalIssue {
   path: string;
   code: "invalid_frontmatter" | "file_read_failed";
@@ -148,7 +149,7 @@ export interface MirrorFileSystem {
   /** True when any filesystem entry occupies this exact portable path. */
   exists(path: string): Promise<boolean>;
   read(path: string): Promise<string | null>;
-  /** Classified byte-aware read; expected I/O failures reject as `SyncError("file_read_failed")`. */
+  /** Exact UTF-8 read, preserving a leading BOM; expected I/O failures reject as `SyncError("file_read_failed")`. */
   readText(path: string): Promise<MirrorTextReadResult>;
   write(path: string, value: string): Promise<void>;
   /** Atomically rename one managed path without changing its bytes. */