fix(sync): preserve BOMs and distinguish diagnostics from blockers

PRODUCT.md

@@ -40,6 +40,7 @@ The local vault remains useful while offline. Synchronization is explicit, obser
 - Enroll using a Connect enrollment code.
 - Store credentials in Obsidian SecretStorage, not plugin data or collection files.
 - Preview and run mirror synchronization.
+- Keep record validation separate from exact-document replication; never silently repair records during sync.
 - Show progress, last successful checkpoint, failures, and conflicts in plain language.
 - Refuse unsafe role changes or resource overwrites.
 

README.md

@@ -178,7 +178,9 @@ disposable vault with the exact installed bundle.
 The Connect protocol and sync SDKs are pinned to `0.1.0-beta.91`, and mdbase
 interop is pinned to `0.1.0-rc.2`. Update `package.json`, regenerate
 `package-lock.json`, and rerun the binary round-trip and mobile gates when
-advancing them.
+advancing them. The pending lossless malformed-frontmatter upgrade is documented
+in [SDK consumer qualification](docs/lossless-sdk-acceptance.md); its strict
+candidate check is separate from the released beta.91 test suite.
 
 ## Compatibility
 

docs/lossless-sdk-acceptance.md

@@ -0,0 +1,80 @@
+# Lossless SDK consumer qualification
+
+## Shipping versus candidate behavior
+
+The tracked dependency pins remain `connect-sync` / `connect-protocol`
+`0.1.0-beta.91`. That released mirror engine still blocks malformed frontmatter.
+This branch prepares consumer messaging and fixes BOM preservation; it does not
+pretend that an unpublished SDK fix is already shipped in the plugin.
+
+Connect commit `87e28391233c` makes readable malformed YAML a nonblocking
+structural diagnostic by default. No option, protocol change, or public
+TypeScript shape change is required. Its immutable candidate artifacts were
+produced with Connect's guarded `package:consumer` script, not published to npm.
+Advance the released pins and lockfile after the SDK release, rerun the strict
+candidate test, and include that check in normal qualification at that point.
+Do not commit machine-local tarball paths or a nonexistent registry version.
+
+## Consumer changes
+
+- Obsidian's binary-to-text adapter preserves UTF-8 BOMs, CRLF and final-newline
+  choices. Invalid UTF-8 still fails safely.
+- The reviewed plan remains the sole apply gate. Diagnostic count and issue code
+  alone do not decide whether sync is paused.
+- Nonblocking diagnostics say `review`, not mandatory `fix`; warning prose no
+  longer promises that all synchronization is paused.
+- Validation remains separate. A regression runs actual plugin schema validation
+  (missing required title, wrong priority type, invalid enum), transfers the
+  unchanged record through `MemoryAuthority` into a second mocked Vault, and
+  verifies that the validation errors are still present afterward.
+
+## Automated results
+
+Node 24.19.0:
+
+| Installed SDK | Unit tests | Build / mobile |
+| --- | --- | --- |
+| Released beta.91 | 96 passed | 687,949 raw / 195,842 gzip bytes |
+| Candidate from `87e28391233c` | 96 passed | 689,242 raw / 196,214 gzip bytes |
+
+Both build sizes satisfy the unchanged mobile budgets. Lint passes with the
+released dependencies. No consumer TypeScript adjustment was needed for the
+candidate. Both package manifests and the lockfile keep their released pins.
+
+`npm run test:sdk-candidate` additionally passes ten exact round trips through
+the real `ObsidianMirrorFileSystem` and portable sync engine, including malformed
+YAML, duplicate keys, scalar/null/list frontmatter, BOM/CRLF, missing final
+newline, BOM-prefixed valid mappings and body-only notes, and a valid sibling. It checks that the preview enables the transfers
+and the receive-only mirror settles without attempting repeated repairs.
+
+The released beta.91 control run fails this stricter command at the expected
+assertion: seven blocking issues instead of zero. The candidate passes without
+changing the test or adding a runtime option. To qualify an immutable
+candidate without changing tracked dependency pins:
+
+```sh
+npm ci --ignore-scripts
+npm install --no-save --package-lock=false --ignore-scripts \
+  /path/to/immutable-protocol.tgz /path/to/immutable-sync.tgz
+npm test
+npm run test:sdk-candidate
+npm run check:mobile
+# Restore released dependencies and the release-pinned generated bundle:
+npm ci --ignore-scripts
+npm run check:mobile
+```
+
+## Boundaries not qualified here
+
+The Vault APIs and authority in these tests are in-memory fixtures. Schema
+validation is real plugin code, but the lightweight Obsidian YAML mock uses
+JSON-compatible YAML. These results are not live Obsidian or deployed-hosted
+acceptance, and do not verify production or staging.
+
+The prior LAB port-ownership failure and requirement for a clean disposable
+Obsidian vault remain outstanding. No guards were bypassed, no existing vault
+metadata was removed, and no LAB fixture, installed plugin, production, or
+staging service was changed in this work.
+
+The recovery workspace remains a separate draft PR; this branch does not merge
+or replace it.

package.json

@@ -19,6 +19,7 @@
     "build:test:staging": "npm run build:staging && node copy-files.mjs",
     "profile:testvault": "node scripts/profile-testvault.mjs",
     "test": "npm run test:unit",
+    "test:sdk-candidate": "tsc -p tsconfig.test.json && node --loader ./test/obsidian-loader.mjs scripts/test-lossless-sync.mjs",
     "test:unit": "tsc -p tsconfig.test.json && node --experimental-specifier-resolution=node --loader ./test/obsidian-loader.mjs --test .test-dist/test/**/*.test.js",
     "copy-files": "node copy-files.mjs",
     "version": "node version-bump.mjs && git add manifest.json versions.json"

scripts/test-lossless-sync.mjs

@@ -0,0 +1,80 @@
+// Candidate qualification, not live Obsidian or hosted acceptance.
+// Run with the Obsidian test loader after installing immutable candidate SDK
+// artifacts without changing package.json/package-lock.json. The released
+// beta.91 SDK is expected to fail this stricter contract until its pin advances.
+import assert from "node:assert/strict";
+import { TFile } from "obsidian";
+import { MemoryAuthority } from "@mdbase-dev/connect-sync";
+import { DirectoryMirror, MemoryMirrorStateStore, WritableDirectoryMirror } from "@mdbase-dev/connect-sync/mirror";
+import { ObsidianMirrorFileSystem } from "../.test-dist/src/connectSync.js";
+import { previewFromPlan } from "../.test-dist/src/syncPreview.js";
+import { syncReviewPresentation } from "../.test-dist/src/syncUx.js";
+
+// Minimal flat-file Vault fixture. All documents and authority state are in
+// memory; this harness cannot write into a user's vault or contact a daemon.
+function vaultFixture() {
+  const files = new Map();
+  return {
+    files,
+    getAbstractFileByPath: (path) => files.get(path)?.file ?? null,
+    getMarkdownFiles: () => [...files.values()].map(({ file }) => file),
+    adapter: {
+      exists: async (path) => files.has(path),
+      readBinary: async (path) => {
+        assert.ok(files.has(path), `Missing fixture: ${path}`);
+        return new TextEncoder().encode(files.get(path).document).buffer;
+      },
+    },
+    create: async (path, document) => {
+      assert.equal(files.has(path), false);
+      const file = new TFile(path);
+      files.set(path, { file, document });
+      return file;
+    },
+    modify: async (file, document) => {
+      assert.ok(files.has(file.path));
+      files.set(file.path, { file, document });
+    },
+  };
+}
+
+const documents = [
+  "---\nbroken: [\n---\nBody",
+  "---\na: 1\na: 2\n---\nBody",
+  "---\nhello\n---\nBody",
+  "---\nnull\n---\nBody",
+  "---\n- one\n- two\n---\nBody",
+  "\uFEFF---\r\nbroken: [\r\n---\r\nExact — bytes\r\n",
+  "---\nbroken: [\n---",
+  "\uFEFF---\r\ntitle: Present\r\n---\r\nExact structured bytes",
+  "\uFEFF# Body-only bytes\r\n",
+  "# Valid sibling\n",
+];
+const hosted = new MemoryAuthority();
+const writerId = hosted.registerReplica({ name: "Candidate writer", mode: "read_write" });
+const readerId = hosted.registerReplica({ name: "Candidate reader", mode: "read_only" });
+const source = vaultFixture();
+const destination = vaultFixture();
+for (const [index, document] of documents.entries()) await source.create(`${index}.md`, document);
+const writer = new WritableDirectoryMirror(writerId, hosted.transport(writerId), {
+  fileSystem: new ObsidianMirrorFileSystem(source), stateStore: new MemoryMirrorStateStore(),
+});
+const reader = new DirectoryMirror(readerId, hosted.transport(readerId), {
+  fileSystem: new ObsidianMirrorFileSystem(destination), stateStore: new MemoryMirrorStateStore(),
+});
+const plan = await writer.inspect();
+assert.equal(plan.summary.blocking_issues, 0, "Candidate SDK must not block readable malformed YAML");
+assert.equal(plan.summary.uploads, documents.length);
+const preview = previewFromPlan(plan);
+const presentation = syncReviewPresentation(plan, preview.entries.length);
+assert.equal(presentation.actionDisabled, false);
+assert.match(presentation.message, /warnings do not block/);
+assert.equal((await writer.apply(plan)).status, "applied");
+await reader.sync();
+for (const [index, document] of documents.entries()) {
+  assert.equal(source.files.get(`${index}.md`)?.document, document);
+  assert.equal(destination.files.get(`${index}.md`)?.document, document);
+}
+assert.equal((await reader.inspect()).summary.blocking_issues, 0);
+assert.deepEqual((await reader.inspect()).actions, []);
+console.log(JSON.stringify({ authority: "in_process_reference", vault: "mock", exact_round_trips: documents.length }));

src/connectSync.ts

@@ -906,7 +906,8 @@ export class ObsidianMirrorFileSystem implements MirrorFileSystem {
       throw new SyncError("file_read_failed", `Could not read ${path}.`);
     }
     try {
-      return new TextDecoder("utf-8", { fatal: true }).decode(bytes);
+      // Preserve the BOM as a character instead of consuming its bytes.
+      return new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }).decode(bytes);
     } catch {
       return {
         kind: "invalid",

src/syncPreview.ts

@@ -6,7 +6,7 @@ import type {
 } from "@mdbase-dev/connect-sync/mirror";
 
 export type SyncPreviewDirection = "download" | "upload" | "attention";
-export type SyncPreviewAction = "create" | "update" | "rename" | "delete" | "replace" | "fix";
+export type SyncPreviewAction = "create" | "update" | "rename" | "delete" | "replace" | "fix" | "review";
 
 export interface SyncPreviewEntry {
   kind: "document" | "file";
@@ -35,7 +35,7 @@ export function previewFromPlan(plan: MirrorSyncPlan): MdbaseSyncPreview {
       kind: "document",
       path: issue.path ?? "Sync engine",
       direction: "attention",
-      action: "fix",
+      action: issue.blocking ? "fix" : "review",
       detail: issue.message,
     })),
   ];

src/syncUx.ts

@@ -58,21 +58,24 @@ export function syncReviewPresentation(
     return {
       actionLabel: "Fix local files before syncing",
       actionDisabled: true,
-      message: "Synchronization is paused. Fix every listed local file, then refresh the review.",
+      message: "Synchronization is paused. Resolve the blocking issues, then refresh the review.",
     };
   }
   const outcomes = plan.actions.filter((action) => action.command !== "advance_checkpoint").length;
   const hasCheckpoint = plan.actions.some((action) => action.command === "advance_checkpoint");
+  const frontmatterWarning = plan.issues.some((issue) => issue.code === "invalid_frontmatter" && !issue.blocking);
   return {
     actionLabel: outcomes
       ? `Sync ${outcomes} ${outcomes === 1 ? "outcome" : "outcomes"}`
       : hasCheckpoint
         ? "Confirm sync checkpoint"
         : "Already up to date",
     actionDisabled: busy || plan.actions.length === 0,
-    message: entryCount
-      ? "Review each transfer below, then sync when ready."
-      : "This vault and the hosted collection are already aligned.",
+    message: frontmatterWarning
+      ? "Frontmatter warnings do not block this sync. Document bytes are preserved; synchronization does not repair YAML."
+      : entryCount
+        ? "Review each transfer below, then sync when ready."
+        : "This vault and the hosted collection are already aligned.",
   };
 }
 

src/workspaceView.ts

@@ -2279,7 +2279,9 @@ export class MdbaseWorkspaceView extends ItemView {
     } else if (preview.local_issues.length) {
       section.createDiv({
         cls: "mdbase-inline-message",
-        text: "Synchronization is paused until every invalid or unreadable local file listed here is fixed.",
+        text: preview.plan.summary.blocking_issues > 0
+          ? "Synchronization is paused. Resolve the blocking issues, then refresh the review."
+          : "These diagnostics do not block synchronization. Document bytes are preserved unchanged.",
       });
     }
   }
@@ -2453,7 +2455,7 @@ export class MdbaseWorkspaceView extends ItemView {
     const section = container.createEl("section", { cls: "mdbase-editor-section" });
     section.createEl("h3", { text: "Local files needing attention" });
     section.createEl("p", {
-      text: "Synchronization is paused to keep the mirror checkpoint exact. Fix every malformed or unreadable file below, then preview again.",
+      text: "Review these file diagnostics. The sync preview identifies which issues block synchronization; a diagnostic alone does not mean syncing is paused.",
     });
     for (const issue of status.local_issues) {
       const row = section.createDiv({ cls: "mdbase-conflict-row" });

test/syncPreview.test.ts

@@ -76,6 +76,16 @@ test("sync preview is a direct projection of the engine-owned plan", () => {
   assert.equal(preview.upload_files, 1);
 });
 
+test("nonblocking diagnostics are optional review, not mandatory repair", () => {
+  const preview = previewFromPlan(plan({
+    issues: [{ code: "invalid_frontmatter", path: "opaque.md", message: "Invalid YAML", blocking: false }],
+  }));
+  assert.equal(preview.entries[0]?.action, "review");
+  assert.equal(preview.local_issues[0]?.path, "opaque.md");
+  assert.equal(preview.plan.summary.blocking_issues, 0);
+  assert.deepEqual(preview.plan.actions, []);
+});
+
 test("an exact idle plan remains an explicit zero-action preview", () => {
   const preview = previewFromPlan(plan({
     base_cursor: 6,

test/syncUx.test.ts

@@ -60,10 +60,35 @@ test("blocking sync reviews use fix-first wording and disable apply", () => {
 
   assert.equal(presentation.actionLabel, "Fix local files before syncing");
   assert.equal(presentation.actionDisabled, true);
-  assert.match(presentation.message, /paused.*fix every listed local file/i);
+  assert.match(presentation.message, /paused.*resolve the blocking issues/i);
   assert.doesNotMatch(`${presentation.actionLabel} ${presentation.message}`, /up to date|sync when ready/i);
 });
 
+test("nonblocking frontmatter diagnostics leave exact transfers enabled", () => {
+  const reviewed = plan({
+    actions: [{
+      command: "put_remote", action_id: "upload", depends_on: [],
+      target: { entity: "record", identity: "r", path: "opaque.md", revision: "r1", payload_revision: "r1" },
+      payload_revision: "r1", expected_remote: { state: "absent" }, expected_local: { state: "absent" },
+      idempotency_key: "upload", reason: "local_change",
+    }],
+    issues: [{ code: "invalid_frontmatter", path: "opaque.md", message: "Invalid YAML", blocking: false }],
+    summary: { uploads: 1, downloads: 0, conflicts: 0, blocking_issues: 0 },
+  });
+  const presentation = syncReviewPresentation(reviewed, 2);
+  assert.equal(presentation.actionDisabled, false);
+  assert.equal(presentation.actionLabel, "Sync 1 outcome");
+  assert.match(presentation.message, /warnings do not block.*bytes are preserved/i);
+  assert.equal(syncReviewPresentation(reviewed, 2, true).actionDisabled, true);
+  const blocked = syncReviewPresentation({ ...reviewed, actions: [], issues: [
+    ...reviewed.issues,
+    { code: "file_read_failed", path: "unreadable.md", message: "Unreadable", blocking: true },
+  ], summary: { uploads: 0, downloads: 0, conflicts: 0, blocking_issues: 1 } }, 2);
+  assert.equal(blocked.actionDisabled, true);
+  assert.match(blocked.message, /blocking issues/);
+  assert.doesNotMatch(blocked.message, /fix every|do not block/i);
+});
+
 test("sync indicator gives transfer, attention, waiting, and synced states stable priority", () => {
   const base = { connected: true, status: status(), progress: null, fileProgress: null, problem: null, validationIssues: 0, localChangeObserved: false };
   assert.equal(syncIndicator(base).state, "synced");