diff --git a/AGENTS.md b/AGENTS.md index 224cda71..c8f3ee02 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -90,13 +90,13 @@ The private Yarn workspace root contains the public library and its consumers. U **Entities** (`src/entity/`): `Entities.getEntity` loads a versioned `entity-models` file. By default it selects `main` plus the dataset `passes` that have no `when` label or whose label is in `options.when` (`getPassList` lists them); `layer`/`layers` select exactly the named layers in order, without passes. `textures` overrides are keyed like `EntityModel.layers`, where a further draw of the same geometry is `#` from 2. `getLayerList` lists available names. Each draw uses the material for its dataset `render` mode (`Materials.createEntityCanvasMaterial`), cached per texture, mode, and tint; modes that vanilla culls get no inward faces. `EntityObject` `tints` maps pass tint labels to colours. `energy_swirl` and `breeze_wind` materials are owned by their object: `Ticker` advances `EntityObject.age` and the texture offset while the object has a parent; removal pauses it and reattachment resumes it. `EntityObject.getLayerGroup` and layer-scoped `getGroupByName`/`getMeshByName` expose each layer’s visibility and poses. Parts form a nested `Object3D` hierarchy with radian poses, inherited texture dimensions, cube growth, and mirrored box UVs. The entity root applies the dataset's `transform` (vanilla's renderer operations for the default state: living entities face +Z with their feet at the origin); models without one get scale `(-1, -1, 1)`. Nested child nodes hold this transform, one per operation, so every operation order is exact and caller position, rotation, and scale apply on top of it. `flip: true` forces that plain flip and `flip: false` keeps raw model space. Without an explicit texture key, use dataset `textureLocation` first, then the resolver (model path, a same-named texture in its directory, then variant data); `getBlock` and `getBlockList` are deprecated aliases. `Entities.getDatasetRoot` loads geometry, lists, animations, and the block index from the nearest published release when the exact dataset version is missing; textures keep the requested root. `src/bedrock/` contains type declarations only. -**World** (`src/world/`): `MineRenderWorld` is an early prototype with sparse chunks keyed by signed coordinates. Cubic 16³ `Chunk`s store `ChunkData` palettes with immutable states and recycled unused entries, 4,096 `Uint16` state indices, and per-cell NBT. Render handles and stable `BlockInfo` views remain separate; `.block` returns detached snapshots. Block coordinates are integers; chunk-local coordinates are 0–15. World and owned-chunk setters refresh `cullface` masks after edits, including across chunk boundaries; only opaque full-cube models occlude neighbors. The versioned `blockCulling` dataset lists vanilla's further face rules, applied in both rendering modes: glass, ice, slime and similar blocks hide faces toward the same block, mangrove roots toward the same block above or below, and connected panes or bars toward each other. Leaves keep their faces, as vanilla's cutout leaves do; data without `blockCulling` applies no such rule. `new MineRenderWorld(scene, { sectionMeshing: true })` merges models of any shape, including cutout, translucent, multipart, and animated models, plus fluid surfaces into per-section meshes with separate solid, translucent, and fluid atlas pages (at most 2048×2048 pixels by default; `maxAtlasSize`), with one mesh per page. Page selection uses texture alpha (`hasTranslucency`), so plain glass and panes use the solid/cutout page where vanilla uses the translucent layer. Transparent model and fluid pages draw their quads back to front: perspective views sort by distance from the camera and re-sort when it enters another block, as vanilla does; orthographic views sort by view depth and re-sort when the view direction changes. Sorting stays within each atlas page; intersecting surfaces can still blend incorrectly, and Three.js sorts the meshes for different pages or sections as whole objects. Per-position flower offsets remain unsupported. Only block entities keep individual objects; only opaque full cubes occlude neighbors. Animated atlases occupy their own pages and redraw on frame changes. Fluid surfaces use an 18×18×18 cell neighborhood and are built in the section worker when available. The section worker entry must stay free of three, DOM, and Node imports, so pure geometry code lives in `SectionGeometry.ts` and `FluidQuads.ts`. Merged `BlockInfo.object` values are `undefined`; edit through world/chunk setters. Replacing, removing, or clearing blocks releases their instance slots. `await world.unloadChunkColumn(x, z)` removes every section at absolute chunk-column coordinates and refreshes neighboring faces and fluids. `await world.unloadChunkColumns([{ x, z }, ...])` validates, copies, and deduplicates a group before removal, then refreshes surviving faces, fluids, lighting, AO, and biome colors once. `await world.setBlockVisibleAt(position, visible)` preserves block data and reveals neighboring faces in both rendering modes; replacement or removal resets visibility. Without an executor, parsed columns copy each section's palette into the chunk, resolve each palette entry once, and fill cells in one synchronous pass; `placeMultiBlock` groups blocks per chunk and resolves each distinct state once per group, with weighted variants chosen per cell and yields after about 8 ms of work. Passing a `BatchedExecutor` keeps the previous per-position batches, order at repeated positions, and wait-for-active-batch failure semantics. Bulk placement applies saved properties before model creation and flushes neighbor culling once, including after partial failure; single-block edits update neighbors immediately. Parsed column placement and unloading mark whole sections and neighboring sections' shared boundary cells dirty; structure placement and single edits queue chunk-local indices, and the culling drain recomputes masks in a flat pass per section and rebuilds each changed section once through `SectionMesh.buildAsync`, which runs the pure `buildSectionGeometry` (`SectionGeometry.ts`, typed arrays in and out) in the browser section worker when `Env.provider.createWorker` returns one and synchronously otherwise; a rebuild superseded by a newer rebuild or `clear()` is disposed. `BatchedExecutor.submit` returns a task promise; `stop()` rejects queued and future tasks while active tasks finish. Idle executors hold no timer; caller-provided executors remain caller-owned. Saved biomes supply grass, foliage, dry-foliage, and per-cell water colors in both rendering modes. Saved lighting, corner ambient occlusion, and biome blending are opt-in; light propagation, exact partial-shape ambient occlusion, and LOD remain unsupported. The demos in `apps/web/` exercise `MineRenderWorld`. +**World** (`src/world/`): `MineRenderWorld` is an early prototype with sparse chunks keyed by signed coordinates. Cubic 16³ `Chunk`s store `ChunkData` palettes with immutable states and recycled unused entries, 4,096 `Uint16` state indices, and per-cell NBT. Render handles and stable `BlockInfo` views remain separate; `.block` returns detached snapshots. Block coordinates are integers; chunk-local coordinates are 0–15. World and owned-chunk setters refresh `cullface` masks after edits, including across chunk boundaries; only opaque full-cube models occlude neighbors. The versioned `blockCulling` dataset lists vanilla's further face rules, applied in both rendering modes: glass, ice, slime and similar blocks hide faces toward the same block, mangrove roots toward the same block above or below, and connected panes or bars toward each other. Leaves keep their faces, as vanilla's cutout leaves do; data without `blockCulling` applies no such rule. `new MineRenderWorld(scene, { sectionMeshing: true })` merges models of any shape, including cutout, translucent, multipart, and animated models, plus fluid surfaces into per-section meshes with separate solid, translucent, and fluid atlas pages (at most 2048×2048 pixels by default; `maxAtlasSize`), with one mesh per page. Page selection uses texture alpha (`hasTranslucency`), so plain glass and panes use the solid/cutout page where vanilla uses the translucent layer. Transparent model and fluid pages draw their quads back to front: perspective views sort by distance from the camera and re-sort when it enters another block, as vanilla does; orthographic views sort by view depth and re-sort when the view direction changes. Sorting stays within each atlas page; intersecting surfaces can still blend incorrectly, and Three.js sorts the meshes for different pages or sections as whole objects. Per-position flower offsets remain unsupported. Only block entities keep individual objects; only opaque full cubes occlude neighbors. Animated atlases occupy their own pages and redraw on frame changes. Fluid surfaces use an 18×18×18 cell neighborhood and are built in the section worker when available. The section worker entry must stay free of three, DOM, and Node imports, so pure geometry code lives in `SectionGeometry.ts` and `FluidQuads.ts`. Merged `BlockInfo.object` values are `undefined`; edit through world/chunk setters. Replacing, removing, or clearing blocks releases their instance slots. `await world.unloadChunkColumn(x, z)` removes every section at absolute chunk-column coordinates and refreshes neighboring faces and fluids. `await world.unloadChunkColumns([{ x, z }, ...])` validates, copies, and deduplicates a group before removal, then refreshes surviving faces, fluids, lighting, AO, and biome colors once. `await world.setBlockVisibleAt(position, visible)` preserves block data and reveals neighboring faces in both rendering modes; replacement or removal resets visibility. Without an executor, parsed columns copy each section's palette into the chunk, resolve each palette entry once, and fill cells in one synchronous pass; `placeMultiBlock` groups blocks per chunk and resolves each distinct state once per group, with weighted variants chosen per cell and yields after about 8 ms of work. `await world.placeChunks(columns)` places parsed columns in input order with one final neighbor refresh, including after partial failure. Passing a `BatchedExecutor` keeps the previous per-position batches, order at repeated positions, and wait-for-active-batch failure semantics. Bulk placement applies saved properties before model creation and flushes neighbor culling once, including after partial failure; single-block edits update neighbors immediately. Parsed column placement and unloading mark whole sections and neighboring sections' shared boundary cells dirty; structure placement and single edits queue chunk-local indices, and the culling drain recomputes masks in a flat pass per section and rebuilds each changed section once through `SectionMesh.buildAsync`, which runs the pure `buildSectionGeometry` (`SectionGeometry.ts`, typed arrays in and out) in the browser section worker when `Env.provider.createWorker` returns one and synchronously otherwise; a rebuild superseded by a newer rebuild or `clear()` is disposed. `BatchedExecutor.submit` returns a task promise; `stop()` rejects queued and future tasks while active tasks finish. Idle executors hold no timer; caller-provided executors remain caller-owned. Saved biomes supply grass, foliage, dry-foliage, and per-cell water colors in both rendering modes. Saved lighting, corner ambient occlusion, and biome blending are opt-in; light propagation, exact partial-shape ambient occlusion, and LOD remain unsupported. The demos in `apps/web/` exercise `MineRenderWorld`. `world.raycast(raycaster)` returns the nearest visible rendered block: integer `position`, detached `block` data, and scene-space `point`, `normal`, and `distance`. It resolves merged geometry through per-quad cell indices and individual geometry through owned model/instance references. Hidden blocks and unrelated scene objects are excluded; transparent texture pixels still participate in geometric picking. `world.raycastEntities(raycaster)` returns the nearest visible rendered mob owned by that world when `renderEntities` is enabled: canonical `type`, copied saved fractional `position` in block units, live `object`, and scene-space `point`, `normal`, and `distance`. Hidden or detached entities are excluded. Compare block and entity hit distances to account for occlusion; the world demo selects the nearer hit and outlines the selected mob's visible bounds. -`WorldStreamer` loads nearby columns in distance order, retains a bounded surrounding area, and serializes placement and unloading. Its world must provide `unloadChunkColumns`, as `MineRenderWorld` does, to group obsolete columns and disposal cleanup. Failed bulk unloads retain ownership for retry and invalidate loaded entries before any replacement loads. `WorldChunkSource.getChunk(x, z, signal?)` takes absolute chunk coordinates; the streamer aborts obsolete source reads and active reads during disposal. `AnvilWorldSource` reads regions on demand with count and byte cache limits; its `AnvilRegionReader(x, z, signal?)` callback takes region coordinates. The optional `readExternalChunk(x, z, signal?)` reads `.mcc` payloads at absolute chunk coordinates in the same dimension without caching them. Optional `readEntityRegion` reads a dimension's separate `entities/r.x.z.mca` files, including entity-only columns, with cache limits and cancellation. All readers must forward the optional signal to cancellable I/O to stop that work. Source failures appear in `failedChunks` without blocking other columns; `retryFailedChunks()` retries them. The world demo opens a local Java save folder or selected `.mca` and `.mcc` files, selects a dimension, and streams around the view center. +`WorldStreamer` loads nearby columns in distance order, retains a bounded surrounding area, and serializes placement and unloading. With `placeChunks`, it keeps at most four pending or decoded source columns and places the ready prefix together; a later unfinished read does not delay ready columns. Custom adapters without this method retain one-at-a-time reads and placement. Its world must provide `unloadChunkColumns`, as `MineRenderWorld` does, to group obsolete columns and disposal cleanup. Failed bulk unloads retain ownership for retry and invalidate loaded entries before any replacement loads. `WorldChunkSource.getChunk(x, z, signal?)` takes absolute chunk coordinates; the streamer aborts obsolete source reads and active reads during disposal. `AnvilWorldSource` reads regions on demand with count and byte cache limits; its `AnvilRegionReader(x, z, signal?)` callback takes region coordinates. The optional `readExternalChunk(x, z, signal?)` reads `.mcc` payloads at absolute chunk coordinates in the same dimension without caching them. Optional `readEntityRegion` reads a dimension's separate `entities/r.x.z.mca` files, including entity-only columns, with cache limits and cancellation. All readers must forward the optional signal to cancellable I/O to stop that work. Source failures appear in `failedChunks` without blocking other columns; `retryFailedChunks()` retries them. The world demo opens a local Java save folder or selected `.mca` and `.mcc` files, selects a dimension, and streams around the view center. **Formats**: `StructureParser` loads `.nbt`; `SchematicParser` converts numeric `.schematic` blocks to modern names; `parse(nbt, customMappings, options)` accepts `id:metadata` overrides and falls back to the selected version's legacy mappings; opt-in `lenient` ignores Materials, tries metadata zero, and skips unmapped blocks. `SpongeSchematicParser.parse(nbt)` loads v2/v3 `.schem` files with local palettes, applies Offset to block/entity positions, and normalizes entity payloads to vanilla NBT; size retains the source dimensions. Biomes, metadata, and global block ID palettes are not imported. `LitematicaParser.parse(nbt)` loads v5–7 `.litematic` files into a combined structure and named `regions`, each usable with `placeMultiBlock`. Region `position` and `signedSize` retain the source selection; block/entity positions use the schematic origin. Combined size encloses all region bounds, and later entries in `regions` replace earlier cells, including with air. Metadata and scheduled ticks are not imported. `AnvilParser` reads numeric and paletted Java `.mca` chunks with gzip, zlib, LZ4, or uncompressed payloads: `getChunkList`/`parseChunk` use region-local coordinates (0–31), while `placeChunk` replaces the absolute chunk column. Pre-1.13 numeric chunks reuse the schematic mappings; parser and `AnvilWorldSource` options accept `legacyMappings` overrides and `lenient` metadata-zero fallback. Neighbor- and block-entity-derived state reconstruction is not performed. For external `.mcc` payloads, pass `region: { x, z }` and `readExternalChunk` in the parser options; the reader receives absolute chunk coordinates. Parsers retain DataVersion and entity NBT without migration. Modern (1.18+) section biome palettes are kept as 64 IDs per section; `placeChunk` copies them into the world for `world.getBiomeAt(x, y, z)`, and uses them for block and water colors. Await `chunk.setBiomes(samples)` to refresh affected blocks and tall-plant halves above the section. With `renderEntities`, supported mobs render at their saved position and yaw; sheep wool colour and shearing and fox, axolotl and parrot variants apply; other appearance state is ignored. The option defaults to false. Column replacement, unloading, and `clear()` dispose the owned entity objects. Unsupported entities remain data only. `AnvilParser.parseEntityChunk` reads separate modern `entities/*.mca` records at region-local coordinates; `AnvilWorldSource` joins them with terrain when `readEntityRegion` is supplied. `NBTHelper.fromBuffer` also preserves NBT format, compression, and byte metadata. diff --git a/ROADMAP.md b/ROADMAP.md index fb12a7a9..786037df 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -108,7 +108,7 @@ Immediate fixes: ~~fix `getChunkAt` to use `Map.get(key)`~~; ~~remove the 4×4× ~~Pick rendered mobs through `world.raycastEntities` and inspect their type and saved position in the world demo.~~ -~~Resolve each block state once per placement group and yield to the event loop instead of a timer-driven batch queue.~~ ~~Build section geometry in a browser worker.~~ Region decoding still runs on the main thread; wasm or worker-side chunk parsing remains a scale option once placement stops dominating. +~~Resolve each block state once per placement group and yield to the event loop instead of a timer-driven batch queue.~~ ~~Build section geometry in a browser worker.~~ ~~Group ready streamed columns with a four-column read window and refresh shared borders once per placement group.~~ Region decoding still runs on the main thread; wasm or worker-side chunk parsing remains a scale option once placement stops dominating. ~~Render sloped water/lava surfaces with still/flow textures and waterlogged blocks.~~ ~~Cover intrinsic water in kelp, seagrass, and bubble columns.~~ For remaining fluid parity, obtain vanilla block-state fluid definitions, face-occlusion shapes, and solidity/flow-blocking flags; add water overlays against glass/leaves. Keep 1 block = 16 units. diff --git a/apps/web/benchmarks/README.md b/apps/web/benchmarks/README.md index 35078cb2..a27f29e0 100644 --- a/apps/web/benchmarks/README.md +++ b/apps/web/benchmarks/README.md @@ -66,7 +66,7 @@ The report separates these measurements: Each stop records loaded, pending, missing, and failed columns, drawing calls and triangles, Three.js geometry and texture counts, and evidence of worker replies or synchronous fallback. Cleanup is timed separately. Garbage collection runs after frame sampling, so its pause is excluded from streaming and frame measurements. Heap samples include shared assets and benchmark records that remain after world cleanup. The runner exits unsuccessfully for source failures or browser errors and preserves completed reports for inspection. -Unloading records one operation per group of columns the streamer removes together, with its `columnCount` and `columns`. +Placement and unloading record one operation per group of columns the streamer handles together, with its `columnCount` and `columns`. Compare total columns as well as operation counts. Source reads can overlap; their summed wall times are not total streaming time. ## Compare runs diff --git a/apps/web/benchmarks/world.ts b/apps/web/benchmarks/world.ts index 2d3b43e4..8bfdb589 100644 --- a/apps/web/benchmarks/world.ts +++ b/apps/web/benchmarks/world.ts @@ -262,6 +262,14 @@ async function runWorldBenchmark(config: WorldBenchmarkConfig) { try { await world.placeChunk(chunk); } finally { operation = null; } }), + placeChunks: chunks => { + const columns = chunks.map(({ x, z }) => ({ x, z })); + return timed("placement", { columnCount: chunks.length, columns }, async () => { + operation = { id: ++nextOperationId, kind: "placement", columns }; + try { await world.placeChunks(chunks); } + finally { operation = null; } + }); + }, unloadChunkColumns: columns => timed("unload", { columnCount: columns.length, columns: columns.map(({ x, z }) => ({ x, z })) }, async () => { diff --git a/packages/minerender/reference/concepts.md b/packages/minerender/reference/concepts.md index 13970130..d0a637e8 100644 --- a/packages/minerender/reference/concepts.md +++ b/packages/minerender/reference/concepts.md @@ -208,12 +208,18 @@ For direct decoding, call `AnvilParser.parseEntityChunk(bytes, localX, localZ)`. Call `updatePosition` after the camera or view center moves. It accepts scene units; `update(x, z)` accepts absolute chunk coordinates. -The streamer aborts obsolete reads and active reads during disposal. Source callbacks must forward the optional signal to cancellable I/O to stop that work. +With `MineRenderWorld`, the streamer reads ahead by at most four columns, counting both pending reads and decoded results. It places the nearest ready columns together and refreshes their shared borders once. An unfinished later read does not delay the ready prefix. Custom world adapters can provide `placeChunks` to use this path; adapters with only `placeChunk` retain one-at-a-time reads and placement. + +The streamer aborts obsolete reads and active reads during disposal. Source callbacks must forward the optional signal to cancellable I/O to stop that work. Reads that ignore cancellation occupy their window slots until they settle. `dispose()` waits for them; a placement failure rejects without waiting for them. `readExternalChunk` handles oversized chunks stored beside their region file. Its coordinates are absolute chunk coordinates, and it reads from the same dimension as the region reader. The source caches region files within its configured limits; external payloads are read on demand without retaining them. A missing external file is a chunk error and can be retried. Source errors appear in `failedChunks` while other columns continue loading. Call `await stream.retryFailedChunks()` to retry them. Placement or unloading failures reject the update. +A failed placement group can have changed any of its columns. The streamer cleans the whole group before loading it again, and retains ownership if cleanup fails. An active placement finishes before eviction or disposal. + +Call `await world.placeChunks(columns)` to replace already decoded columns together. The method validates and copies all column and section coordinates before mutation, preserves input order, and refreshes queued biomes, faces, fluids, saved lighting, and ambient occlusion after the group. If a column fails, later columns are skipped and applied changes refresh before the promise rejects. The first placement error takes precedence over a refresh error. The input's section data remains caller-owned; keep it unchanged until placement finishes. + Use `await world.unloadChunkColumns([{ x: -1, z: 0 }, { x: 0, z: 0 }])` to remove several columns and refresh surviving faces, fluids, saved lighting, ambient occlusion, and biome colors together. Coordinates are absolute chunk columns. The method validates and copies the whole input before removal, ignores duplicates and missing columns, and refreshes surviving neighbors once. The streamer groups obsolete columns and disposal cleanup through this method. Its world must provide `unloadChunkColumns`; `MineRenderWorld` provides it. A rejected bulk unload keeps its column ownership for retry and invalidates the group's loaded entries, because some columns may already have been removed. The next update finishes that cleanup before loading the latest requested area. diff --git a/packages/minerender/src/util/util.ts b/packages/minerender/src/util/util.ts index 013b38b3..b3547652 100644 --- a/packages/minerender/src/util/util.ts +++ b/packages/minerender/src/util/util.ts @@ -40,10 +40,11 @@ export async function sleep(timeout: number): Promise { }); } -/** Yields to the next macrotask without timer clamping when the platform supports it. */ +/** + * Yields to the next macrotask without timer clamping when the platform supports it. + * Not `scheduler.yield()`: Chrome runs its continuations ahead of rendering, so chained yields delay frames. + */ export function yieldToEventLoop(): Promise { - const scheduler = (globalThis as typeof globalThis & { scheduler?: { yield?: () => Promise } }).scheduler; - if (scheduler?.yield) return scheduler.yield(); return new Promise(resolve => { if (typeof MessageChannel !== "undefined") { const channel = new MessageChannel(); diff --git a/packages/minerender/src/world/MineRenderWorld.ts b/packages/minerender/src/world/MineRenderWorld.ts index cae39bdc..8000a78b 100644 --- a/packages/minerender/src/world/MineRenderWorld.ts +++ b/packages/minerender/src/world/MineRenderWorld.ts @@ -22,6 +22,7 @@ import { loadBlockCulling } from "./_culling/BlockCulling"; import { logger } from "../util/log"; const log = logger("MineRenderWorld"); +const YIELD_INTERVAL_MS = 8; //TODO: maybe make this an Object3D to add children /** @@ -165,7 +166,7 @@ export class MineRenderWorld { failures.set(chunk, error); if (this._chunks.get(`${chunk.x}_${chunk.y}_${chunk.z}`) === chunk) this.pendingBiomes.set(chunk, undefined); } - if (performance.now() - sliceStart > 8) { + if (performance.now() - sliceStart > YIELD_INTERVAL_MS) { await yieldToEventLoop(); sliceStart = performance.now(); } @@ -366,7 +367,7 @@ export class MineRenderWorld { chunk.y * 16 + Math.floor(index / 256), chunk.z * 16 + Math.floor(index / 16) % 16)); for (const pos of positions) changes.set(`${pos.x},${pos.y},${pos.z}`, pos); } - if (performance.now() - sliceStart > 8) { + if (performance.now() - sliceStart > YIELD_INTERVAL_MS) { await yieldToEventLoop(); sliceStart = performance.now(); } @@ -376,6 +377,58 @@ export class MineRenderWorld { /** Replaces a chunk column's blocks, biomes, and saved light and, with `renderEntities`, its supported saved mobs. */ public async placeChunk(chunk: AnvilChunk, executor?: BatchedExecutor): Promise { + return this.placeColumn(chunk, executor); + } + + /** + * Replaces parsed columns in input order, then refreshes neighboring geometry once. + * Duplicate coordinates retain their order. A failed column stops placement; queued refreshes finish before rejection. + */ + public async placeChunks(chunks: readonly AnvilChunk[]): Promise { + const columns: AnvilChunk[] = []; + for (const chunk of chunks) { + const { x, z } = chunk ?? {}; + if (![x, z].every(Number.isSafeInteger)) { + throw new RangeError("Chunk column coordinates must be safe integers"); + } + const sections = Array.from(chunk.sections, section => { + const { y } = section ?? {}; + if (!Number.isSafeInteger(y)) throw new RangeError("Chunk section coordinates must be safe integers"); + return { ...section, y }; + }); + columns.push({ ...chunk, x, z, sections }); + } + if (!columns.length) return; + await this.culling; + let failed = false, failure: unknown, sliceStart = performance.now(); + try { + for (const column of columns) { + await this.placeColumn(column, undefined, true); + if (performance.now() - sliceStart > YIELD_INTERVAL_MS) { + await yieldToEventLoop(); + sliceStart = performance.now(); + } + } + } catch (error) { + failed = true; + failure = error; + } finally { + try { + const failures = await this.refreshBiomes(true); + if (failures.size) log.warn("Failed to refresh biome colors", failures.values().next().value); + } catch (error) { + if (!failed) { failed = true; failure = error; } + } + try { + await this.updateCulling([]); + } catch (error) { + if (!failed) { failed = true; failure = error; } + } + } + if (failed) throw failure; + } + + private async placeColumn(chunk: AnvilChunk, executor?: BatchedExecutor, deferCulling = false): Promise { if (![chunk.x, chunk.z].every(Number.isSafeInteger)) { throw new RangeError("Chunk column coordinates must be safe integers"); } @@ -438,7 +491,7 @@ export class MineRenderWorld { } finally { this.markSectionChanged(target); } - if (performance.now() - sliceStart > 8) { + if (performance.now() - sliceStart > YIELD_INTERVAL_MS) { await yieldToEventLoop(); sliceStart = performance.now(); } @@ -459,9 +512,12 @@ export class MineRenderWorld { await placeEntities?.(); } finally { this.markBiomesChanged(chunk.x, chunk.z, undefined, false); - const failures = await this.refreshBiomes(); - if (failures.size) log.warn("Failed to refresh biome colors", failures.values().next().value); - await this.updateCulling([...changes.values()]); + if (deferCulling) await this.updateCulling([...changes.values()], true); + else { + const failures = await this.refreshBiomes(); + if (failures.size) log.warn("Failed to refresh biome colors", failures.values().next().value); + await this.updateCulling([...changes.values()]); + } } } @@ -493,7 +549,7 @@ export class MineRenderWorld { for (const { x, z } of columns.values()) { this.entities?.clearColumn(x, z); await this.clearChunkColumn(x, z); - if (performance.now() - sliceStart > 8) { + if (performance.now() - sliceStart > YIELD_INTERVAL_MS) { await yieldToEventLoop(); sliceStart = performance.now(); } @@ -639,15 +695,26 @@ export class MineRenderWorld { for (const [index, mask] of masks) await chunk.setCullMaskIndex(index, mask); await chunk.refreshLight(indices === "all" ? undefined : indices); chunk.refreshAmbientOcclusion(this._ambientOcclusion); - if ((this._ambientOcclusion || this.ambientOcclusionPending) && performance.now() - sliceStart > 8) { + if (performance.now() - sliceStart > YIELD_INTERVAL_MS) { + await yieldToEventLoop(); + sliceStart = performance.now(); + } + } + const pending: Promise[] = []; + for (const chunk of [...batch.keys()]) { + const build = (async () => { + await chunk.rebuildSectionMesh(); + batch.delete(chunk); + })(); + // Handle rejections while dispatch yields; all started builds settle before their results are checked. + void build.catch(() => undefined); + pending.push(build); + if (performance.now() - sliceStart > YIELD_INTERVAL_MS) { await yieldToEventLoop(); sliceStart = performance.now(); } } - const builds = await Promise.allSettled([...batch.keys()].map(async chunk => { - await chunk.rebuildSectionMesh(); - batch.delete(chunk); - })); + const builds = await Promise.allSettled(pending); const failure = builds.find(result => result.status === "rejected"); if (failure?.status === "rejected") throw failure.reason; } catch (error) { diff --git a/packages/minerender/src/world/WorldStreamer.ts b/packages/minerender/src/world/WorldStreamer.ts index ef9fa1ff..e0126bde 100644 --- a/packages/minerender/src/world/WorldStreamer.ts +++ b/packages/minerender/src/world/WorldStreamer.ts @@ -14,6 +14,18 @@ interface ChunkPosition { z: number; } +interface ChunkRead { + position: ChunkPosition; + controller: AbortController; + promise: Promise; + ready: boolean; + chunk?: AnvilChunk; + failed: boolean; + error?: unknown; +} + +const MAX_PREFETCHED_COLUMNS = 4; + /** * Loads nearby chunk columns in distance order and unloads distant columns. * Updates coalesce to the latest center and serialize world mutations. Use a dedicated world: @@ -30,12 +42,14 @@ export class WorldStreamer { private desired: ChunkPosition[] = []; private center?: ChunkPosition; private loading?: { position: ChunkPosition; controller: AbortController }; + private readonly reads = new Map(); private running?: Promise; private disposal?: Promise; private disposed = false; constructor( - private readonly world: Pick, "placeChunk" | "unloadChunkColumns">, + private readonly world: Pick, "placeChunk" | "unloadChunkColumns"> + & Partial, "placeChunks">>, private readonly source: WorldChunkSource, options: WorldStreamerOptions = {} ) { @@ -88,6 +102,12 @@ export class WorldStreamer { if (this.loading && !this.within(this.loading.position, this.loadRadius)) { this.loading.controller.abort(); } + for (const [key, read] of this.reads) { + if (!this.within(read.position, this.loadRadius)) { + read.controller.abort(); + if (read.ready) this.reads.delete(key); + } + } } return this.running ??= Promise.resolve().then(() => this.drain()); } @@ -111,8 +131,11 @@ export class WorldStreamer { this.desired = []; this.center = undefined; this.loading?.controller.abort(); + for (const read of this.reads.values()) read.controller.abort(); return this.disposal ??= Promise.resolve().then(async () => { await this.running?.catch(() => undefined); + await Promise.all([...this.reads.values()].map(read => read.promise)); + this.reads.clear(); try { await this.unloadMany([...this.owned.values()]); } finally { @@ -141,6 +164,10 @@ export class WorldStreamer { await this.unloadMany(obsolete); continue; } + if (this.world.placeChunks) { + if (await this.placeReadyChunks()) continue; + return; + } const position = this.desired.find(position => this.needsLoad(position)); if (!position) return; let chunk: AnvilChunk | undefined; @@ -163,26 +190,105 @@ export class WorldStreamer { continue; } this.owned.set(this.key(position), position); + await this.placeOwnedColumns([position], () => this.world.placeChunk(chunk!)); + } + } finally { + for (const [key, read] of this.reads) { + read.controller.abort(); + if (read.ready) this.reads.delete(key); + } + this.running = undefined; + } + } + + private fillReadSlots(): void { + while (!this.disposed && this.reads.size < MAX_PREFETCHED_COLUMNS) { + const position = this.desired.find(position => this.needsLoad(position) && !this.reads.has(this.key(position))); + if (!position) return; + const key = this.key(position); + const read: ChunkRead = { + position, controller: new AbortController(), promise: Promise.resolve(), ready: false, failed: false + }; + this.reads.set(key, read); + read.promise = (async () => { try { - await this.world.placeChunk(chunk); - this.loaded.set(this.key(position), position); + read.chunk = await this.source.getChunk(position.x, position.z, read.controller.signal); + if (read.chunk) this.validateChunk(read.chunk, position); } catch (error) { - try { - await this.unloadMany([position]); - } catch (cleanupError) { - const failure = error instanceof Error ? error : new Error(String(error)); - try { - Object.defineProperty(failure, "cause", { value: cleanupError, configurable: true }); - } catch { - throw Object.assign(new Error(failure.message), { cause: cleanupError }); - } - throw failure; + read.failed = true; + read.error = error; + } finally { + read.ready = true; + if (read.controller.signal.aborted || this.disposed || !this.within(position, this.loadRadius)) { + this.reads.delete(key); } - throw error; + } + })(); + } + } + + /** Places only the ready prefix, so later source reads cannot delay the nearest ready column. */ + private async placeReadyChunks(): Promise { + this.fillReadSlots(); + const position = this.desired.find(position => this.needsLoad(position)); + if (!position) { + if (!this.reads.size) return false; + await Promise.all([...this.reads.values()].map(read => read.promise)); + return true; + } + const head = this.reads.get(this.key(position)); + if (!head) { + for (let index = this.desired.length - 1; index >= 0; index--) { + const key = this.key(this.desired[index]); + if (this.reads.get(key)?.ready) { + this.reads.delete(key); + return true; } } - } finally { - this.running = undefined; + // Aborted reads still occupy slots until their source work settles. + await Promise.race([...this.reads.values()].map(read => read.promise)); + return true; + } + if (!head.ready) { + await head.promise; + return true; + } + const chunks: AnvilChunk[] = [], positions: ChunkPosition[] = []; + for (const desired of this.desired) { + if (!this.needsLoad(desired)) continue; + const key = this.key(desired), read = this.reads.get(key); + if (!read?.ready) break; + this.reads.delete(key); + if (read.failed) this.failed.set(key, { ...desired, error: read.error }); + else if (!read.chunk) this.missing.set(key, desired); + else { + chunks.push(read.chunk); + positions.push(desired); + } + } + if (!chunks.length) return true; + for (const position of positions) this.owned.set(this.key(position), position); + await this.placeOwnedColumns(positions, () => this.world.placeChunks!(chunks)); + return true; + } + + private async placeOwnedColumns(positions: readonly ChunkPosition[], place: () => Promise): Promise { + try { + await place(); + for (const position of positions) this.loaded.set(this.key(position), position); + } catch (error) { + try { + await this.unloadMany(positions); + } catch (cleanupError) { + const failure = error instanceof Error ? error : new Error(String(error)); + try { + Object.defineProperty(failure, "cause", { value: cleanupError, configurable: true }); + } catch { + throw Object.assign(new Error(failure.message), { cause: cleanupError }); + } + throw failure; + } + throw error; } } diff --git a/packages/minerender/test/World.test.ts b/packages/minerender/test/World.test.ts index f4d4a599..324e2e50 100644 --- a/packages/minerender/test/World.test.ts +++ b/packages/minerender/test/World.test.ts @@ -322,6 +322,86 @@ for (const useExecutor of [false, true]) test.serial(`parsed chunk columns repla if (executor) t.is(await executor.submit(() => "reusable"), "reusable"); }); +test.serial("column batches preserve signed replacements and repeated column order", async t => { + const { world, scene } = fixture(t); + const getIndex = BlockEntities.getIndex; + BlockEntities.getIndex = async () => ({}); + t.teardown(() => { BlockEntities.getIndex = getIndex; }); + await world.setBlockAt(-32, 100, 48, block); + const neighbor = await world.setBlockAt(-16, 100, 48, block); + const first = new ChunkData(), last = new ChunkData(), removed = new ChunkData(); + first.set(0, { ...block, properties: { axis: "x" } }); + last.set(4095, { ...block, properties: { axis: "z" }, nbt: { items: [1] } }); + removed.set(0, block); + const columns = Object.freeze([ + Object.freeze({ x: -2, z: 3, sections: [{ y: 0, data: first }] }), + Object.freeze({ x: 1, z: -1, sections: [{ y: 1, data: removed }] }), + Object.freeze({ x: -2, z: 3, sections: [{ y: -4, data: last }] }), + Object.freeze({ x: 1, z: -1, sections: [] }) + ]); + await world.placeChunks(columns); + t.is(world.getBlockAt(-32, 100, 48), undefined); + t.is(world.getBlockAt(-32, 0, 48), undefined); + t.deepEqual(world.getBlockAt(-17, -49, 63)!.block, last.get(4095)); + t.deepEqual(world.getBlockAt(-17, -49, 63)!.object.getPosition().toArray(), [-272, -784, 1008]); + t.is(world.getBlockAt(16, 16, -16), undefined); + t.is(world.getBlockAt(-16, 100, 48), neighbor); + t.is(scene.stats.instanceCount, 2); + t.deepEqual(first.get(0)!.properties, { axis: "x" }); + scene.dirty = false; + await world.placeChunks([]); + t.false(scene.dirty); + t.is(scene.stats.instanceCount, 2); +}); + +test.serial("column batches validate every location before changing existing columns", async t => { + const { world, scene, loads } = fixture(t); + const existing = await world.setBlockAt(0, 0, 0, block); + const data = new ChunkData(); + data.set(0, block); + loads.length = 0; + for (const value of [0.5, NaN, Infinity, Number.MAX_SAFE_INTEGER + 1]) { + for (const invalid of [ + { x: value, z: 1, sections: [] }, + { x: 1, z: value, sections: [] }, + { x: 1, z: 1, sections: [{ y: value, data }] } + ]) { + await t.throwsAsync(world.placeChunks([{ x: 0, z: 0, sections: [] }, invalid]), { instanceOf: RangeError }); + t.is(world.getBlockAt(0, 0, 0), existing); + t.is(scene.stats.instanceCount, 1); + t.deepEqual(loads, []); + } + } +}); + +test.serial("column batches snapshot column and section locations before awaiting active culling", async t => { + const { world } = fixture(t); + const getIndex = BlockEntities.getIndex; + BlockEntities.getIndex = async () => ({}); + t.teardown(() => { BlockEntities.getIndex = getIndex; }); + const data = new ChunkData(); + data.set(4095, block); + let release!: () => void; + world["culling"] = new Promise(resolve => { + release = () => { world["culling"] = undefined; resolve(); }; + }); + t.teardown(() => release()); + const section = { y: -1, data }; + const column = { x: -2, z: -1, sections: [section] }; + const columns = [column]; + const placing = world.placeChunks(columns); + column.x = 3; + column.z = 4; + section.y = 2; + column.sections.length = 0; + columns.length = 0; + t.is(world.getBlockAt(-17, -1, -1), undefined); + release(); + await placing; + t.deepEqual(world.getBlockAt(-17, -1, -1)!.block, block); + t.is(world.getBlockAt(63, 47, 79), undefined); +}); + test.serial("parsed columns clear unknown states, place remaining cells and reject with the first error", async t => { const { world, scene, loads } = fixture(t); await world.setBlockAt(0, 16, 0, block); diff --git a/packages/minerender/test/WorldCulling.test.ts b/packages/minerender/test/WorldCulling.test.ts index 09080266..16b43dcb 100644 --- a/packages/minerender/test/WorldCulling.test.ts +++ b/packages/minerender/test/WorldCulling.test.ts @@ -1406,6 +1406,107 @@ test.serial("chunk column placement culls across sections and empty replacement t.is(scene.stats.instanceCount, 1); }); +test.serial("column batches match sequential meshes across signed fluid, lighting, AO, and biome boundaries", async t => { + const { world, scene, colorAt } = biomeBlendFixture(t, 1, { savedLighting: true, ambientOcclusion: true }); + const left = new ChunkData(), right = new ChunkData(), north = new ChunkData(), diagonal = new ChunkData(); + left.set(15 + 12 * 16, { type: "test:cube" }); + left.set(15 + 8 * 16, { type: "oak_leaves" }); + left.set(255, { type: "water", properties: { level: "4" } }); + right.set(12 * 16, { type: "test:cube" }); + right.set(12 * 16 + 256, { type: "test:cube" }); + right.set(13 * 16 + 256, { type: "test:cube" }); + right.set(15 * 16, { type: "water" }); + north.set(15, { type: "water", properties: { level: "4" } }); + diagonal.set(0, { type: "water" }); + const columns = [ + { x: -1, z: -1, data: left, biome: "test:red", light: 0xaa }, + { x: 0, z: -1, data: right, biome: "test:blue", light: 0x44 }, + { x: -1, z: 0, data: north, biome: "test:red", light: 0x77 }, + { x: 0, z: 0, data: diagonal, biome: "test:blue", light: 0xee } + ].map(({ x, z, data, biome, light }) => ({ x, z, lightValid: true, sections: [ + { y: 0, data, biomes: Array(64).fill(biome), light: { sky: new Uint8Array(2048).fill(light) } } + ] })); + const meshState = () => scene.children.filter(child => child instanceof SectionMesh).flatMap(section => + section.children.flatMap(child => { + const geometry = (child as Mesh).geometry; + const attributes = ["position", "normal", "color", "shade"].map(name => geometry.getAttribute(name)); + return Array.from({ length: geometry.getAttribute("position").count / 4 }, (_, quad) => + JSON.stringify([section.position.toArray(), ...attributes.map(attribute => + Array.from(attribute.array.slice(quad * 4 * attribute.itemSize, (quad + 1) * 4 * attribute.itemSize)))])); + })).sort(); + let isolatedColor: number | undefined; + for (const [index, column] of columns.entries()) { + await world.placeChunk(column); + if (index === 0) isolatedColor = colorAt([-1, 0, -8]); + } + const expected = meshState(), corners = topCornerColors(world, [-1, 0, -4]); + const fluidPositions: TripleArray[] = [[-1, 0, -1], [0, 0, -1], [-1, 0, 0], [0, 0, 0]]; + const fluids = fluidPositions.map(position => assertFluidPositions(t, world, scene, [position])); + const color = colorAt([-1, 0, -8]); + t.not(color, isolatedColor); + t.true(new Set(corners).size > 1); + t.is(cullMaskAt(world, [-1, 0, -4]), 1); + await world.clear(); + const build = SectionMesh.buildAsync; + let builds = 0; + SectionMesh.buildAsync = async function (...args) { builds++; return build.apply(this, args); }; + t.teardown(() => { SectionMesh.buildAsync = build; }); + await world.placeChunks(columns); + t.is(builds, 4); + t.deepEqual(meshState(), expected); + t.deepEqual(topCornerColors(world, [-1, 0, -4]), corners); + t.is(colorAt([-1, 0, -8]), color); + t.is(cullMaskAt(world, [-1, 0, -4]), 1); + for (const [index, position] of fluidPositions.entries()) { + t.deepEqual(assertFluidPositions(t, world, scene, [position]), fluids[index]); + } + t.deepEqual(world.getLightAt(-1, 0, -4), { block: 0, sky: 10 }); + t.is(world.getBiomeAt(-1, 0, -8), "test:red"); + t.is(world.getBiomeAt(0, 0, -8), "test:blue"); +}); + +for (const failure of [new Error("column asset failed"), undefined]) { + test.serial(`column batches flush partial writes before rejecting and leave later columns untouched (${String(failure)})`, async t => { + const { world, scene, place } = fixture(t, { sectionMeshing: true }); + const survivor = await place([15, 0, 0]), untouched = await place([48, 0, 0]); + const first = new ChunkData(), broken = new ChunkData(), later = new ChunkData(); + first.set(0, { type: "test:cube" }); + broken.set(0, { type: "test:failure" }); + broken.set(1, { type: "test:cube" }); + later.set(1, { type: "test:cube" }); + const get = BlockStates.get, build = SectionMesh.buildAsync; + BlockStates.get = async key => { if (key.path === "failure") throw failure; return get(key); }; + if (failure === undefined) SectionMesh.buildAsync = async () => { throw new Error("later section flush failed"); }; + t.teardown(() => { BlockStates.get = get; SectionMesh.buildAsync = build; }); + let rejected = false, caught: unknown; + try { + await world.placeChunks([ + { x: 1, z: 0, sections: [{ y: 0, data: first }] }, + { x: 2, z: 0, sections: [{ y: 0, data: broken }] }, + { x: 3, z: 0, sections: [{ y: 0, data: later }] } + ]); + } catch (error) { rejected = true; caught = error; } + t.true(rejected); + t.is(caught, failure); + SectionMesh.buildAsync = build; + if (failure === undefined) await world.unloadChunkColumns([{ x: 99, z: 99 }]); + t.is(world.getBlockAt(15, 0, 0), survivor); + t.is(world.getBlockAt(48, 0, 0), untouched); + t.is(world.getBlockAt(49, 0, 0), undefined); + t.is(world.getBlockAt(32, 0, 0), undefined); + t.is(world.getBlockAt(33, 0, 0)!.block.type, "test:cube"); + t.is(renderedIndexCountAt(world, [15, 0, 0]), 30); + t.is(renderedIndexCountAt(world, [16, 0, 0]), 30); + t.is(renderedIndexCountAt(world, [33, 0, 0]), 36); + t.is(broken.get(0)!.type, "test:failure"); + await world.unloadChunkColumns([{ x: 1, z: 0 }, { x: 2, z: 0 }]); + t.is(world.getBlockAt(16, 0, 0), undefined); + t.is(world.getBlockAt(33, 0, 0), undefined); + t.is(renderedIndexCountAt(world, [15, 0, 0]), 36); + t.is(scene.children.filter(child => child instanceof SectionMesh).length, 2); + }); +} + test.serial("parsed columns match single edits for section geometry and per-block cull masks", async t => { const { world, scene, states, place } = fixture(t, { sectionMeshing: true }); states.set("test:entity", { key: AssetKey.parse("blockstates", "test:entity"), diff --git a/packages/minerender/test/WorldStreamer.test.ts b/packages/minerender/test/WorldStreamer.test.ts index c2394c07..cabd9afc 100644 --- a/packages/minerender/test/WorldStreamer.test.ts +++ b/packages/minerender/test/WorldStreamer.test.ts @@ -45,18 +45,26 @@ test("streaming retains biome-only sections until eviction and restores them on function fixture(options: { loadRadius?: number; unloadRadius?: number } = { loadRadius: 0, unloadRadius: 0 }, hooks: { get?: (x: number, z: number, signal?: AbortSignal) => Promise; place?: (value: AnvilChunk) => Promise; + placeChunks?: (chunks: readonly AnvilChunk[]) => Promise; unloadColumns?: (columns: readonly { x: number; z: number }[]) => Promise; } = {}) { const events: string[] = [], resident = new Set(); const bulkCalls: { x: number; z: number }[][] = []; + const placements: { x: number; z: number }[][] = []; let active = 0, peak = 0; + let activeReads = 0, peakReads = 0, activeMutations = 0, peakMutations = 0; async function operation(event: string, action: () => Promise): Promise { events.push(event); peak = Math.max(peak, ++active); + const reading = event.startsWith("get:"); + if (reading) peakReads = Math.max(peakReads, ++activeReads); + else peakMutations = Math.max(peakMutations, ++activeMutations); try { return await action(); } finally { active--; + if (reading) activeReads--; + else activeMutations--; } } const source = { @@ -74,7 +82,17 @@ function fixture(options: { loadRadius?: number; unloadRadius?: number } = { loa for (const { x, z } of requested) resident.delete(key(x, z)); }) }; - return { streamer: new WorldStreamer(world, source, options), events, resident, bulkCalls, peak: () => peak, world, source }; + if (hooks.placeChunks) Object.assign(world, { + placeChunks: (chunks: readonly AnvilChunk[]) => operation("place-many", async () => { + placements.push(chunks.map(({ x, z }) => ({ x, z }))); + for (const { x, z } of chunks) resident.add(key(x, z)); + await hooks.placeChunks!(chunks); + }) + }); + return { + streamer: new WorldStreamer(world, source, options), events, resident, bulkCalls, placements, + peak: () => peak, peakReads: () => peakReads, peakMutations: () => peakMutations, world, source + }; } test("streaming validates bounded radii and floors signed scene coordinates into chunk coordinates", async t => { @@ -830,3 +848,419 @@ test("disposing an empty bulk-capable streamer does not call world unload method t.deepEqual(setup.streamer.loadedChunks, []); await t.throwsAsync(() => setup.streamer.update(0, 0)); }); + +test("bulk placement prefetches four columns and groups ready columns in distance order", async t => { + t.timeout(3000); + const reads: { x: number; z: number; gate: ReturnType> }[] = []; + const started = deferred(), placing = deferred(), release = deferred(); + let first = true; + const setup = fixture({ loadRadius: 1, unloadRadius: 1 }, { + get: async (x, z) => { + if (reads.length < 4) { + const gate = deferred(); + reads.push({ x, z, gate }); + if (reads.length === 4) started.resolve(); + return gate.promise; + } + return chunk(x, z); + }, + placeChunks: async () => { + if (first) { + first = false; + placing.resolve(); + await release.promise; + } + } + }); + t.teardown(() => { reads.forEach(({ x, z, gate }) => gate.resolve(chunk(x, z))); release.resolve(); }); + const update = setup.streamer.update(0, 0); + await started.promise; + t.deepEqual(setup.placements, []); + for (const { x, z, gate } of reads) gate.resolve(chunk(x, z)); + await placing.promise; + t.deepEqual(setup.placements[0], reads.map(({ x, z }) => ({ x, z }))); + t.is(setup.placements[0].length, 4); + t.is(setup.events.filter(event => event.startsWith("get:")).length, 4); + release.resolve(); + await update; + const requested = setup.events.filter(event => event.startsWith("get:")).map(event => event.slice(4)); + t.deepEqual(setup.placements.flat().map(({ x, z }) => key(x, z)), requested); + const distances = setup.placements.flat().map(({ x, z }) => x * x + z * z); + t.deepEqual(distances, distances.slice().sort((a, b) => a - b)); + t.true(setup.placements.every(group => group.length > 0 && group.length <= 4)); + t.is(setup.streamer.loadedChunks.length, 9); + t.is(setup.peakReads(), 4); + t.is(setup.peakMutations(), 1); + t.false(setup.events.some(event => event.startsWith("place:"))); + await setup.streamer.dispose(); +}); + +test("a ready nearest column is placed before slow later reads, which stay within the prefetch window", async t => { + t.timeout(3000); + const initial = deferred(), firstPlacement = deferred(), fifthRead = deferred(); + const reads: { x: number; z: number; gate: ReturnType> }[] = []; + let allowReads = false; + const setup = fixture({ loadRadius: 1, unloadRadius: 1 }, { + get: async (x, z) => { + if (allowReads) return chunk(x, z); + const gate = deferred(); + reads.push({ x, z, gate }); + if (reads.length === 4) initial.resolve(); + if (reads.length === 5) fifthRead.resolve(); + return gate.promise; + }, + placeChunks: async () => { firstPlacement.resolve(); } + }); + t.teardown(() => reads.forEach(({ x, z, gate }) => gate.resolve(chunk(x, z)))); + const update = setup.streamer.update(0, 0); + await initial.promise; + for (const index of [0, 2, 3]) { + const { x, z, gate } = reads[index]; + gate.resolve(chunk(x, z)); + } + await firstPlacement.promise; + t.deepEqual(setup.placements, [[{ x: 0, z: 0 }]]); + t.is(reads.length, 4); + await fifthRead.promise; + await new Promise(resolve => setImmediate(resolve)); + t.is(reads.length, 5); + t.deepEqual(setup.placements, [[{ x: 0, z: 0 }]]); + allowReads = true; + for (const { x, z, gate } of reads) gate.resolve(chunk(x, z)); + await update; + const requested = setup.events.filter(event => event.startsWith("get:")).map(event => event.slice(4)); + t.deepEqual(setup.placements.flat().map(({ x, z }) => key(x, z)), requested); + t.is(setup.streamer.loadedChunks.length, 9); + await setup.streamer.dispose(); +}); + +test("a center move starts the nearest read when a finished read occupies the full window", async t => { + t.timeout(3000); + const started = deferred(), nearest = deferred(); + const reads: { x: number; z: number; gate: ReturnType>; signal?: AbortSignal }[] = []; + let allowReads = false; + const setup = fixture({ loadRadius: 1, unloadRadius: 1 }, { + get: async (x, z, signal) => { + if (allowReads) return chunk(x, z); + const gate = deferred(); + reads.push({ x, z, gate, signal }); + if (reads.length === 4) started.resolve(); + return gate.promise; + }, + placeChunks: async chunks => { + if (chunks.some(({ x, z }) => x === 0 && z === 1)) nearest.resolve(); + } + }); + t.teardown(() => reads.forEach(({ x, z, gate }) => gate.resolve(chunk(x, z)))); + const first = setup.streamer.update(0, 0); + await started.promise; + const latest = setup.streamer.update(0, 1); + t.true(reads.find(({ x, z }) => x === 0 && z === -1)!.signal?.aborted); + allowReads = true; + reads[0].gate.resolve(chunk(0, 0)); + await nearest.promise; + t.true(setup.placements.flat().some(({ x, z }) => x === 0 && z === 1)); + for (const { x, z, gate } of reads) gate.resolve(chunk(x, z)); + await Promise.all([first, latest]); + t.is(setup.streamer.loadedChunks.length, 9); + t.is(setup.streamer.pendingChunks, 0); + t.is(setup.peakReads(), 4); + await setup.streamer.dispose(); +}); + +test("retrying a failed column loads it when the full window holds a finished read", async t => { + t.timeout(3000); + const started = deferred(), full = deferred(), retried = deferred(); + const reads: { x: number; z: number; gate: ReturnType> }[] = []; + let allowReads = false; + const setup = fixture({ loadRadius: 1, unloadRadius: 1 }, { + get: async (x, z) => { + if (allowReads) { + if (x === 0 && z === 0) retried.resolve(); + return chunk(x, z); + } + const gate = deferred(); + reads.push({ x, z, gate }); + if (reads.length === 4) started.resolve(); + if (reads.length === 5) full.resolve(); + return gate.promise; + }, + placeChunks: async () => {} + }); + t.teardown(() => reads.forEach(({ x, z, gate }) => gate.resolve(chunk(x, z)))); + const update = setup.streamer.update(0, 0); + await started.promise; + reads[0].gate.reject(new Error("source failed")); + await full.promise; + reads[3].gate.resolve(chunk(reads[3].x, reads[3].z)); + await new Promise(resolve => setImmediate(resolve)); + t.is(setup.streamer.failedChunks.length, 1); + t.is(reads.length, 5); + allowReads = true; + const retry = setup.streamer.retryFailedChunks(); + reads[1].gate.resolve(chunk(reads[1].x, reads[1].z)); + await retried.promise; + for (const { x, z, gate } of reads) gate.resolve(chunk(x, z)); + await Promise.all([update, retry]); + t.true(setup.streamer.loadedChunks.some(({ x, z }) => x === 0 && z === 0)); + t.is(setup.streamer.loadedChunks.length, 9); + t.deepEqual(setup.streamer.failedChunks, []); + t.is(setup.peakReads(), 4); + await setup.streamer.dispose(); +}); + +test("moving the bulk target aborts every obsolete read and discards ignored late results", async t => { + t.timeout(3000); + const started = deferred(); + const reads: { x: number; z: number; gate: ReturnType>; signal?: AbortSignal }[] = []; + let allowReads = false; + const setup = fixture({ loadRadius: 1, unloadRadius: 1 }, { + get: async (x, z, signal) => { + if (allowReads) return chunk(x, z); + const gate = deferred(); + reads.push({ x, z, gate, signal }); + if (reads.length === 4) started.resolve(); + return gate.promise; + }, + placeChunks: async () => {} + }); + t.teardown(() => reads.forEach(({ x, z, gate }) => gate.resolve(chunk(x, z)))); + const first = setup.streamer.update(0, 0); + await started.promise; + const latest = setup.streamer.update(6, 0); + t.true(reads.every(read => read.signal?.aborted)); + t.is(setup.events.length, 4); + allowReads = true; + for (const { x, z, gate } of reads) gate.resolve(chunk(x, z)); + await Promise.all([first, latest]); + t.is(setup.streamer.loadedChunks.length, 9); + t.true(setup.placements.flat().every(({ x, z }) => Math.abs(x - 6) <= 1 && Math.abs(z) <= 1)); + t.deepEqual(setup.streamer.failedChunks, []); + t.is(setup.peakReads(), 4); + t.is(setup.peakMutations(), 1); + await setup.streamer.dispose(); +}); + +test("a center change during grouped placement unloads that whole group before placing the new area", async t => { + t.timeout(3000); + const started = deferred(), gate = deferred(); + let first = true; + const setup = fixture({ loadRadius: 1, unloadRadius: 1 }, { placeChunks: async () => { + if (first) { + first = false; + started.resolve(); + await gate.promise; + } + } }); + t.teardown(() => gate.resolve()); + const initial = setup.streamer.update(0, 0); + await started.promise; + const placed = setup.placements[0]; + const latest = setup.streamer.update(6, 0); + t.deepEqual(setup.bulkCalls, []); + gate.resolve(); + await Promise.all([initial, latest]); + t.deepEqual(setup.bulkCalls[0], placed); + t.true(setup.placements.slice(1).flat().every(({ x, z }) => Math.abs(x - 6) <= 1 && Math.abs(z) <= 1)); + t.is(setup.resident.size, 9); + t.is(setup.streamer.loadedChunks.length, 9); + t.is(setup.peakMutations(), 1); + await setup.streamer.dispose(); +}); + +test("disposal aborts bulk prefetch and waits for active grouped placement before cleanup", async t => { + t.timeout(3000); + const started = deferred(), placement = deferred(); + const signals: AbortSignal[] = []; + const reads: ReturnType>[] = []; + const setup = fixture({ loadRadius: 1, unloadRadius: 1 }, { + get: async (x, z, signal) => { + if (x === 0 && z === 0) return chunk(x, z); + const read = deferred(); + reads.push(read); + signals.push(signal!); + signal!.addEventListener("abort", () => read.reject(signal!.reason), { once: true }); + return read.promise; + }, + placeChunks: async () => { started.resolve(); await placement.promise; } + }); + t.teardown(() => { reads.forEach(read => read.resolve(chunk(0, 0))); placement.resolve(); }); + const update = setup.streamer.update(0, 0); + await started.promise; + const dispose = setup.streamer.dispose(); + t.true(signals.every(signal => signal.aborted)); + t.is(signals.length, 3); + t.deepEqual(setup.bulkCalls, []); + placement.resolve(); + await Promise.all([update, dispose]); + t.deepEqual(setup.bulkCalls, [[{ x: 0, z: 0 }]]); + t.deepEqual(setup.streamer.loadedChunks, []); + t.deepEqual(setup.streamer.failedChunks, []); + t.deepEqual([...setup.resident], []); + t.is(setup.streamer.pendingChunks, 0); + t.is(setup.peakMutations(), 1); +}); + +test("bulk source failures, missing columns, and coordinate mismatches do not prevent later ready columns", async t => { + const error = new Error("unreadable center"); + let fail = true; + const setup = fixture({ loadRadius: 1, unloadRadius: 1 }, { + get: async (x, z) => { + if (fail && x === 0 && z === 0) throw error; + if (x === 0 && z === -1) return undefined; + return chunk(fail && x === -1 && z === 0 ? x + 1 : x, z); + }, + placeChunks: async () => {} + }); + await setup.streamer.update(0, 0); + t.is(setup.streamer.loadedChunks.length, 6); + t.is(setup.streamer.failedChunks.length, 2); + t.is(setup.streamer.failedChunks[0].error, error); + t.is(setup.streamer.pendingChunks, 0); + const count = setup.events.filter(event => event.startsWith("get:")).length; + await setup.streamer.update(0, 0); + t.is(setup.events.filter(event => event.startsWith("get:")).length, count); + fail = false; + await setup.streamer.retryFailedChunks(); + t.is(setup.streamer.loadedChunks.length, 8); + t.deepEqual(setup.streamer.failedChunks, []); + t.is(setup.streamer.pendingChunks, 0); + await setup.streamer.dispose(); +}); + +test("a failed grouped placement cleans every attempted owner before retrying", async t => { + const error = new Error("partial placement"); + let fail = true; + const setup = fixture({ loadRadius: 1, unloadRadius: 1 }, { placeChunks: async chunks => { + if (fail) { + fail = false; + for (const { x, z } of chunks.slice(1)) setup.resident.delete(key(x, z)); + throw error; + } + } }); + let rejected = false; + try { await setup.streamer.update(0, 0); } + catch (failure) { rejected = true; t.is(failure, error); } + t.true(rejected); + t.deepEqual(setup.bulkCalls[0], setup.placements[0]); + t.deepEqual(setup.streamer.loadedChunks, []); + t.deepEqual(setup.streamer.failedChunks, []); + t.deepEqual([...setup.resident], []); + await setup.streamer.update(0, 0); + t.is(setup.streamer.loadedChunks.length, 9); + t.is(setup.resident.size, 9); + t.is(setup.peakMutations(), 1); + await setup.streamer.dispose(); +}); + +test("failed grouped-placement cleanup retains even unattempted owners for a later retry", async t => { + const placementError = new Error("partial placement"), cleanupError = new Error("cleanup failed"); + let failPlace = true, failUnload = true; + const setup = fixture({ loadRadius: 1, unloadRadius: 1 }, { + placeChunks: async chunks => { + if (failPlace) { + failPlace = false; + for (const { x, z } of chunks.slice(1)) setup.resident.delete(key(x, z)); + throw placementError; + } + }, + unloadColumns: async () => { + if (failUnload) { + failUnload = false; + throw cleanupError; + } + } + }); + const error = await t.throwsAsync(setup.streamer.update(0, 0), { message: placementError.message }); + t.is((error as Error & { cause?: unknown }).cause, cleanupError); + const attempted = setup.placements[0]; + t.is(attempted.length, 4); + t.deepEqual(setup.bulkCalls[0], attempted); + t.deepEqual(setup.streamer.loadedChunks, []); + t.is(setup.resident.size, 1); + const resumedAt = setup.events.length; + await setup.streamer.update(0, 0); + t.is(setup.events[resumedAt], "unload-many"); + t.deepEqual(setup.bulkCalls[1], attempted); + t.is(setup.streamer.loadedChunks.length, 9); + t.is(setup.resident.size, 9); + t.is(setup.peakMutations(), 1); + await setup.streamer.dispose(); +}); + +test("a placement rejection does not await unrelated reads, but disposal still waits for them", async t => { + t.timeout(3000); + const error = new Error("placement failed"); + const reads: { x: number; z: number; gate: ReturnType>; signal?: AbortSignal }[] = []; + const setup = fixture({ loadRadius: 1, unloadRadius: 1 }, { + get: async (x, z, signal) => { + if (x === 0 && z === 0) return chunk(x, z); + const gate = deferred(); + reads.push({ x, z, gate, signal }); + return gate.promise; + }, + placeChunks: async () => { throw error; } + }); + t.teardown(() => reads.forEach(({ x, z, gate }) => gate.resolve(chunk(x, z)))); + await t.throwsAsync(setup.streamer.update(0, 0), { is: error }); + t.is(reads.length, 3); + t.true(reads.every(read => read.signal?.aborted)); + t.deepEqual(setup.bulkCalls, [[{ x: 0, z: 0 }]]); + t.deepEqual([...setup.resident], []); + let disposed = false; + const dispose = setup.streamer.dispose().then(() => { disposed = true; }); + await new Promise(resolve => setImmediate(resolve)); + t.false(disposed); + for (const { x, z, gate } of reads) gate.resolve(chunk(x, z)); + await dispose; + t.true(disposed); + t.deepEqual(setup.streamer.loadedChunks, []); + t.deepEqual(setup.streamer.failedChunks, []); + t.is(setup.placements.length, 1); + t.is(setup.peakMutations(), 1); +}); + +test("retrying after placement failure keeps aborted reads bounded and discards their late in-area results", async t => { + t.timeout(3000); + const error = new Error("first placement failed"), placing = deferred(), release = deferred(); + const reads: { x: number; z: number; gate: ReturnType>; signal?: AbortSignal }[] = []; + let firstPass = true, firstRetry = true; + const setup = fixture({ loadRadius: 1, unloadRadius: 1 }, { + get: async (x, z, signal) => { + if (!firstPass || x === 0 && z === 0) return chunk(x, z); + const gate = deferred(); + reads.push({ x, z, gate, signal }); + return gate.promise; + }, + placeChunks: async () => { + if (firstPass) throw error; + if (firstRetry) { + firstRetry = false; + placing.resolve(); + await release.promise; + } + } + }); + t.teardown(() => { reads.forEach(({ x, z, gate }) => gate.resolve(chunk(x, z))); release.resolve(); }); + await t.throwsAsync(setup.streamer.update(0, 0), { is: error }); + t.is(reads.length, 3); + t.true(reads.every(read => read.signal?.aborted)); + firstPass = false; + const retry = setup.streamer.update(0, 0); + await placing.promise; + t.deepEqual(setup.placements, [[{ x: 0, z: 0 }], [{ x: 0, z: 0 }]]); + t.is(setup.events.filter(event => event.startsWith("get:")).length, 5); + t.is(setup.peakReads(), 4); + for (const { x, z, gate } of reads) gate.resolve(chunk(x, z)); + release.resolve(); + await retry; + for (const { x, z } of reads) t.is(setup.events.filter(event => event === `get:${x},${z}`).length, 2); + t.is(setup.events.filter(event => event.startsWith("get:")).length, 13); + t.is(setup.placements.slice(1).flat().length, 9); + t.is(setup.streamer.loadedChunks.length, 9); + t.is(setup.resident.size, 9); + t.deepEqual(setup.streamer.failedChunks, []); + t.is(setup.peakReads(), 4); + t.is(setup.peakMutations(), 1); + await setup.streamer.dispose(); +});