Skip to content

mesh export: cut define-phase cost and memory on fragment-heavy meshes - #1118

Merged
howetuft merged 10 commits into
LuxCoreRender:mainfrom
howetuft:perf/export-mem-cpu
Nov 13, 2026
Merged

howetuft merged 10 commits into
LuxCoreRender:mainfrom
howetuft:perf/export-mem-cpu

Conversation

@howetuft

Copy link
Copy Markdown
Contributor

Measured on HALL_BENCH (46 objects, 3589 submeshes): export wall 22.1s -> 12.7s, peak RSS 12.6GB -> 4.7GB, define phase 12.6s -> 1.2s.

  1. Bake the object transform in numpy during compute (points + pre-rotated loop normals) instead of passing transformation to DefineMeshExt. Previously the C++ side deep-copied + rotated the normals of every submesh sharing the point buffer in the serial define phase (~3.1ms per 157k-point submesh; 736 submeshes = ~7s + ~1.4GB). The new normals_pretransformed flag (LuxCore PR Blender 3.1.2 linux (Ubuntu 22.10) crashes when entering material or render view #766) skips that work. Mirror transforms (det < 0) keep the C++ path.
  2. Compact submeshes to their used points for objects without a subdivision shape (local points + remapped triangles + compact attr arrays). Removes the full-object-size per-submesh fallback attribute arrays (~2.2GB of numpy allocations on the 736-submesh mint object). Subdivided objects keep the shared point buffer required by the C++ subdiv envelope.
  3. Per-object in-flight window (BLC_EXPORT_INFLIGHT, default 4) instead of 16-object chunks, bounding memory of extracted-but-undefined meshes.

Also sets crease_pairs only on the first submesh of an object (identical for all submeshes; the subdiv envelope does not read per-submesh creases).

Correctness: PLY dumps of all 736 mint submeshes identical between compact and forced-shared paths (triangles/normals/UVs); DanishMood raw film vs the 2.11.2 PyPI wheel matches at noise level with PhotonGI off (mean|d|=0.009) and at reference brightness with it on.

Companion LuxCore change: PR LuxCoreRender/LuxCore#766 (normals_pretransformed flag).

howetuft and others added 10 commits November 12, 2026 20:09
Three changes, measured on HALL_BENCH (46 objects, 3589 submeshes):
export wall 22.1s -> 12.7s, peak RSS 12.6GB -> 4.7GB, define phase
12.6s -> 1.2s.

1. Bake the object transform into the welded points and pre-rotate the
   loop normals in numpy during the parallel compute phase, instead of
   passing 'transformation' to DefineMeshExt. Previously the C++ side
   deep-copied + rotated the normals of EVERY submesh sharing the point
   buffer in the serial define phase (~3.1ms per 157k-point submesh).
   The new 'normals_pretransformed' flag (LuxCore) skips that work.
   Negative-determinant (mirror) transforms keep the C++ path.

2. Compact each submesh to the welded points it actually uses (local
   point array + remapped triangles + compact attribute arrays) for
   objects without a subdivision shape. Subdivided objects keep the
   shared point buffer, which the C++ subdiv envelope needs. This
   removes the full-object-size per-submesh fallback attribute arrays
   (~2.2GB of numpy allocations on the 736-submesh mint object).

3. Replace the 16-object chunk pipeline with a per-object in-flight
   window (BLC_EXPORT_INFLIGHT, default 4), bounding the memory held by
   extracted-but-not-yet-defined meshes.

Also: set crease_pairs only on the first submesh of an object (they are
identical for all submeshes sharing the point buffer, and the subdiv
envelope does not read per-submesh creases).

Correctness: PLY dumps of all 736 mint submeshes are identical between
the compact and forced-shared paths (triangles, normals, UVs);
DanishMood raw film vs the 2.11.2 wheel matches at noise level with
PhotonGI off (mean|d|=0.009) and at reference brightness with it on.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
…t color arrays

- Triangles are grouped by (material, island) with a single argsort +
  boundary split when there are many groups (previously one full scan of
  tri_group per group, O(groups x tris) on fragment-heavy meshes).
- Corner color rgb/alpha arrays are made contiguous once per object and
  POINT-domain colors are indexed from the pre-welded arrays, instead of
  re-slicing per submesh.
- BLC_TIME_STAGES=1 prints per-stage timings of compute_submeshes.

Verified: PLY dumps of all 736 submeshes of the fragment-heavy mint
object are triangle-identical between the compact and forced-shared
paths.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
…lementation

compute_submeshes() now calls pyluxcore.compute_submeshes_c() when the
loaded pyluxcore build provides it (the whole weld/islands/grouping/
gather pipeline runs in C++ with the GIL released, returning zero-copy
numpy arrays), and falls back to the previous numpy implementation
(_compute_submeshes_py) when the symbol is missing or BLC_COMPUTE=py.

The Python orchestration (extract -> compute -> define pipeline,
payload dict shape, info stats) is unchanged.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
…E_LOCK

pyluxcore now serializes scene mesh mutations internally (recursive
mesh-def mutex in slg::Scene) and releases the GIL around the C++ calls,
so DefineMeshExt/SetMeshTriangleAOV are safe to call from worker threads.

convert_objects() now runs compute_submeshes() + define_submeshes() fused
in the thread pool: the calling thread only does the Blender-API extraction
and collects results, so the serial define phase in the main thread is gone.
The in-flight window (BLC_EXPORT_INFLIGHT) now bounds extract+compute+define.
_Scene_lock is removed; define_submeshes() no longer takes it.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
…g one per occurrence

Packed images used by many textures/materials produced one log line per
reuse (e.g. 110 lines for the same file in DanishMood). The skips are now
counted per image and reported once at the end of the export (and in
cleanup), with the number of times each image was skipped.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
The 'Exporting <mesh> - N submesh(es)' block was printed from the worker
threads, so the lines of different meshes interleaved. Workers now buffer
their mesh's log lines and the main thread prints them at collect time, in
job order, so each mesh's block stays together.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
The separator now shows the elapsed wall time since the start of the
current export pass (e.g. '[BLC] (wall 3.456s)'), so the per-mesh blocks
form a readable timeline. convert_objects resets the reference at each
export pass; the serial convert() path sets it if unset.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Add log()/log_str() helpers in utils/log.py (wall time in seconds since
the addon was loaded, same idea as the [SDL][t] timestamps of LuxCore)
and route all [BLC] print() calls through them: luxloader, utils
register/unregister, viewport view_update timings, image export (incl.
the already-exported summary) and mesh export. In the parallel exporter,
worker threads buffer their mesh's messages already formatted (via
log_str) so the per-mesh blocks keep their timestamps while staying
grouped when printed from the main thread. The buffer parameter was
renamed to log_lines to not shadow the log() function.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
LuxCore timestamps its [SDL] messages as WallClockTime() - lcInitTime,
where lcInitTime is the wall-clock time of the luxcore::Init() call
(src/luxcore/luxcoreinit.cpp). utils.log now exposes reset_t0(), called
in register() right before pyluxcore.Init(), so [BLC][t] and [SDL][t]
share the same reference and interleaved log lines line up (previously
[BLC] used the addon-import time, drifting by the wheel-install time).

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
first_run() now logs 'Exporting meshes' when the mesh export begins and
'Mesh export finished in <t>s' (wall time, [BLC][t] format) when it
completes, so the per-mesh blocks are framed by clear boundaries.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
@howetuft
howetuft merged commit c38cf97 into LuxCoreRender:main Nov 13, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant