require [group], a require after the walk, the JIT's emitter-free cache hit, and per-program symbol tables - #3982
Merged
Merged
Conversation
…ered under it: a descriptor's register_module_group(group, member) is a manifest grp row (format 3) replayed like the path rows, a C++ module registers from its constructor, the text collector and the parser expand the same list with the group's guard and public on each member, and an unregistered group adds nothing; the module-cache record header carries the requires a parse took (serializer 206) and the reader cuts the stream off with 'require set changed' when the collector's answer for the file differs, so a member joining a group or a guard a build flips re-parses the requirer instead of serving its old dependency set; tree-sitter grammar, corpus and the grammar canary carry the form
…om a macro, a simulate macro or a running script it compiles a shared module and its prerequisites into the process under the caller's policies, or answers the one already there, and the module stays out of the caller's scope; find_macro_context gives its macro context a mutex for invoke_in_context, and daslib/cross_context wraps the pair (require_module_now, macro_context_of, call_in_context with the result through a pointer); the late walk keeps a module cache of its own beside the host's - the host's is finished before the program simulates - keyed like the host's, so a warm run serves the late modules too
…s the emitter-free half of the driver - the resolved options, the candidate set, the DLL key folded over the candidates, the probe, the extern-slot resolver that asks the DLL instead of the intrinsic table, and the sim-node install - llvm_jit_link is the entry that binds a hit from it alone and brings llvm_macro in by name through require_module_now for a miss and every other artifact, and llvm_macro is the emitter's context reached only by name; a warm -jit hello world goes from 0.20 s to 0.08 s and its compile from 0.17 s to 0.065 s; requireModuleNow restores every shared function's and global's used flag after its walk, since the calling program may be mid-JIT reading them; the exe sweep skips llvm_jit_link as it skips llvm_macro, a codegen failure drains the accumulators before the next program in the process, and the emitter pin gains the two files (codegen version 0x78)
…move into the folder documents - src/ast/ARCHITECTURE.md gets the path guard and the skip-without-resolving boundary of a guarded require, the null-plus-logs answer and the used-flag restore of the late require; src/ast/REVIEW.md the guard-decision parity with the text collector and the freeLateModuleCaches placement; daslib/ARCHITECTURE.md the [export] pair a by-name call into a macro context rests on; daslib/REVIEW.md the no-callback rule of call_in_context; modules/dasLLVM/REVIEW.md the externs-only rule of the jit finalizer path - and the sources keep one-line pointers; the cross_context stdlib page carries the caller's contract, the manifest row legend gets its false: label back, and the emitter pin is recomputed over the trimmed sources
…up("group", "entry", args...) expands at compile time to one member::entry(args...) call per registered member - the registry hands the members back sorted by path, since the scan registers in readdir order, which no platform promises - and the module_cache group test grows the arm; sql_boost requires [sql_provider] and calls register_provider on every member instead of naming sqlite, duckdb and postgres behind guards and static_ifs, sqlite_provider defines the entry and dasSQLITE joins the group from its descriptor and from initMain; linq_fold requires [linq_fold_source], linq_fold_common gains the recognizer registry a member fills from register_linq_fold_source, and pugixml's linq_fold_xml joins with its xml recognizer; llvm_user_modules requires [llvm_code_generator] and calls register_llvm_code_generators on every member, which dasllama_gemm_register now defines; the rtti module gains module_group_for_each_member, the broken late-require fixture carries its lint-skip header, and the emitter pin follows llvm_user_modules
…s in that program's context, leave Function and Variable and live on Program - usedFunctions and usedVariables as sets, functionIndices and variableIndices as maps, behind isUsed, setUsed, indexOf and setIndex (-1 never allocated, -2 allocated and unused): a shared module's objects are one instance for every program in the process, so a compile nested inside another - a macro's compile, a late require, the folding program - used to overwrite the outer program's answers mid-simulate, and requireModuleNow's snapshot of every used flag goes away with the cause; the symbol-use pass, the stack allocator, simulate, alias derivation, the constant folder and the global-order lint read the tables, the lint keeps its initialized set locally instead of the -3 sentinel, eval_single_expression's bare context is handed the compiling program and the simulate visitor answers -1 for a context with none; the das side reads is_used, function_index and variable_index from the ast module (raw and ProgramPtr overloads) in aot_cpp, aot_standalone, validate_code, the mcp aot subtool and the JIT plan, exe, dll and emitter files, Function and Variable lose their index fields and the used bit of FunctionFlags and VariableFlags, the serializer streams neither (207), and a C++ test compiles two programs over one shared module and checks each program's answers stand after the other's compile
…tune and llvm/daslib/llvm_code under tune_framework - from the llvm witness module's constructor for a static host, from the descriptor where the witness's shared_module was built - so membership follows the configure the way the ?llvm guard did, and daslib/tune, dasLLAMA's five framework requirers, the dasllama server and das2rst spell require [tune_framework] instead of the guard; the style lint reads a group require as one unit - a used member keeps the line, a member used through its re-export never draws STYLE029, and an unused group is one STYLE030 naming its members - with the module_cache group test growing the arm that lints both shapes
… to 0x79 - master's 0x78 and this branch's met at one number for two emitters - and the emitter pin follows the merged sources
…bles and the tune framework group: the facts the added comments carried move into the documents - src/ast/ARCHITECTURE.md sec.4 gets the eval context's program and the no-program answer, daslib/ARCHITECTURE_LINT.md a cited sec.4.1 for the group-aware STYLE029/030 walks, modules/dasLLVM/REVIEW.md the ban on naming a generator package in llvm_user_modules, the JIT-entry doc the 0x79 the rebase moved it to, the dasLLAMA engine charter and the dasllama-server checklist the group spelling their guards became, ARCHITECTURE_LINQ and the dynamic-modules skill the constructor half of a group join, the comment-hygiene skill the C++ section pointer as the kept citation form - the sources keep one-line pointers and the doc comments das2rst prints, two stale test-side comments go, the serializer ledger reads in the present tense, and the emitter pin follows the trimmed wiring doc
… is a registry key and its member a require path, neither a module file - and the jit finalizer rule names what it bans: a jittable function on the path, or a helper passing a block to C++
…the shared_module's presence, not on a load answer that a deferred sibling also nulls; the tune framework and the [llvm_code] annotation are two groups, so a requirer of the framework alone re-exports no annotation; a context simulating outside any program resolves a callee through its own table by mangled-name hash instead of a slot it cannot have; the late module caches and the group registry are process-wide and go with the last environment, not the first; the module-cache record stamps the collector's own answer for the file, so writer and reader compare one function's output and a host's custom require no longer cuts the stream on every warm read; the style lint tells a group require by the [group] spelling on its source line, so a one-member group is a group; the JIT-entry doc drops the restored-flags sentence the tables made false, and the generated unit_test include keeps master's line endings
…ister_module_group(group, member, guard) is the require ?guard member a hand would write, evaluated live wherever the list is read, so the dasSQLITE, dasPUGIXML and dasLLVM rows join only where the build has the C++ module (manifest format 4; the fexist gate goes, since a manifest replays a descriptor's rows without seeing the build's artifacts) and the tune framework is one group again; the group registry lives as long as the process, like the native paths, since a descriptor registers once; a late require has no module cache of its own - a stream holds many compiles in order, shared modules written once, so the late modules are records of the compile that issued the require, read from the bound serializer and written back from its cursor, the hosts keep their cache armed through simulate for the simulate macro's and [init]'s late requires, and the late cache machinery, its environment fields and the late~ files go; a late require is a compile's - the wrapper and the builtin refuse a running script, a null access is the compiling program's own (Program::access), a target is named by its file, the debugger switches serialization off as for the host, an RAII scope restores what the walk rebinds, and Program::simulate keeps the program bound and isSimulating set through its simulate macros; every macro context gets its mutex at creation and drops its program binding after its simulate; the index tables clear per allocation and a null program throws from the four symbol builtins; the JIT's hit path calls a DLL that binds nothing a miss, answers an empty candidate set without the emitter, hands unresolved externs to the emitter instead of a sticky panic, times the emitter's load and logs one miss line; the collector reads require[group] with no space, STYLE030 tells a group require by its bracket after the keyword, the extern resolver builds its messages only on failure, the alias walk tests its cheap bits first, the simulate loops look each slot up once, and the tests cover the guarded row, the no-space spelling, the misnamed late module, the running-script panic, the host-stream records and the pinned DLL another program wrote
…acro during a parse, a macro module's [init] during its record's read - hides the host's stream for its duration (LateRequireEnvScope, the reader's readingRecord), parses its modules from source and pushes no record, since the stream is read at a parse's start and written at its end and a nested walk's records would land ahead of the requirer's or be read mid-record, and only a walk after the compile - the JIT's simulate macro, an [init] of the program - reads and writes the stream; a module whose parse failed pushes no record; a relative late target and a member row's path guard resolve against the compiling file in every reader; the collector reads a newline inside require [...] as the parser does; the JIT's miss path prints the link side's plan and probe times with the emitter's load, and a program with nothing to jit says so without the emitter; call_module_group binds an argument that is not a constant or a plain variable to a local ahead of the calls, so it is evaluated once; the CLI finishes its module cache on every path; the handmade FunctionFlags and VariableFlags pages drop the used line the fields lost, the find_macro_context page names where the mutex comes from, the portable tier plan names the group spelling, the generated unit_test include keeps master's line endings, the require_group_name node gets its highlight, and the emitter pin follows; tests cover the nested walk's two warm runs and a failed compile keeping the cache, the block form refused from a running script, a fixture with no macro context, a path-guarded member in the walk, the parse and the calls, a member registered twice, the name on its own line, a side-effecting argument, a call without its entry name, an empty member row, a damaged grp row, the [no_jit] function a DLL holds, the empty candidate set, and the tune_framework group carrying its member; the checklists the round audited say what they meant - every guard site named, the pin rule as a ban on weakening, the finalizer rule around [no_jit], the dasLLAMA region rule in three, the record-header version duty beside the manifest's
…on steps to 0x7a - master's 0x79 and this branch's met at one number for two emitters - and the emitter pin follows the merged sources
…_standalone marker beside a module's CMakeLists.txt makes the root build skip the module, as the root CMakeLists.txt reads it, so its examples and tutorials have no artifact to require and are the box's, not the sweep's
…twelve REVIEW and skill files restored whole, and in src/builtin/REVIEW.md, daslib/REVIEW_LINQ.md and the dasLLAMA engine charter only the rules the arc's mechanisms changed stay
…twice: pinvoke_named asks findFunction for uniqueness and throws on a second function of the name instead of taking the first slot and checking only its count, the late fixture grows two late_pick overloads the test calls once to see the refusal, the cross_context and invoke_in_context docs say a name is held once, and the review ledger loses the row - it also gains the uncached nested late require and the grp-row agreement check as follow-ups, and its heading and date read as written on the day
…unction: Program gains jitSelected with isJitSelected/setJitSelected/clearJitSelection and the ast module is_jit_selected/set_jit_selected/clear_jit_selection, mark_jit_selection writes the table instead of stamping requestJit across every function, the emitter visitor and the extern resolver carry the program and read the table, requestJit is the [jit] annotation alone - the plan's candidate gate and AOT's skip keep reading it - and the emitter pin follows; tests/ast/test_jit_selection pins the table against two programs and a null holder, the symbol-state C++ test gains the selection, the handmade stubs the arc's signature changes orphaned move to the keys the generator uses, and the ledger drops the row
…the hello-world timings it carried are the ledger's
…aslib/just_in_time - since llvm_macro is the emitter now and carries no simulate macro; the whole-tree lint read its old require as unused and the run form failed its verify
…context under an unsafe the caller no longer wrote, with a reference parameter for the pointer and nothing the call does not check itself, so the JIT link module, the late-require fixture and the test call invoke_in_context under their own unsafe; the daslib checklist bans a wrapper of that shape, the docs name the builtin, and the emitter pin follows the link module
…hat freed an in-memory engine through a private helper sits inside free_jit_context, so a jitted program carries the finalizer and nothing else of the link module - the count hello world had before the split - and the emitter pin follows
…o 0x7d - 0x7a went to the lambda reinterpret and 0x7c to the debug-info rail while this branch's candidate-set key sat on 0x7a - the JIT report carries the --jit-debug tail on the link module's hit path as it does on the emitter's, and the emitter pin follows the merged sources
…hitespace, as the parser does - a tab there made every member private in the walk while the parse re-exported them; the group test's public arm gains the tab spelling
…anions named: the entry paragraph and the key paragraph say the same in fewer lines
…ess group, and the AOT header declares require_module_now and has_module so a test calling them compiles under AOT
…he message only where it survives - AOT-compiled code need not carry the text through last_exception - so the test is green under the interpreter, the JIT and AOT alike
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
It introduces a wide ABI/API shift plus substantial compiler/runtime/JIT/module-cache changes across many subsystems, so it warrants final human verification despite limited concrete review findings.
Pull request overview
This PR restructures daslang’s module loading and JIT pipeline around (1) module groups (require [group]) and (2) per-Program symbol state, while also splitting the JIT driver so a warm DLL cache hit avoids loading the LLVM emitter context.
Changes:
- Add
require [group]+ descriptor/C++ registration (register_module_group/registerModuleGroupMember) and a macro helpercall_module_group(...). - Move
used/index/JIT-selection state off sharedFunction/Variableand intoProgram, updating APIs, serialization, and call sites. - Split JIT entry/link/plan/emitter so cache hits bind without loading the emitter; update module-cache/manifest versions and tests/docs accordingly.
File summaries
| File | Description |
|---|---|
| utils/mcp/subtools/aot.das | Use per-program is_used for AOT collection |
| utils/internal/preflight/config.das | Exclude standalone module trees from sweep |
| utils/dasllama-server/REVIEW.md | Update conditional-require rule for groups |
| utils/dasllama-server/openai_server.das | Switch tune dependency to tune_framework group |
| utils/dasllama-server/main.das | Switch tune dependency to tune_framework group |
| utils/daslang/main.cpp | Finish module-cache after simulate (late-require support) |
| utils/daslang-live/main.cpp | Finish module-cache after simulate (late-require support) |
| tree-sitter-daslang/test/corpus/declarations.txt | Add corpus coverage for group require |
| tree-sitter-daslang/src/node-types.json | Add require_group_name node + fields |
| tree-sitter-daslang/src/grammar.json | Extend grammar for require [group] |
| tree-sitter-daslang/queries/highlights.scm | Highlight group names like modules |
| tree-sitter-daslang/grammar.js | Implement JS grammar for group require |
| tests/module_cache/test_descriptor_manifest.das | Manifest version bump + grp-row validation |
| tests/module_cache/REVIEW.md | Tighten assertions/trim rules wording |
| tests/module_cache/ARCHITECTURE.md | Document new group + late-require tests |
| tests/module_cache/_fixtures/mc_late_plain.das | Late-require fixture module |
| tests/module_cache/_fixtures/mc_late_nomacro.das | Late-require “no macro context” fixture |
| tests/module_cache/_fixtures/mc_late_named.das | Late-require “wrong module name” fixture |
| tests/module_cache/_fixtures/mc_late_mod.das | Late-require target module (exports) |
| tests/module_cache/_fixtures/mc_late_macro_user.das | Late-require from call-macro fixture |
| tests/module_cache/_fixtures/mc_late_broken.das | Late-require compile-failure fixture |
| tests/language/optional_require.das | Adjust optional-require commentary |
| tests/daslib/test_tune_shells.das | Assert tune group brings framework when present |
| tests/ast/test_jit_selection.das | New per-program JIT-selection tests |
| tests/ast/_fixture_jit_selection.das | Fixture for JIT-selection tests |
| tests-cpp/small/test_symbol_state.cpp | New C++ test: per-program used/index/JIT state |
| tests-cpp/small/test_symbol_state_shared.das | Shared-module fixture for symbol-state test |
| tests-cpp/small/test_symbol_state_b.das | Program B fixture |
| tests-cpp/small/test_symbol_state_a.das | Program A fixture |
| tests-cpp/small/test_jit_emitter_pin.cpp | Include new JIT plan/link files in emitter hash set |
| src/parser/parser_impl.h | Add ast_requireModuleGroup API |
| src/parser/parser_impl.cpp | Implement guard logic + group require expansion |
| src/parser/ds2_parser.ypp | Parse require [group] (with optional guard/public) |
| src/builtin/REVIEW.md | Update cache/version + codegen-version checklist refs |
| src/builtin/module_builtin_rtti.cpp | Expose group member iteration + guard evaluation |
| src/builtin/module_builtin_fio.cpp | Add register_module_group + manifest recording |
| src/builtin/module_builtin_debugger.cpp | Reject ambiguous by-name pinvoke resolution |
| src/builtin/module_builtin_ast_serialize.cpp | Stop serializing function/variable indices |
| src/builtin/module_builtin_ast_flags.cpp | Remove used bits from Function/Variable flags |
| src/builtin/module_builtin_ast_annotations_1.cpp | Remove bound index fields from managed types |
| src/ast/REVIEW.md | Add require-spelling consistency checklist |
| src/ast/dyn_modules.cpp | Manifest v4 + replay grp rows |
| src/ast/ast.cpp | Remove Function::index initialization |
| src/ast/ast_module.cpp | Implement process-wide module groups registry |
| src/ast/ast_lint.cpp | Track initialized globals without Variable::index |
| src/ast/ast_derive_alias.cpp | Use Program::isUsed instead of shared flags |
| src/ast/ast_const_folding.cpp | Use Program::indexOf instead of func->index |
| skills/xml.md | Update linq-fold XML source wiring to group model |
| skills/tune.md | Update framework reachability to group model |
| skills/linq_fold_patterns.md | Update XML fold source description for groups |
| skills/internal/cpp_codebase_notes.md | Document new per-program symbol APIs |
| skills/dynamic_modules.md | Document register_module_group + usage |
| skills/daslang/references/modules-and-stdlib.md | Document require [group] syntax/behavior |
| skills/das_macros.md | Document group require + late-require cross-context |
| skills/comment_style_hygiene.md | Allow C++ section-pointer comment form |
| plans/jit_compile_time.md | Add follow-up ledger for remaining JIT time |
| modules/dasUnitTest/unit_test.das.inc | Regenerated embedded script blob |
| modules/dasSQLITE/src/dasSQLITE.main.cpp | Join sql_provider group (C++ side) |
| modules/dasSQLITE/PROVIDER_CONTRACT.md | Update provider wiring contract to groups |
| modules/dasSQLITE/daslib/sqlite_provider.das | Rename provider entry to register_provider |
| modules/dasSQLITE/.das_module | Register sql_provider group member (guarded) |
| modules/dasPUGIXML/src/dasPUGIXML.cpp | Join linq_fold_source group (C++ side) |
| modules/dasPUGIXML/daslib/linq_fold_xml.das | Add register_linq_fold_source group entry |
| modules/dasPUGIXML/.das_module | Register linq_fold_source member (guarded) |
| modules/dasLLVM/src/dasLLVM.cpp | Join tune_framework group (C++ side) |
| modules/dasLLVM/REVIEW.md | Update JIT driver/checklist rules for new split |
| modules/dasLLVM/examples/hello_jit.das | Switch JIT opt-in to daslib/just_in_time |
| modules/dasLLVM/daslib/llvm_user_modules.das | Replace hard-coded requires with group dispatch |
| modules/dasLLVM/daslib/llvm_macro.das | Convert to emitter macro-context entry module |
| modules/dasLLVM/daslib/llvm_jit_common.das | Pull plan module, remove moved helpers |
| modules/dasLLVM/daslib/llvm_exe.das | Use per-program used/index/JIT selection APIs |
| modules/dasLLVM/daslib/llvm_dll_utils.das | Use per-program used flags in UID sweep |
| modules/dasLLVM/ARCHITECTURE.md | Update pipeline + key description for split driver |
| modules/dasLLVM/ARCHITECTURE_JIT_ENTRY.md | New doc: entry module + emitter-free cache hit |
| modules/dasLLVM/.das_module | Install new plan/link modules + group registration |
| modules/dasLLAMA/portable_tier_plan.md | Update tuning/framework guard narrative |
| modules/dasLLAMA/performance/profile_common.das | Switch to tune_framework group |
| modules/dasLLAMA/dasllama/dasllama_tune.das | Switch to tune_framework group |
| modules/dasLLAMA/dasllama/dasllama_tune_scope.das | Switch to tune_framework group |
| modules/dasLLAMA/dasllama/dasllama_math_gen.das | Switch to tune_framework group + doc updates |
| modules/dasLLAMA/dasllama/dasllama_gemm_register.das | Rename entry to register_llvm_code_generators |
| modules/dasLLAMA/dasllama/dasllama_fat_start.das | Switch to tune_framework group |
| modules/dasLLAMA/dasllama/dasllama_common.das | Switch to tune_framework group (public) |
| modules/dasLLAMA/benchmarks/lcpp_bench.das | Switch to tune_framework group |
| modules/dasLLAMA/ARCHITECTURE_ENGINE.md | Update tuning/framework reachability wording |
| modules/dasLLAMA/.das_module | Join llvm_code_generator group |
| modules/dasImgui/tests/test_grammar_canary.das | Add group-require grammar canary snippet |
| include/daScript/simulate/aot_builtin_rtti.h | Add AOT prototypes for new rtti/late-require APIs |
| include/daScript/simulate/aot_builtin_ast.h | Add AOT prototypes for per-program symbol APIs |
| include/daScript/ast/REVIEW.md | Add checklist: per-program state belongs on Program |
| include/daScript/ast/dyn_modules.h | Add manifest row kind + replay API for groups |
| include/daScript/ast/ast.h | Add module group APIs + Program symbol tables |
| include/daScript/ast/ast_serializer.h | Add module-cache bookkeeping + bump stream version |
| doc/source/stdlib/sec_ast.rst | Include generated docs for new stdlib modules |
| doc/source/stdlib/introduction.rst | Mention cross_context + module_group modules |
| doc/source/stdlib/handmade/typedef-ast-VariableFlags.rst | Remove obsolete used flag doc |
| doc/source/stdlib/handmade/typedef-ast-FunctionFlags.rst | Remove obsolete used flag doc |
| doc/source/stdlib/handmade/module-module_group.rst | New stdlib doc stub for module_group |
| doc/source/stdlib/handmade/module-cross_context.rst | New stdlib doc stub for cross_context |
| doc/source/stdlib/handmade/function-rtti-module_group_for_each_member-0x64b0c8812c8d58d6.rst | Doc new rtti helper |
| doc/source/stdlib/handmade/function-fio-register_module_group-0xbfd5bb5408ae846e.rst | Doc new descriptor builtin |
| doc/source/stdlib/handmade/function-debugapi-try_invoke_in_context-0x3b138017b43557c8.rst | Document ambiguous-name rejection |
| doc/source/stdlib/handmade/function-debugapi-invoke_in_context-0xdc8aad5ea68674bf.rst | Document ambiguous-name rejection |
| doc/source/stdlib/handmade/function-debugapi-invoke_in_context-0x6a037dd8b7986835.rst | Document ambiguous-name rejection |
| doc/source/stdlib/handmade/function-debugapi-invoke_in_context-0x19a0ddf922c9ad95.rst | Document ambiguous-name rejection |
| doc/source/stdlib/handmade/function-ast-variable_index-0xbf5517480aaeffb1.rst | Doc new per-program variable index query |
| doc/source/stdlib/handmade/function-ast-set_jit_selected-0xe95e6598b9f2772b.rst | Doc new per-program JIT selection setter |
| doc/source/stdlib/handmade/function-ast-set_jit_selected-0xb4bb112e5f39b93c.rst | Doc ProgramPtr overload |
| doc/source/stdlib/handmade/function-ast-require_module_now-0x3321f96caa45428c.rst | Doc late-require builtin |
| doc/source/stdlib/handmade/function-ast-is_used-0x5ff424eace25cf4c.rst | Doc per-program used query |
| doc/source/stdlib/handmade/function-ast-is_jit_selected-0x3081601f8756204d.rst | Doc per-program JIT selection query |
| doc/source/stdlib/handmade/function-ast-has_macro_context-0x4f257a25fcd331a4.rst | Doc macro-context predicate |
| doc/source/stdlib/handmade/function-ast-function_index-0x8093f578250f3115.rst | Doc per-program function index query |
| doc/source/stdlib/handmade/function-ast-find_macro_context-0x598d596c1e881f4f.rst | Doc macro-context accessor |
| doc/source/stdlib/handmade/function-ast-clear_jit_selection-0xe0b782a1c6f1586c.rst | Doc ProgramPtr overload |
| doc/source/stdlib/handmade/function-ast-clear_jit_selection-0xd4c94e0272357bbb.rst | Doc selection clear semantics |
| doc/source/reference/language/modules.rst | Add language reference section for module groups |
| doc/source/reference/language/lint.rst | Specify STYLE029/030 behavior for group requires |
| doc/reflections/das2rst.das | Generate docs for new modules/APIs |
| doc/internal/error_audit.csv | Update audited error text after message tweak |
| dastest/review_gate.das | Exclude group rows from descriptor census gate |
| daslib/validate_code.das | Switch validation to per-program is_used |
| daslib/tune.das | Replace conditional requires with tune_framework group |
| daslib/sql_provider.das | Update provider registry commentary for groups |
| daslib/sql_boost.das | Use call_module_group for provider registration |
| daslib/REVIEW.md | Add rule: don’t wrap invoke_in_context to hide unsafe |
| daslib/REVIEW_LINQ.md | Update ordering rule to include group-registered sources |
| daslib/module_group.das | New macro helper for group member calls |
| daslib/linq_fold.das | Load/dispatch fold sources via linq_fold_source group |
| daslib/linq_fold_common.das | Add source recognizer registry types/storage |
| daslib/just_in_time.das | JIT opt-in now requires link-only entry |
| daslib/cross_context.das | New late-require + macro-context helper module |
| daslib/ARCHITECTURE.md | Update tune framework reachability + add cross_context section |
| daslib/ARCHITECTURE_LINT.md | Document STYLE029/030 mechanics for group requires |
| daslib/ARCHITECTURE_LINQ.md | Document registered-source recognizer order |
| daslib/ARCHITECTURE_EMIT.md | Update global init ordering source-of-truth |
| daslib/aot_cpp.das | Use per-program used/index APIs throughout AOT emission |
| CLAUDE.md | Update JIT codegen-version bump reference path |
Review details
- Files reviewed: 152/155 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…ng may carry - require ./foo.das names module foo, so call_module_group's qualified call does too; the group test pins the three spellings
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
It spans a wide ABI-breaking refactor across core compiler/runtime, JIT caching, module resolution, and documentation/tests, so it needs final human verification of cross-platform and embedder impacts.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
utils/internal/preflight/config.das:100
sweep_excludednow excludes any path undermodules/<name>/...whenmodules/<name>/.daspkg_standaloneexists. This is new behavior buttest_sweep_excludeddoesn’t pin it, so a future change could silently re-include standalone module trees (or exclude too much) without a failing test.
Add a focused test case that creates a temporary modules/<name>/.daspkg_standalone marker (or uses an existing fixture) and asserts sweep_excluded("modules/<name>/something.das") == true and that a sibling module without the marker stays false.
- Files reviewed: 152/155 changed files
- Comments generated: 0 new
- Review effort level: Lite
…ess: deserialize_program gains an access overload that sets Program::access, and dastest's --deser run passes the access it serialized with, so a late require from a restored program's [init] walks as it does from a parsed one - CI's ser/deser sweep failed the late-require test with no file access
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
call_module_group currently evaluates side-effecting argument expressions even when the target group is empty, violating the documented “expands to nothing” behavior.
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
daslib/module_group.das:36
call_module_groupis documented as “an empty group to nothing”, but the macro currently emitslet _cmg_arg_* = <expr>bindings for non-const arguments before it iterates members. If the group has zero members, those bindings still run at runtime, so side effects in arguments execute even though no calls are emitted.
Consider short-circuiting when the group is empty (macro-time), before emitting any argument bindings or calls.
utils/internal/preflight/config.das:100
collect_sweep_rootscallssweep_excludedfor every discovered.dasfile. With the newstandalone_module_treefallback, that meansfexist("modules/<mod>/.daspkg_standalone")can run once per file undermodules/…, which is potentially thousands of filesystem probes and can noticeably slow the preflight sweep.
Consider memoizing the per-module result so each module root is checked at most once per process.
- Files reviewed: 154/157 changed files
- Comments generated: 0 new
- Review effort level: Lite
…cmd.exe strips a command line's first and last quotes, which the windows sweep read as a bad path on every child - the executable and the das root ride unquoted as the sibling tests spell them
…what happened - the child exited, with its status, before its summary: a crash, an abort, or a panic outside any test
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
There are at least two concrete correctness hazards in the new code (AOT-mode visitor calling is_jit_selected with a null program, and non-hygienic synthesized names in call_module_group) that can break compilation or runtime behavior.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
daslib/module_group.das:46
call_module_groupbinds non-trivial arguments to synthesized locals named_cmg_arg_<n>. Those names are not hygienic, so multiple expansions on the same scope (or a user variable with the same name) can collide and fail compilation.
Make the synthesized identifier unique per call site (e.g., include expr.at.line/expr.at.column in a backtick-quoted name) before using it in $i(...).
- Files reviewed: 154/157 changed files
- Comments generated: 1
- Review effort level: Lite
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
ast_const_folding.cpp now checks Program::indexOf(...) == -1, but indices are explicitly set to -2 for unused symbols, so the guard never triggers and breaks the intended internal consistency check.
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
src/ast/ast_const_folding.cpp:1681
Program::indexOfnow returns-2for functions the allocation pass saw but marked unused (see ast_allocate_stack.cpp), so this guard (== -1) will never fire and stops protecting against folding a symbol that wasn’t marked used. This effectively disables the intended internal-consistency check.
utils/internal/preflight/config.das:109sweep_excludednow excludes entire module trees when amodules/<name>/.daspkg_standalonemarker exists, but the preflight tests only cover the static prefix list. Adding at least one unit test that creates (or points at) a standalone module marker would prevent regressions in this new exclusion rule.
- Files reviewed: 154/157 changed files
- Comments generated: 0 new
- Review effort level: Lite
… exe's does, so its constructor check reads the program's jit selection instead of a null holder
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core language/toolchain behavior (parser, module cache/manifest, JIT entry/caching, and ABI-visible AST structures), so it needs careful human validation beyond automated review.
Review details
- Files reviewed: 155/158 changed files
- Comments generated: 1
- Review effort level: Lite
…ue to the call site - make_unique_private_name, the repo's spelling for a macro that lands in the caller's scope - and the group test calls beside a local of the plain spelling
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
It introduces a broad ABI break and deep cross-cutting compiler/JIT/module-cache changes whose correctness and compatibility impact is difficult to fully validate from diffs alone.
Review details
- Files reviewed: 155/158 changed files
- Comments generated: 0 new
- Review effort level: Lite
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ABI break:
FunctionandVariableloseindexand theusedbit, andFunction::requestJitis the[jit]annotation alone - a host reading them switches toprogram->indexOf(fn)/program->isUsed(fn)/program->isJitSelected(fn), and das code tofunction_index(prog, fn)/is_used(prog, fn)/is_jit_selected(prog, fn);getModuleGroupMembersanswers{member, guard}records; a by-nameinvoke_in_contextrefuses a name the target context holds twice; a script that opted into the JIT withrequire llvm/daslib/llvm_macrorequiresdaslib/just_in_timenow; the module-cache stream is 207, the descriptor manifest 4 and the JIT codegen version 0x7d, so each cache rebuilds once.Why. An optional dependency was spelled backwards - the module that wants every installed provider named each one behind its own guard - and a warm
-jitrun spent 0.17 s of its 0.20 s reading the LLVM emitter for a DLL already on disk. Fixing the second exposed the first-class bug behind it: a compile nested in another rewrote the outer program'susedflags, because they lived on shared AST objects.What changes.
require [group]expands to every module registered under the name (descriptorregister_module_group(group, member, guard), C++registerModuleGroupMember); a member's guard is whatrequire ?guard memberwould carry, evaluated live wherever the list is read, so a member whose file requires a C++ module joins only where the build has it; the module-cache record stamps the require set and cuts off when it changes.call_module_group("group", "entry", args...)(daslib/module_group) expands to onemember::entry(args...)per member; sql providers, linq_fold sources,[llvm_code]generators and the tune framework are the four groups.require_module_nowcompiles asharedmodule into the process after the walk, from a macro, a simulate macro or an[init]- a running script is refused; its modules are records of the host's module-cache stream, and the CLI and daslang-live keep their cache armed through simulate.daslib/cross_contextwraps it and answers a module's macro context; the call into that context isinvoke_in_contextunder the caller's ownunsafe.llvm_jit_link+llvm_jit_plan); a miss, a DLL binding nothing, or an extern the process cannot fill brings the emitter in by name; an empty candidate set never loads it. A jitted program carries one function of the link module, the finalizer.Program, not onFunction/Variable;Program::simulatekeeps itself bound through its simulate macros; every macro context carries its mutex from creation.invoke_in_contextrefuses a name the target context holds twice - an overload, a same-named function in another module - where the first slot used to win with only its argument count checked.Observable behavior.
require ?duckdb duckdb/duckdb_providerper provider ->require [sql_provider];require ?llvm llvm/daslib/llvm_tune->require [tune_framework]; a build withDAS_SQLITE_DISABLED=ON(the CMake default) still compilessql_boostandlinq_boost.-jithello world 240 ms -> 90 ms; the interpreter stays at 13 ms. A script that ever missed carries the emitter's records in its own module cache (~34 MB).fn.flags.used,fn.index,var.indexin das ->is_used(prog, fn),function_index(prog, fn),variable_index(prog, var); a null program there throws.require llvm/daslib/llvm_macrorequiresdaslib/just_in_timenow (what-jitrequires;llvm/daslib/llvm_jit_linkunderneath) -llvm_macrois the emitter, reached by name, and carries no simulate macro;modules/dasLLVM/examples/hello_jit.dasis the one in-tree case.-jitsimulatesmainandfree_jit_context, as before; nothing of the emitter enters a program's context.Where to look.
include/daScript/ast/ast.h(theProgramsymbol tables,Program::access),src/ast/ast_export.cppandast_allocate_stack.cpp(their writers),src/ast/ast_parse.cpp(requireModuleNow, the collector's group expansion,writebackModules' cursor),src/parser/parser_impl.cpp(the parser's),src/builtin/module_builtin_debugger.cpp(pinvoke_named's uniqueness),modules/dasLLVM/daslib/llvm_jit_plan.dasandllvm_jit_link.das(the hit path; the[no_jit]finalizer stays extern-only),daslib/module_group.das,src/ast/ARCHITECTURE.mdsec.2-4.Validation, claims, ledger
Validation
tests/jit_testssweep (389/390, one skip) - the batch form is what found theused-flag contamination, the exe-sweep skip and the accumulator drain, none of which the isolated CI form shows; tests/module_cache (59), tests/ast (21), tests/jit (21), tests/daslib (293), tests/linq (2071), tests/dasSQLITE (922), tests/language (1730), tests-cpp-small (137); the nightly whole-tree lint's two passes, the standalone-exe build (all_utils_exe, the-exedas-fmt and das-lint), the utils tests (244) and CI'srun_examplestarget (19 of 19) as the extended-checks mirrors; the profelis dascript-plugin's validator (1.3.7, server 0.6.3) on this binary - it readsmarked_usedon globals and no function flag this arc moved, and reports an unused global as before;sql_boost,linq_boostandtunecompiled with the sqlite and pugixml.shared_modulefiles moved aside, cold and replayed, and adaslib/tuneroot answersmodule_exists(llvm_tune)false withdasModuleLLVM.shared_modulemoved aside (cold and replayed) and true once restored; dasDuckDB's sql conformance suite (83) built against this tree with its descriptor on the group; dasPostgreSQL compile-only (no local server).-jitrun).Program::simulatebinding; then the nested-walk rule, the collector's newline, the miss-path timings, the once-evaluated group arguments), three codex rounds - the second and third, over each batch's tip, found nothing with an exec trace covering every changed file - and a ruling pass over the changed documents, which moved the JIT selection off the sharedFunction, made the by-name cross-context call refuse ambiguity, and dropped thecall_in_contextwrappers that hid anunsafe. Every name the new doc pages tell a reader to type was checked against the tree.hello_jit.das's JIT opt-in, fixed here; its five stale-nolint findings are master's (the same five with master's ownstyle_lint.dasunder this binary).publicafter a group's]only past spaces where the parser takes any whitespace - fixed, with the tab spelling in the group test.test_aot, the nightly's andpreflight --full's) compiles every daslib module and every test as C++, and it is where a builtin a das file calls with no declaration in its AOT header shows: the arc'sis_used/function_index/variable_index/ jit-selection builtins,require_module_now, and the upstreamhas_module,has_macro_contextandfind_macro_contextthatcross_context.dasis the first AOT-compiled caller of - all declared inaot_builtin_ast.h/aot_builtin_rtti.hnow. The fast tier's compile-sweep red on this box (three Vulkan example roots, dasVulkan being standalone here) had skipped that lane on the earlier full run, so every skipped lane was run once by name: lint (85 files, both rails), tests-cpp, tests-interp, tests-jit, utils-tests, docs, tests-aot (13315 tests, the one AOT-only red - an assertion onlast_exception's text, which AOT-compiled code need not preserve - rewritten and re-run under AOT in isolation).llvm_jit_debug_info.das, and pristine master fails the same eight in a separate worktree on this box. The cause is not this branch's:LLVMDIBuilderCreateFunctiontakes fourteen arguments, and dasbind's extern wrapper on Apple arm64 lays every stack argument out at eight bytes where the Apple ABI packs 32-bit ones, soIsDefinition,ScopeLineandIsOptimizedare read one slot late and the subprogram comes out as a declaration (the verifier's "scope points into the type hierarchy"). That is a dasbind fix on its own, outside this PR.run.das -- --suite model-free: 72 of 74 files pass;test_vulkan_kernels.dasandtest_vulkan_moe_cm2.dasfail withmissing prerequisite 'vulkan'- this box's configure has nodasModuleVulkantarget, so the two are the same red on master here, not the arc's.bbatkin/sql-provider-group, not yet pushed: their descriptors call the new builtin with the guard, so they land after this PR; until then those two providers are absent fromsql_booston a tree that has them.Claims - stated, not tested
initMainfor the binder-generated dasSQLITE); no in-tree test builds such a host.[init]during its record's read) parses its modules from source and leaves the stream alone; the tested shapes are a call macro's late require (two clean warm runs, a failed compile keeping the cache) and the JIT's simulate-time require. A persistent-serializer host (dagor, dastest--ser) has not run a late require.Not done
LineInfoArg(therequire_module_nowblock form) is not JIT-able -unresolved expression __lineinfo__- and the emitter's content gate does not decline it; the finalizer path avoids it, the gate gap stays.grprows agree with the module constructor's - are ledgered inplans/jit_compile_time.md.llvm_user_moduleskeepsoptions _dasllama_internal: dasLLAMA's lint sees the group's expanded require of its internal module.eval_single_expressionon a caller's context binds the environment's program to it; the function cannot be made correct (globals) and is a deprecation candidate, left as is.sql_boost's provider registration order is now the group's sorted order (duckdb, postgres, sqlite) instead of the hand-written one; the registry's only order-sensitive consumer (readTableSchema) refuses ambiguity rather than picking first.plans/review_md_splits.mdride the docs-only PR at the chain's end; the MCP custom-modules/fixture, the exe form of the supervised tools and the watchdog for both stay in the two ROADMAPs.