Repository navigation
feat(extraction): represent process launches as SPAWNS edges to Process nodes (#158) - #171
Conversation
…ss nodes (octacity-org#158) Process launches (subprocess.run, exec.Command, child_process.spawn, posix_spawn, Process.Start, Command::new, etc.) were either omitted from the graph or bound to project functions of the same name via guessed CALLS edges. Emit SPAWNS edges to Process nodes instead: - internal/cbm/spawn_patterns: per-language launch API table across ~30 languages and program name extraction rules (literal first argument/word or <dynamic>). - src/pipeline/pass_spawns: resolve spawn sites and emit Process nodes (__process__<program>) and SPAWNS edges. Veto heuristic name guesses for launch APIs that do not resolve to project code. - src/pipeline/pass_parallel.zig: integrate spawn site checks into call resolution. - tests/test_spawns.c: test recognition through imports, bare names, program argument extraction, graph representation, and parallel parity. Closes octacity-org#158. Signed-off-by: mertcankaratass <184436995+mertcankaratass@users.noreply.github.com> # Conflicts: # Makefile.cbm # build.zig
tests/test_github_workflows.sh validates that every default suite in test_main.c is assigned to exactly one CI test shard in scripts/ci-test-shard.sh. Signed-off-by: mertcankaratass <184436995+mertcankaratass@users.noreply.github.com>
Remove CBM_LANG_ARKTS, which is not part of this repository's language enum, and drop source_origin field access from CBMCall. Signed-off-by: mertcankaratass <184436995+mertcankaratass@users.noreply.github.com>
Initialize CBMCall.args as array members rather than pointer assignment, call cbm_mcp_server_new() with no arguments, and use cbm_store_open_path_query(). Signed-off-by: mertcankaratass <184436995+mertcankaratass@users.noreply.github.com>
Signed-off-by: mertcankaratass <184436995+mertcankaratass@users.noreply.github.com>
4ce9913 to
449e383
Compare
ata-sesli
left a comment
There was a problem hiding this comment.
Thanks for this. The feature is useful, the launch-API tables are thorough, and the veto on name-guess resolutions is the right approach. A few things need to change before merge.
1. Please write the pipeline part in Zig
src/pipeline/ has been fully ported to Zig: PIPELINE_SRCS in Makefile.cbm was empty before this PR, and the last C pieces (lsp_resolve, the sequential resolver) were removed recently. src/pipeline/pass_spawns.c adds C back to that directory, plus new C prototypes in pipeline_internal.h, the PIPELINE_SRCS entry and the native_sources entry in build.zig.
Please move cbm_pipeline_spawn_site / cbm_pipeline_emit_spawn and their helpers into a Zig module (for example src/pipeline/spawns.zig) that pass_parallel.zig imports with @import, and drop pass_spawns.c, its build entries and the pipeline_internal.h additions. src/pipeline/lsp_resolve.zig is a recent example of the pattern. If tests/test_spawns.c needs to call them from C, export them with pub export fn and keep a small header with only those prototypes, as lsp_resolve.h does.
internal/cbm/spawn_patterns.c can stay in C, since internal/cbm (extraction) is still C.
2. The sequential resolver no longer exists
The sequential resolver (pass_calls.c, pass_definitions.c, pass_usages.c) was deleted in faf79b3, and pass_parallel.c is now pass_parallel.zig. Several places still describe two resolvers:
- the
pass_spawns.cheader ("Shared by both call resolvers (pass_calls.c, pass_parallel.c)"); - the
pipeline_internal.hcomment and the PR description ("sequential vs parallel parity"); spawns_parallel_matches_sequentialintests/test_spawns.c. Both runs now go through the same parallel resolver, so this test cannot catch a divergence. Please remove it, or rename it to say what it does check (for example that the result does not depend on the file count).
Also, the pipeline_internal.h comment says the hook runs on "the registry answer after the cross-language veto". In resolveFileCalls the hook runs before the cross-language guard (cbm_suppress_cross_language_suffix_match). Because suffix_match is treated as a guess, I don't think this changes the output, but the comment should match the code.
3. Bare PHP exec likely matches PDO calls (not reproduced)
CBM_SPAWN_BARE compares only the callee text and does not look at whether the call has a receiver. For PHP member calls the extractor records the bare method name (see the comment above the scoped_call_expression case in extract_calls.c: "every $obj->get() would become a route"). So $pdo->exec("DELETE FROM jobs"), a very common PDO call, would likely match the exec entry. It does not resolve to project code, so it would become a SPAWNS edge to a Process node named DELETE. The same may apply to other bare entries called as methods (Ruby obj.exec, Julia run). I found this by reading the code and did not reproduce it.
Suggested fix: skip CBM_SPAWN_BARE entries for calls that have a receiver, and add a test where $pdo->exec("...") produces no SPAWNS edge.
| @@ -0,0 +1,103 @@ | |||
| /* | |||
There was a problem hiding this comment.
Please write this in Zig (see the review summary): a module under src/pipeline/ imported by pass_parallel.zig, instead of C linked through PIPELINE_SRCS and pipeline_internal.h.
| * pass_spawns.c — SPAWNS edges: a call that starts another program, to the | ||
| * Process node of that program (internal/cbm/spawn_patterns.h). | ||
| * | ||
| * Shared by both call resolvers (pass_calls.c, pass_parallel.c), which ask |
There was a problem hiding this comment.
pass_calls.c and pass_parallel.c no longer exist. The only call resolver is resolveFileCalls in pass_parallel.zig, and it calls this before the cross-language guard, not after it.
|
|
||
| /* The sequential (few files) and parallel (past MIN_FILES_FOR_PARALLEL) | ||
| * resolvers ask at the same point and emit the same SPAWNS edges. */ | ||
| TEST(spawns_parallel_matches_sequential) { |
There was a problem hiding this comment.
Since faf79b3 there is no sequential resolver, so both runs here use the parallel path and this comparison cannot fail for the reason in its name. Please remove it or rename it to what it does check.
| }; | ||
|
|
||
| static const cbm_spawn_api_t php[] = { | ||
| {"exec", NULL, U, 0, -1}, |
There was a problem hiding this comment.
Likely false positive (not reproduced): PHP member calls are extracted with the bare method name, and CBM_SPAWN_BARE does not check for a receiver, so $pdo->exec("DELETE FROM jobs") would become a SPAWNS edge to a Process named DELETE. Consider skipping bare entries for calls with a receiver, and add a test for the PDO case.
d6c98b0 to
5d9a6f8
Compare
…d calls - Move cbm_pipeline_spawn_site and cbm_pipeline_emit_spawn into src/pipeline/spawns.zig - Add src/pipeline/spawns.h with C ABI prototypes and export in src/cbm_zig.zig - Drop src/pipeline/pass_spawns.c, PIPELINE_SRCS in Makefile.cbm, and native_sources entry - Update pass_parallel.zig to import spawns.zig directly - Set call.is_method for PHP member calls and Ruby method calls with receivers - Skip CBM_SPAWN_BARE and CBM_SPAWN_BARE_IMPORTED when call.is_method is true - Rename test spawns_parallel_matches_sequential to spawns_result_invariant_to_file_count - Add PHP regression test ensuring $pdo->exec does not emit a SPAWNS edge Signed-off-by: mertcankaratass <184436995+mertcankaratass@users.noreply.github.com>
5d9a6f8 to
efe06ba
Compare
ata-sesli
left a comment
There was a problem hiding this comment.
Thanks for the quick update. The Zig port is what we asked for: pass_spawns.c is gone, spawns.zig is imported by pass_parallel.zig, the C exports for the tests sit behind a small spawns.h like lsp_resolve, PIPELINE_SRCS is empty again, and checkExports covers the new module. The stale sequential-resolver references are fixed too.
One change from this round needs to come back out, and that is on me.
Correction: the PDO false positive in my previous review did not exist
In point 3 of my previous review I said $pdo->exec("DELETE FROM jobs") would match the bare PHP exec entry. That was wrong. PHP's member_call_expression has an object field, and extract_callee_from_fields joins it to the method name (extract_calls.c, the name + object branch), so the callee text is "$pdo.exec", not "exec". Bare entries only match the exact text, so $pdo->exec(...) never produced a SPAWNS edge, even before this commit. The comment I cited is about Route::get (scoped_call_expression), which has no object field. I misread it. Sorry for the extra work.
The new is_method check disables Ruby launch APIs called with a receiver (found by reading the code, not reproduced)
- The extractor now sets
is_method = truefor every Rubycallwith a receiver. - The Ruby extractor names those calls receiver plus method (
"Open3.capture3","IO.popen"), and the Ruby table lists exactly those spellings asCBM_SPAWN_BAREentries (Kernel.*,Process.*,IO.popen,PTY.spawn,Open3.*). site()now rejects every bare match whenis_methodis set.
So 13 of the 17 Ruby entries can never match: Open3.capture3("ls"), IO.popen("ls"), Kernel.system("make") and Process.spawn("x") no longer produce SPAWNS edges. Only receiver-less system, exec and spawn still work. No test catches this, because the Ruby test (spawn_match_bare_names) calls cbm_spawn_match directly and never reaches site().
For PHP the change has no effect, since the callee text already includes the receiver. The new $pdo->exec pipeline case passes with or without the fix. No other code reads is_method for PHP or Ruby; the existing readers are gated to Perl, Python/JS/TS and Zig.
Requested changes
- Revert the
is_methodadditions ininternal/cbm/extract_calls.c, thecbm.hcomment change and the new comment above the PHP table inspawn_patterns.c. - Remove the
is_methodcheck fromsite()inspawns.zig. Without the extractor change it does nothing. - Keep the
$pdo->execcase inspawns_in_the_graphas a regression guard, and add a Ruby case there (for exampleOpen3.capture3("ls")producing a SPAWNS edge tols) so receiver-form Ruby calls are covered end to end. - Minor: the failure message in
spawns_result_invariant_to_file_countstill printssequential:/parallel:.
|
|
||
| // A bare launch API (e.g. PHP exec, Ruby exec) called as a method on a receiver | ||
| // (such as $pdo->exec("DELETE FROM ...") or obj.exec(...)) is not a process spawn. | ||
| if (call.is_method and (api.*.rule == c.CBM_SPAWN_BARE or api.*.rule == c.CBM_SPAWN_BARE_IMPORTED)) { |
There was a problem hiding this comment.
This rejects the Ruby receiver-form entries (Open3.capture3, IO.popen, Kernel.system, Process.spawn, ...). They are CBM_SPAWN_BARE entries whose callee text already includes the receiver, and the extractor now marks those calls is_method. For PHP $pdo->exec the callee text is "$pdo.exec", which never matched exec anyway, so this check can go. See the review summary.
| strcmp(ts_node_type(node), "nullsafe_member_call_expression") == 0)) { | ||
| call.is_method = true; | ||
| } | ||
| if (ctx->language == CBM_LANG_RUBY && |
There was a problem hiding this comment.
Please revert the PHP and Ruby is_method additions. Ruby: it makes site() drop every receiver-form launch API. PHP: it has no effect, because the callee is already "$obj.method". No other reader of is_method covers PHP or Ruby.
| sp_cleanup(&p); | ||
| ASSERT_TRUE(seq[0] != '\0'); | ||
| if (strcmp(seq, par) != 0) { | ||
| fprintf(stderr, " sequential:\n%s parallel:\n%s", seq, par); |
There was a problem hiding this comment.
Minor: the message still says sequential: / parallel:; something like few files: / many files: would match the new test name.
…orm Ruby
- Revert is_method setting for PHP and Ruby in extract_calls.c and comments in cbm.h and spawn_patterns.c
- Drop is_method check from spawns.zig site() so receiver-form Ruby entries (e.g. Open3.capture3) are recognized
- Keep $pdo->exec regression test and add worker.rb with Open3.capture3("ls") to test_spawns.c
- Update file-count invariance test error message to print "few files: / many files:"
Signed-off-by: mertcankaratass <184436995+mertcankaratass@users.noreply.github.com>
What does this PR do?
Closes #158.
Process launches (
subprocess.run(["git", ...]),exec.Command("kubectl", ...),child_process.spawn,posix_spawn,Process.Start,Command::new, etc.) were previously either invisible in the graph or resolved to project functions of the same name via guessedCALLSedges.This PR represents process launches as
SPAWNSedges from the calling function to aProcessnode named after the program, vetoing heuristic call guesses:Launch API Tables & Program Extraction (
internal/cbm/spawn_patterns.h,spawn_patterns.c):QUALIFIED,BARE,BARE_IMPORTED,STRING_RECEIVER).import subprocess as sp->sp.run)./usr/bin/git status->git), handles literals, and falls back to<dynamic>when not a literal string.Pipeline Integration (
src/pipeline/pass_spawns.c,src/pipeline/pass_parallel.zig):cbm_pipeline_spawn_siteandcbm_pipeline_emit_spawn.unique_name,suffix_match,fuzzy, weaksame_module) for launch APIs that do not resolve to actual project definitions.Processnode (__process__<program>) andSPAWNSedge with call line and confidence.pass_parallel.zigcall resolution loop (resolveFileCalls).Build & Test Wiring (
Makefile.cbm,build.zig,tests/test_spawns.c,tests/test_main.c):spawnstest suite covering recognition through imports, bare names, argument extraction rules, graph nodes/edges, and sequential vs parallel parity.SPAWNStoALL_EDGE_TYPESand updatedREADME.mdgraph data model.Checklist
git commit -s) — required, CI rejectsunsigned commits (DCO, see CONTRIBUTING.md)
make -f Makefile.cbm test)make -f Makefile.cbm lint-ci)