Skip to content

Commit 4a1f6e7

Browse files
committed
JS: Reuse API instance and other changes
1 parent 744dac6 commit 4a1f6e7

7 files changed

Lines changed: 208 additions & 148 deletions

File tree

‎javascript/extractor/lib/typescript/README.md‎

Lines changed: 66 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -46,11 +46,8 @@ run it separately without replacing the wrapper's build/runtime dependency:
4646
npm exec --yes --package=tslint@6.1.3 --package=typescript@5.9.2 -- tslint --project .
4747
```
4848

49-
`SEMMLE_TYPESCRIPT_HOME` still overrides the runtime dependency. It must point to
50-
the installed TypeScript package directory (containing `package.json`), with the
51-
native `typescript/sync` and `typescript/ast` entry points and matching executable.
52-
A legacy 5.x module or a directory from which the bundled package merely happens
53-
to be resolvable is not a valid override.
49+
The wrapper always uses the bundled TypeScript dependency.
50+
`SEMMLE_TYPESCRIPT_HOME` is no longer supported and is ignored.
5451

5552
Distribution packaging
5653
----------------------
@@ -59,12 +56,12 @@ The Bazel build produces two archives: `:typescript` (`javascript.zip`) contains
5956
the compiled ESM wrapper and JavaScript API; `:native` (`native.zip`) contains the
6057
matching native packages for the target CodeQL distribution:
6158

62-
| Distribution | Native packages |
63-
| --- | --- |
64-
| `linux64` | `@typescript/typescript-linux-x64` |
65-
| `linux-arm64` | `@typescript/typescript-linux-arm64` |
66-
| `osx64` | `@typescript/typescript-darwin-arm64` and `@typescript/typescript-darwin-x64` |
67-
| `win64` | `@typescript/typescript-win32-x64` |
59+
| Distribution | Native packages |
60+
| ------------- | ----------------------------------------------------------------------------- |
61+
| `linux64` | `@typescript/typescript-linux-x64` |
62+
| `linux-arm64` | `@typescript/typescript-linux-arm64` |
63+
| `osx64` | `@typescript/typescript-darwin-arm64` and `@typescript/typescript-darwin-x64` |
64+
| `win64` | `@typescript/typescript-win32-x64` |
6865

6966
The build installs the selected packages explicitly with `--force --no-save`,
7067
using the pinned versions and integrities in the lockfile. Merely passing
@@ -97,12 +94,13 @@ Linux arm64, and Windows x64, including Windows 2022.
9794
Native lifecycle and memory
9895
---------------------------
9996

100-
Each file owns a fresh native API/process. AST projection and JSON serialization
101-
finish before the program is disposed and the API is closed, including on
102-
failure. No native process or compiler cache is retained between files, during
103-
stdout backpressure, or across reset/quit. This deliberately favors bounded
104-
resource lifetime over reusing a compiler process; the measured cost is described
105-
below.
97+
A single native API/process is created lazily and reused across files and
98+
prepare-files batches. Each file still owns a separate program: AST projection
99+
and JSON serialization finish before the program is disposed, including on
100+
failure. Program disposal releases its snapshot and reference-counted source-file
101+
caches without restarting the compiler. Reset closes the API, and the next parse
102+
creates a new one. Quit, EOF, failure, and completion of single-file mode also
103+
close the API.
106104

107105
`SEMMLE_TYPESCRIPT_MEMORY_THRESHOLD` remains a positive megabyte budget, defaulting
108106
to 1000 when the Java extractor does not supply it. At native-process creation,
@@ -111,9 +109,9 @@ and passes the remaining bytes, with a minimum of one, as the child's
111109
`GOMEMLIMIT`. The parent's original `GOMEMLIMIT` is restored after spawning.
112110
This is a Go garbage-collector soft limit, not a hard combined RSS ceiling:
113111
Node allocations during extraction, native non-Go allocations, and oversized
114-
individual files can exceed the budget. Closing the native process after every
115-
file prevents native caches and heap capacity from accumulating across files.
116-
Node RSS exceeding the budget produces a warning on stderr.
112+
individual files can exceed the budget. The child's budget is fixed until the
113+
API is closed; later changes to Node RSS do not resize it. Node RSS exceeding
114+
the budget at native-process creation produces a warning on stderr.
117115

118116
Native stderr is inherited separately from the JSON stdout protocol. Syntax
119117
diagnostics remain successful AST responses; native, I/O, and protocol failures
@@ -137,7 +135,7 @@ Projection checks only the whitelisted properties applicable to each syntax kind
137135
including optional properties, and falls back to the full whitelist for unknown
138136
kinds. `src/ast_extractor.ts` loads the private
139137
`dist/api/node/protocol.generated.js` metadata once from the locally installed,
140-
pinned TypeScript dependency, independently of `SEMMLE_TYPESCRIPT_HOME`.
138+
pinned TypeScript dependency.
141139
Updating TypeScript may require adapting this internal path or metadata format.
142140
Scalar getters and kind-specific booleans are handled separately.
143141
The Java converter consumes native fields directly, including `postfixToken`,
@@ -146,8 +144,8 @@ spelling. Namespace classification and variable declaration kinds are derived
146144
in Java without synthetic flags or properties.
147145

148146
The tests cover AST projection, metadata, UTF-16 source positions, syntax
149-
diagnostics, TSX, repeated parses, native process disposal, the child memory
150-
budget, reset/quit/EOF, cancelled prefetch, runtime overrides, and fatal native
147+
diagnostics, TSX, repeated parses, native process reuse and disposal, the child memory
148+
budget, reset/quit/EOF, cancelled prefetch, the ignored legacy override, and fatal native
151149
failures. Run them with Node.js 16.20.0 to verify the minimum supported runtime.
152150

153151
Compatibility and regression coverage
@@ -200,19 +198,56 @@ codeql test run -j4 --search-path=javascript/extractor-pack:. \
200198
Performance comparison
201199
----------------------
202200

203-
On macOS arm64 with Node.js 18.15.0, a comparison against the pre-migration
204-
wrapper from `1971c368e1d` compiled with TypeScript 5.9.2 gave these medians over three
205-
runs:
201+
The current shared-API implementation was compared with refreshed `main` at
202+
`fb0ba8c8a8a` on October 21, 2026. Both ran on macOS arm64 with Node.js 18.15.0;
203+
`main` was built and run with TypeScript 5.9.2, and this branch used the pinned
204+
7.1 native compiler. These are medians of five fresh-process runs after one
205+
warm-up per variant and workload, alternating variant order:
206206

207-
| Workload | 5.9.2 time | 7.1 time | 5.9.2 combined RSS | 7.1 combined RSS |
207+
| Workload | Main time | Branch time | Main combined RSS | Branch combined RSS |
208208
| --- | --- | --- | --- | --- |
209-
| 103 TypeScript fixtures (33,589 bytes) | 0.237 s | 1.349 s | 124.1 MiB | 113.9 MiB |
210-
| Five parses of TypeScript's 588,128-byte declaration file | 0.991 s | 1.116 s | 269.3 MiB | 249.1 MiB |
209+
| 104 TypeScript fixtures (33,779 bytes) | 0.081 s | 0.120 s | 105.5 MiB | 110.4 MiB |
210+
| Five parses of TypeScript's 588,128-byte declaration file | 0.886 s | 0.992 s | 263.7 MiB | 247.2 MiB |
211+
212+
The branch takes 47.3% more time on the fixtures and 11.9% more on the declaration
213+
workload. Sampled combined RSS is 4.7% higher and 6.3% lower, respectively.
214+
Both variants parse identical inputs. Timing starts after the metadata handshake
215+
and includes prepare-files, AST extraction, serialization, and protocol responses,
216+
including the branch's first native-process startup. It excludes Java conversion
217+
and database import. RSS is the maximum sampled sum of Node and its native
218+
descendants, sampled at a nominal 5 ms interval; short-lived peaks may be missed.
219+
Current `main` includes parser cleanup absent from the older baseline below.
220+
221+
### Historical and API-lifetime comparisons
222+
223+
The initial implementation created a native process per file. On macOS arm64
224+
with Node.js 18.15.0, a comparison against the pre-migration wrapper from
225+
`1971c368e1d` compiled with TypeScript 5.9.2 gave these medians over three runs:
226+
227+
| Workload | 5.9.2 time | 7.1 time | 5.9.2 combined RSS | 7.1 combined RSS |
228+
| --------------------------------------------------------- | ---------- | -------- | ------------------ | ---------------- |
229+
| 103 TypeScript fixtures (33,589 bytes) | 0.237 s | 1.349 s | 124.1 MiB | 113.9 MiB |
230+
| Five parses of TypeScript's 588,128-byte declaration file | 0.991 s | 1.116 s | 269.3 MiB | 249.1 MiB |
211231

212232
Timing covers prepare-files, extraction, serialization, and protocol responses
213233
after the metadata handshake, not Java conversion or database import. Combined
214234
RSS is the maximum sampled sum of Node and its native descendants, sampled every
215235
5 ms; short-lived peaks may be missed. The per-file native process costs about
216236
5.7x on tiny files and 1.13x on the declaration workload, while measured memory is
217-
slightly lower. The bounded lifecycle is retained deliberately; this migration
218-
does not claim a speedup.
237+
slightly lower.
238+
239+
A follow-up comparison on October 21, 2026, using the same platform and Node.js
240+
version, measured the per-file and shared-API lifetimes directly (medians of three
241+
runs):
242+
243+
| Workload | Per-file API time | Shared API time | Per-file combined RSS | Shared combined RSS |
244+
| --- | --- | --- | --- | --- |
245+
| 104 TypeScript fixtures (33,779 bytes) | 1.017 s | 0.118 s | 115.8 MiB | 109.4 MiB |
246+
| Five parses of the same declaration file | 1.044 s | 0.985 s | 250.2 MiB | 260.2 MiB |
247+
248+
AST responses were byte-identical between lifetimes. A longer shared-API run on
249+
Node.js 26.7.0 processed 50,000 distinct generated TypeScript files: native RSS
250+
was 27.9 MiB after 1,000 files and 30.4 MiB after 50,000, while combined RSS
251+
stabilized around 176 MiB. Another run completed 500 parses of the declaration
252+
file with peak native RSS of 56.5 MiB. These workloads showed no unbounded
253+
per-file retention, so the wrapper reuses the API without a fixed batch limit.

‎javascript/extractor/lib/typescript/package-runtime.mjs‎

Lines changed: 30 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,35 +1,53 @@
1-
import { chmodSync, mkdirSync, readFileSync, renameSync, rmSync, statSync } from "node:fs";
1+
import {
2+
chmodSync,
3+
mkdirSync,
4+
readFileSync,
5+
renameSync,
6+
rmSync,
7+
statSync,
8+
} from "node:fs";
29
import { join } from "node:path";
310

411
const platforms = {
5-
"linux64": ["linux-x64"],
12+
linux64: ["linux-x64"],
613
"linux-arm64": ["linux-arm64"],
7-
"osx64": ["darwin-arm64", "darwin-x64"],
8-
"win64": ["win32-x64"],
14+
osx64: ["darwin-arm64", "darwin-x64"],
15+
win64: ["win32-x64"],
916
};
1017
const packagesOnly = process.argv[2] === "--packages";
1118
const distribution = process.argv[packagesOnly ? 3 : 2];
1219
if (!Object.hasOwn(platforms, distribution)) {
1320
throw new Error("Unsupported CodeQL distribution: " + distribution);
1421
}
15-
const readJson = filename => JSON.parse(readFileSync(filename, "utf8"));
22+
const readJson = (filename) => JSON.parse(readFileSync(filename, "utf8"));
1623
const typescript = readJson("node_modules/typescript/package.json");
1724
if (packagesOnly) {
18-
console.log(platforms[distribution]
19-
.map(platform => "@typescript/typescript-" + platform + "@" + typescript.version).join(" "));
25+
console.log(
26+
platforms[distribution]
27+
.map(
28+
(platform) =>
29+
"@typescript/typescript-" + platform + "@" + typescript.version,
30+
)
31+
.join(" "),
32+
);
2033
process.exit(0);
2134
}
2235
mkdirSync("native");
2336
for (const platform of platforms[distribution]) {
2437
const name = "typescript-" + platform;
2538
const source = join("node_modules", "@typescript", name);
2639
const manifest = readJson(join(source, "package.json"));
27-
if (manifest.name !== "@typescript/" + name
28-
|| manifest.version !== typescript.version
29-
|| typescript.optionalDependencies[manifest.name] !== manifest.version) {
30-
throw new Error("Native runtime does not match TypeScript: " + manifest.name);
40+
if (
41+
manifest.name !== "@typescript/" + name ||
42+
manifest.version !== typescript.version ||
43+
typescript.optionalDependencies[manifest.name] !== manifest.version
44+
) {
45+
throw new Error(
46+
"Native runtime does not match TypeScript: " + manifest.name,
47+
);
3148
}
32-
const executable = join(source, "lib", platform.startsWith("win32-") ? "tsc.exe" : "tsc");
49+
const executableName = platform.startsWith("win32-") ? "tsc.exe" : "tsc";
50+
const executable = join(source, "lib", executableName);
3351
if (!statSync(executable).isFile()) {
3452
throw new Error("Native executable is not a file: " + executable);
3553
}

‎javascript/extractor/lib/typescript/src/main.ts‎

Lines changed: 47 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,10 @@ let stopped = false;
4141
let readLineInterface: readline.Interface | undefined;
4242
let reportedMemoryPressure = false;
4343

44+
// The API instance used to interact with the TypeScript compiler. Should be
45+
// accessed through `getAPI` and `closeAPI`.
46+
let api: API | undefined;
47+
4448
function createAPI(): API {
4549
const threshold = Number(process.env.SEMMLE_TYPESCRIPT_MEMORY_THRESHOLD ?? 1000) * 1000000;
4650
if (!Number.isSafeInteger(threshold) || threshold <= 0) {
@@ -63,29 +67,34 @@ function createAPI(): API {
6367
}
6468
}
6569

70+
function getAPI(): API {
71+
return api ??= createAPI();
72+
}
73+
74+
function closeAPI() {
75+
const previous = api;
76+
api = undefined;
77+
previous?.close();
78+
}
79+
6680
function extractFile(filename: string): string {
67-
const api = createAPI();
81+
const program = getAPI().createProgram([pathlib.resolve(filename)], {
82+
experimentalDecorators: true,
83+
jsx: JsxEmit.Preserve,
84+
noResolve: true,
85+
noLib: true,
86+
});
6887
try {
69-
const program = api.createProgram([pathlib.resolve(filename)], {
70-
experimentalDecorators: true,
71-
jsx: JsxEmit.Preserve,
72-
noResolve: true,
73-
noLib: true,
74-
});
75-
try {
76-
const ast = program.getSourceFile(pathlib.resolve(filename));
77-
if (ast == null) {
78-
throw new Error("TypeScript did not return a source file for " + filename);
79-
}
80-
return JSON.stringify({
81-
type: "ast",
82-
ast: extractAst(ast, program.getSyntacticDiagnostics()),
83-
});
84-
} finally {
85-
program.dispose();
88+
const ast = program.getSourceFile(pathlib.resolve(filename));
89+
if (ast == null) {
90+
throw new Error("TypeScript did not return a source file for " + filename);
8691
}
92+
return JSON.stringify({
93+
type: "ast",
94+
ast: extractAst(ast, program.getSyntacticDiagnostics()),
95+
});
8796
} finally {
88-
api.close();
97+
program.dispose();
8998
}
9099
}
91100

@@ -99,10 +108,16 @@ function prepareNextFile() {
99108
function fail(error: unknown) {
100109
stopped = true;
101110
state = new State();
102-
readLineInterface?.close();
103-
process.stdin.destroy();
104111
console.error(error);
105112
process.exitCode = 1;
113+
try {
114+
closeAPI();
115+
} catch (closeError) {
116+
console.error(closeError);
117+
} finally {
118+
readLineInterface?.close();
119+
process.stdin.destroy();
120+
}
106121
}
107122

108123
function afterResponse(expectedState: State) {
@@ -141,6 +156,11 @@ function runReadLineInterface() {
141156
stopped = true;
142157
state = new State();
143158
process.stdin.destroy();
159+
try {
160+
closeAPI();
161+
} catch (error) {
162+
fail(error);
163+
}
144164
});
145165
rl.on("error", fail);
146166
rl.on("line", (line: string) => {
@@ -158,6 +178,7 @@ function runReadLineInterface() {
158178
break;
159179
case "reset":
160180
state = new State();
181+
closeAPI();
161182
console.log('{"type":"reset-done"}');
162183
break;
163184
case "get-metadata":
@@ -187,7 +208,11 @@ try {
187208
if (argument === "--version") {
188209
console.log("parser-wrapper with TypeScript " + version);
189210
} else if (pathlib.extname(argument) === ".ts" || pathlib.extname(argument) === ".tsx") {
190-
process.stdout.write(extractFile(argument) + "\n");
211+
try {
212+
process.stdout.write(extractFile(argument) + "\n");
213+
} finally {
214+
closeAPI();
215+
}
191216
} else {
192217
console.error("Unrecognized file or flag: " + argument);
193218
process.exitCode = 1;

0 commit comments

Comments
 (0)