Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ is published under the public `@bidilens` npm scope.
preserving `$...$`, `$$...$$`, `$$$$`, and `\(...\)` recognition.
- Expressed UTF-16-to-code-point range construction with bounded typed-array
fills so input-derived offsets cannot be interpreted as object properties.
- Restricted downloaded Unicode data writes to the two repository-owned,
version-pinned destination paths after exact SHA-256 verification.
- Removed the generator's network-to-filesystem refresh mode; Unicode table
generation now reads only two vendored, version- and SHA-256-pinned files.

### Direction correctness

Expand Down
5 changes: 2 additions & 3 deletions docs/SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,8 @@ pops/openers, formatting that crosses an isolate boundary, and U+200B.
- recursive CLI scanning skips symbolic links.
- raw-text math delimiter discovery is a single forward scan; repeated
unmatched delimiters cannot trigger polynomial regular-expression work.
- the optional Unicode download command accepts only two source/destination
pairs pinned in repository code and verifies each exact SHA-256 before a
write; untrusted text never reaches that build-time path.
- Unicode generation is offline-only: it reads two vendored, SHA-256-pinned
source files and exposes no network-to-filesystem refresh path.

## False-positive policy

Expand Down
1 change: 0 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@
"build:visual-fixtures": "pnpm --filter @bidilens/core run build && pnpm --filter @bidilens/markdown run build && pnpm --filter @bidilens/react run build && pnpm --filter @bidilens/web-component run build",
"test:visual": "pnpm run build:visual-fixtures && playwright test",
"unicode:generate": "tsx scripts/generate-bidi-data.ts",
"unicode:download": "tsx scripts/generate-bidi-data.ts --download",
"unicode:check": "tsx scripts/generate-bidi-data.ts --check",
"release:check": "tsx scripts/release-check.ts",
"android:check": "tsx scripts/run-android.ts",
Expand Down
28 changes: 7 additions & 21 deletions scripts/generate-bidi-data.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,6 @@ const UNICODE_SOURCES = [
}
] as const satisfies readonly UnicodeSource[];

const PINNED_UNICODE_PATHS = new Set(UNICODE_SOURCES.map((source) => source.path));

const CLASS = {
L: 0,
R: 1,
Expand Down Expand Up @@ -270,36 +268,24 @@ ${renderRustRanges('NATURAL_LETTER_RANGES', ranges.naturalLetters)}
`;
}

async function sourceBytes(source: UnicodeSource, download: boolean): Promise<Uint8Array> {
let bytes: Uint8Array;
if (download) {
const response = await fetch(source.url);
if (!response.ok) throw new Error(`${source.label} download failed: HTTP ${response.status}`);
bytes = new Uint8Array(await response.arrayBuffer());
} else {
bytes = await readFile(source.path);
}
function verifiedBytes(source: UnicodeSource, bytes: Uint8Array): Uint8Array {
const actualSha256 = createHash('sha256').update(bytes).digest('hex');
if (actualSha256 !== source.sha256) {
throw new Error(`${source.label} checksum mismatch: expected ${source.sha256}, received ${actualSha256}`);
}
if (download) {
if (!PINNED_UNICODE_PATHS.has(source.path)) {
throw new Error(`Refusing to write outside the pinned Unicode source paths: ${source.path}`);
}
await mkdir(dirname(source.path), { recursive: true });
await writeFile(source.path, bytes);
}
return bytes;
}

async function localSourceBytes(source: UnicodeSource): Promise<Uint8Array> {
return verifiedBytes(source, await readFile(source.path));
}

async function main(): Promise<void> {
const download = process.argv.includes('--download');
const check = process.argv.includes('--check');
const [bidiSource, generalCategorySource] = UNICODE_SOURCES;
const [bidiBytes, generalCategoryBytes] = await Promise.all([
sourceBytes(bidiSource, download),
sourceBytes(generalCategorySource, download)
localSourceBytes(bidiSource),
localSourceBytes(generalCategorySource)
]);
const ranges = collectGeneratedRanges(
new TextDecoder().decode(bidiBytes),
Expand Down
9 changes: 6 additions & 3 deletions unicode/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,14 @@ To reproduce the generated table from the vendored source:
pnpm unicode:generate
```

To deliberately refresh the pinned upstream file, first update the version,
URLs, and both expected SHA-256 values in `scripts/generate-bidi-data.ts`, then run:
To deliberately refresh the pinned upstream files, download both exact source
URLs shown in `scripts/generate-bidi-data.ts` outside this repository, verify
their expected SHA-256 values, replace only the two vendored files, then update
the version and hashes in the generator. The generator deliberately has no
network-to-filesystem mode. Review with:

```bash
pnpm unicode:download
pnpm unicode:generate
git diff -- unicode packages/core/src/generated/bidi-ranges.ts
pnpm run check
```
Expand Down