[YeungKC]

Summary

tokenize() did not free the buffer from toNativeUtf8(), so each call leaked a copy of the input string. CFStringCreateWithCString copies the bytes, so the buffer is freed after the call.

+  final utf8 = string.toNativeUtf8();
   final cfString = _bindings.CFStringCreateWithCString(
     nullptr,
-    string.toNativeUtf8().cast(),
+    utf8.cast(),
     binding.CFStringBuiltInEncodings.kCFStringEncodingUTF8.value,
   );
+  malloc.free(utf8);

Evidence

Call tokenize() 200 times on a 1 MB string, then compare RSS (macOS):

• Before: RSS +146 MB
  After: RSS -11 MB

tokenize('hello world 你好世界') returns [hello, world, 你, 好, 世界] before and after.

Merge Danger

Door: two-way

Blast Radius: small

No API change. CFStringCreateWithCString does not keep the pointer.

[Copilot]

🟢 Approval recommended

The buffer is freed with the matching allocator after its final use, with no blocking issues identified.

No open issues 🎉

What changed in this PR

Fixes a per-call native memory leak in string_tokenizer.tokenize() without changing its public API.

Changes:

• Retains the temporary UTF-8 pointer and frees it after Core Foundation copies its contents.

File	Description
packages/​string_tokenizer/​lib/​string_tokenizer.dart	Frees the temporary UTF-8 allocation after string creation.

_{Review effort: Balanced}

[YeungKC]

#548 replaced string_tokenizer with word_segmenter and also fixed this UTF-8 buffer leak, so this PR is no longer necessary.