Repository navigation
Safely dump unknown strings. #873
Description
Sometimes we produce JSON from unknown sources, e.g. system supplied strings which could contain invalid byte sequences. We need to do something like produce JSON for a logging system or to capture details of the operation (result of HTTP request, RPC, etc).
It's kind of tricky to do this correctly and I've found that often things look good in testing but fail in non-test environments due to invalid strings.
It would be nice if JSON.dump could be a bit more robust by default. So rather than failing when encountering malformed UTF-8, we could instead encode it, e.g. string.encode(Encoding::UTF_8, invalid: :replace, undef: :replace) for all strings.
Such a feature could be opt-in, e.g.
JSON.dump(thing, encode: true) # or safe: true or something else
Here is an example implementation: https://github.com/socketry/console/blob/main/lib/console/format/safe.rb which I use for logging. It's a bit more elaborate as it handles even more safety issues.
Activity
eregon commentedon Oct 22, 2025
That's losing data though, because all these invalid bytes will just become
�and it's irrecoverable from there.I'd guess this happens mostly when the original encoding isn't detected correctly.
I guess you mean opt-in and non-default, right?
It seems clear data loss can't be the default anyway.
samuel-williams-shopify commentedon Oct 22, 2025
If the choice is between data loss and complete failure, there are many cases where data loss is acceptable (e.g. logging).
byroot commentedon Oct 23, 2025
Yeah, that's why I'm always puzzled at JSON being used for logging.
But anyways. One thing to investigate is having
JSON::Coderyield invalid strings, this should allow users to define whatever logic they want.Not sure when I'll have time to look at this though.
Invoke `as_json` callback for strings with invalid encoding
Invoke `as_json` callback for strings with invalid encoding
byroot commentedon Oct 30, 2025
So it would look like that #877 (Still need to implement the JRuby and TruffleRuby versions).
Invoke `as_json` callback for strings with invalid encoding
Invoke `as_json` callback for strings with invalid encoding
Invoke `as_json` callback for strings with invalid encoding
Invoke `as_json` callback for strings with invalid encoding
Invoke `as_json` callback for strings with invalid encoding
Invoke `as_json` callback for strings with invalid encoding
Invoke `as_json` callback for strings with invalid encoding
[ruby/json] Invoke `as_json` callback for strings with invalid encoding