Repository navigation
Honor a legacy CHARSET property parameter when parsing bytes - #2080
rahul05ranjan wants to merge 4 commits into
Conversation
|
Thanks for the pull request, @rahul05ranjan! A maintainer will review it shortly. |
Automation signalsActivity patterns show signs of automation. Evidence
Last 2 PRs: This is an automated analysis by AgentScan |
|
One or more PR-quality checks did not pass. Review the PR Quality Checks summary in the workflow run for the failed checks and their details, then make any necessary updates. Checklist items may remain unchecked while work is in progress. Please tick each item as you complete it. If you used AI assistance, follow the project's Responsible AI use policy and provide the required disclosures. |
3cf74d7 to
ea787e1
Compare
…ytes from_ical(bytes) decoded non-UTF-8 input with errors=replace before the property parameters were ever looked at, so a value that explicitly declared its encoding with a legacy CHARSET parameter (the marker pre-RFC 5545 producers left behind) still came back as U+FFFD soup. Decode each content line with the charset its CHARSET parameter declares; lines without the parameter keep the previous best-effort UTF-8 decoding, and an unknown charset falls back the same way. Fixes PyCalendar#2078 AI: GLM-5.3 (z-ai, via OpenRouter). How used: AI coding assistant. I supplied the repro from issue PyCalendar#2078 and the fix design (decode per content line, restricted to the parameter section before the first colon); the model drafted the patch, the regression tests and this message from my instructions. I reviewed the change and verified it by running the test suite. Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
ea787e1 to
d84f9ce
Compare
| NEWLINE_BYTES = re.compile(rb"\r?\n") | ||
| # a legacy CHARSET parameter declares the encoding of one property value | ||
| CHARSET_PARAM_BYTES = re.compile( | ||
| rb";[ \t]*CHARSET[ \t]*=[ \t]*([^;:\r\n]+)", re.IGNORECASE |
There was a problem hiding this comment.
Can you please use the possessive quantifier.
There was a problem hiding this comment.
Your are right, but one can detect the python version used and use possessive quantifiers when available.
|
|
||
| def test_charset_text_inside_quoted_param_value_is_not_a_directive_issue_2078(): | ||
| """UTF-8 text with a quoted parameter value mentioning CHARSET= must | ||
| still decode as UTF-8, not as the encoding named in the quotes.""" |
There was a problem hiding this comment.
Where does this come from?
There was a problem hiding this comment.
To be clear, where in the RFC is this specified? §3.1 is about gramar, not the charset.
There was a problem hiding this comment.
Why aren't you using threads to answer? I'm personally aren't sure how this should be implemented because I do not find suficcient information on it. When CHARSET is specified, to what part does it apply? line, properties and value(s) or just value(s)? Please give references to a specification for that. Otherwise I would reject this or at least call for other maintainers.
There was a problem hiding this comment.
Point taken on threading — answering here in this thread.
Answer: CHARSET applies to the value(s) of the single property that carries it — not to the whole line, and not to other lines or properties. The primary sources are the two versit consortium specs (both dated September 18, 1996) that define the parameter — vCalendar 1.0 and vCard 2.1 — and they use identical wording in §2.1 "Encoding Characteristics" → "Character Set":
The default character set is ASCII. The default character set can be overridden for an individual property value by using the "CHARSET" property parameter. This property parameter may be used on any property. However, the use of this parameter on some properties may not make sense. Any character set registered with the Internet Assigned Numbers Authority (IANA) can be specified by this property parameter. For example, ISO 8859-8 or the Latin/Hebrew character set is specified by:
DESCRIPTION;CHARSET=ISO-8859-8:...
(the vCard 2.1 spec has the same sentence with ADR;CHARSET=ISO-8859-8:...)
- vCalendar 1.0: http://web.archive.org/web/20160321023312/https://www.imc.org/pdi/vcal-10.txt
- vCard 2.1: http://web.archive.org/web/20160402232011/https://www.imc.org/pdi/vcard-21.txt
So the ASCII default stays in force everywhere else; each property overrides it individually, and only for its value. Both grammars also pin the rest of the line to ASCII: param = ... / "CHARSET" [ws] "=" [ws] charsetval with charsetval = <a character set string as defined in Section 7.1 of RFC 1521>, and word = <any printable 7bit us-ascii except []=:., > — property names and parameter values are 7-bit ASCII tokens, so the declared charset is only meaningful for the value after the colon.
The parameter is not part of RFC 5545 at all (§3.1.4: "There is not a property parameter to declare the charset used in a property value. The default charset for an iCalendar stream is UTF-8"), and it disappeared from vCard after 2.1 — RFC 6350 Appendix A.2 lists it among removed features ("The CONTEXT and CHARSET parameters are no more."). That is why this PR treats it purely as legacy tolerance for pre-5545 files (issue #2078), never as an iCalendar feature.
Among other parsers, the clearest reference I found is vobject, which decodes the property value with the CHARSET parameter on its QUOTED-PRINTABLE path: self.value = codecs.decode(self.value.encode("utf-8"), "quoted-printable").decode(self.params['CHARSET'][0]) — vobject/base.py L339-342. libical and ical.js do not honor CHARSET at all (both assume UTF-8 per RFC 5545), so they neither contradict nor confirm the semantics.
What this PR does: for each content line whose parameter section (before the first colon) carries an unquoted ;CHARSET=, that line is decoded with the declared charset; every other line keeps the existing UTF-8 handling. Strictly per the specs the parameter covers only the value, but the implementation decodes the whole raw content line with it — which is indistinguishable from value-only decoding for every line that conforms to the grammar above: the name and parameters are 7-bit ASCII, and the charsets legacy files actually use (ISO-8859-x, windows-125x, KOI8-R, Shift_JIS, …) are ASCII-transparent, so the header decodes identically either way. The difference only shows for non-ASCII bytes inside the parameter section, which those grammars do not allow; there the PR prefers the declared charset over emitting U+FFFD. I am happy to narrow it to decode exactly the value after the colon if you prefer strict conformance over that tolerance.
One fix that came out of re-checking this against the sources: the helper's docstring claimed valid UTF-8 input is never rerouted through the CHARSET path, but the code had no such gate — a file that had been re-encoded to UTF-8 while keeping a vestigial CHARSET=ISO-8859-1 parameter had its UTF-8 bytes decoded as latin-1 and turned into mojibake. Fixed in 6d0241e: valid UTF-8 input now parses exactly as on main, and only input that is not valid UTF-8 as a whole gets the per-line CHARSET decoding; a regression test pins the vestigial-parameter case, and the quoted-parameter test now mixes in a latin-1 line so it still exercises the scan. Locally, the parsing/encoding suites give 12 failed / 538 passed against 12 failed / 536 passed on the parent commit — the 12 are the same pre-existing timezone-data host artifacts in both runs.
|
Thanks for the review @angatha — on both points: Possessive quantifier: happy to switch the capture group to "Where does this come from?" (test_charset_text_inside_quoted_param_value...): that case is one I constructed defensively — it is not from a real-world file or from issue #2078 itself. The scenario behind the PR (legacy |
…ariants Review feedback on PyCalendar#2080 (thank you @angatha): - RFC 5545 §3.1.4, not the §3.1 grammar, is the section on charset: it states "There is not a property parameter to declare the charset used in a property value. The default charset for an iCalendar stream is UTF-8". The quoted-parameter-value test now cites what actually supports it: §3.1's `quoted-string = DQUOTE *QSAFE-CHAR DQUOTE` (QSAFE-CHAR = any character except CONTROL and DQUOTE), which lets ";" and "=" appear inside a quoted parameter value, and §3.1.4, which is why an unquoted legacy CHARSET= is only ever tolerated (issue PyCalendar#2078), never conforming. - The CHARSET parameter regex now uses a possessive quantifier where the running Python supports it, following the existing version-guard pattern from the XML property (prop/xml.py): Python 3.11+ gets `([^;:\r\n]++)`, Python 3.10 keeps the plain greedy form (the character class excludes the delimiters, so both match the same strings). The three issue PyCalendar#2078 tests are parametrized over both regexes so each runs everywhere it compiles. Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
|
Thanks @angatha — both points addressed in f8467d3. On the RFC question: you are right, and the section on charset is §3.1.4, "Character Set": "There is not a property parameter to declare the charset used in a property value. The default charset for an iCalendar stream is UTF-8." So to be clear in return: On the possessive quantifier: done, following the existing version-guard pattern from the XML property ( Full suite on this machine with the change: 21,634 passed / 584 failed / 578 xfailed, against 21,631 passed / the same 584 failed / 578 xfailed on the parent commit — the 584 are the timezone-data/dateutil host artifacts that reproduce identically without my change. |
The _decode_with_charset docstring promised that valid UTF-8 input is never rerouted through the CHARSET parameter, but the code checked no such gate: a file that was re-encoded to UTF-8 while keeping a vestigial legacy CHARSET=ISO-8859-1 parameter decoded its UTF-8 bytes as latin-1 and turned non-ASCII text into mojibake, where the pre-CHARSET code parsed it correctly. Valid UTF-8 input (a conforming RFC 5545 stream) is now decoded as before, so a vestigial CHARSET cannot override the charset the stream actually uses. Only input that is not valid UTF-8 as a whole gets the per-line legacy CHARSET decoding. The quoted-parameter test gains a latin-1 DESCRIPTION line so it still exercises the CHARSET scan under that gate, and a new regression test pins the vestigial-parameter case. Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
|
@niccokunzmann, @SashankBhamidi this sounds more like a separate tool to convert old files into an RFC5545 compatible encoding. In my view, merging this is definetly unspecified behavior for RFC5545 and not some kind of backwards compatibility. What is your opinion? |
Description
from_ical(bytes)runs the input throughto_unicode(), whoseerrors="replace"fallback fires before property parameters are ever looked at. So when an .ics file is not valid UTF-8 but a property explicitly declares its charset with the legacyCHARSETparameter — the marker pre-RFC 5545 producers left behind — the information needed to decode correctly is available and still ignored:This PR decodes each content line with the charset its
CHARSETparameter declares:CHARSET=is restricted to the parameter section (before the first:), so a quoted parameter value mentioningCHARSET=...is not mistaken for a directive;CHARSETparameter keep the previous best-effort UTF-8 decoding, byte for byte;Old exports from Outlook/Lotus tooling that spell non-UTF-8 text this way still circulate (migration dumps, mailing-list archives); for every non-ASCII value in them the summary/location/description currently comes back as U+FFFD soup.
Checklist
Additional information
Verification: three new tests in
src/icalendar/tests/test_encoding.py— the latin-1 repro (fails on current main with U+FFFD, passes here), the unknown-charset graceful fallback, and the quoted-CHARSET=-text guard. Targeted run (test_parsing.py,test_components_break_on_bad_ics.py,test_encoding.py,test_bom_calendar.py,tests/prop): 1,909 passed / 12 failed, where all 12 are the timezone-data-relatedtzidhost artifacts that reproduce identically on unmodified main on this machine (Python 3.13.5, run from a checkout without installing; the same 12 are listed in the full-suite evidence on my related #2079). No API change, so no docs were touched.