Skip to content

Honor a legacy CHARSET property parameter when parsing bytes - #2080

Open
rahul05ranjan wants to merge 4 commits into
PyCalendar:mainfrom
rahul05ranjan:fix/charset-parameter-decoding
Open

rahul05ranjan wants to merge 4 commits into
PyCalendar:mainfrom
rahul05ranjan:fix/charset-parameter-decoding

Conversation

@rahul05ranjan

Copy link
Copy Markdown

Description

from_ical(bytes) runs the input through to_unicode(), whose errors="replace" fallback fires before property parameters are ever looked at. So when an .ics file is not valid UTF-8 but a property explicitly declares its charset with the legacy CHARSET parameter — the marker pre-RFC 5545 producers left behind — the information needed to decode correctly is available and still ignored:

ics = b"...\r\nSUMMARY;CHARSET=ISO-8859-1:Caf\xe9 Meeting\r\n..."
event = Calendar.from_ical(ics).walk("VEVENT")[0]
str(event["SUMMARY"])   # 'Caf\N{REPLACEMENT CHARACTER} Meeting' — the é is gone

This PR decodes each content line with the charset its CHARSET parameter declares:

  • the search for CHARSET= is restricted to the parameter section (before the first :), so a quoted parameter value mentioning CHARSET=... is not mistaken for a directive;
  • lines without a CHARSET parameter keep the previous best-effort UTF-8 decoding, byte for byte;
  • an unknown charset name falls back to the previous behavior instead of raising.

Old exports from Outlook/Lotus tooling that spell non-UTF-8 text this way still circulate (migration dumps, mailing-list archives); for every non-ASCII value in them the summary/location/description currently comes back as U+FFFD soup.

Checklist

  • I added a change log entry, following the instructions in all subsections under Change log requirements.
  • I followed icalendar's Artificial intelligence policy and disclosed my Responsible AI use in my commit messages, if applicable.
  • I added or updated tests, if applicable.
  • I ran and ensured all tests pass locally by following Run tests.
  • I added or edited documentation as necessary, both as docstrings to be rendered in the API documentation and narrative documentation, following the Style guide.

Additional information

Verification: three new tests in src/icalendar/tests/test_encoding.py — the latin-1 repro (fails on current main with U+FFFD, passes here), the unknown-charset graceful fallback, and the quoted-CHARSET=-text guard. Targeted run (test_parsing.py, test_components_break_on_bad_ics.py, test_encoding.py, test_bom_calendar.py, tests/prop): 1,909 passed / 12 failed, where all 12 are the timezone-data-related tzid host artifacts that reproduce identically on unmodified main on this machine (Python 3.13.5, run from a checkout without installing; the same 12 are listed in the full-suite evidence on my related #2079). No API change, so no docs were touched.

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Thanks for the pull request, @rahul05ranjan! A maintainer will review it shortly.

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Automation signals

Activity patterns show signs of automation.

View full analysis →

Report this account →

Evidence
  • Fork spike detected: Burst of 10 fork events in a single 24-hour window
  • Issue comment and PR within minutes: Commented on an issue and opened a PR to the same repo within 5 minutes, in 2 repositories (the fastest took 101s)

Last 2 PRs:

This is an automated analysis by AgentScan

@github-actions github-actions Bot added the agentscan:automated-account AgentScan flagged this item as coming from an automated account, and it needs review label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

One or more PR-quality checks did not pass. Review the PR Quality Checks summary in the workflow run for the failed checks and their details, then make any necessary updates.

Checklist items may remain unchecked while work is in progress. Please tick each item as you complete it.

If you used AI assistance, follow the project's Responsible AI use policy and provide the required disclosures.

@github-actions github-actions Bot added the ai-suspicion This contribution is possibly created with lots of AI help without enough human understanding. label Nov 8, 2026
@rahul05ranjan
rahul05ranjan force-pushed the fix/charset-parameter-decoding branch 4 times, most recently from 3cf74d7 to ea787e1 Compare November 8, 2026 23:39
…ytes

from_ical(bytes) decoded non-UTF-8 input with errors=replace before the
property parameters were ever looked at, so a value that explicitly
declared its encoding with a legacy CHARSET parameter (the marker
pre-RFC 5545 producers left behind) still came back as U+FFFD soup.
Decode each content line with the charset its CHARSET parameter
declares; lines without the parameter keep the previous best-effort
UTF-8 decoding, and an unknown charset falls back the same way.

Fixes PyCalendar#2078

AI: GLM-5.3 (z-ai, via OpenRouter). How used: AI coding assistant. I
supplied the repro from issue PyCalendar#2078 and the fix design (decode per
content line, restricted to the parameter section before the first
colon); the model drafted the patch, the regression tests and this
message from my instructions. I reviewed the change and verified it by
running the test suite.

Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
@rahul05ranjan
rahul05ranjan force-pushed the fix/charset-parameter-decoding branch from ea787e1 to d84f9ce Compare November 8, 2026 23:42
NEWLINE_BYTES = re.compile(rb"\r?\n")
# a legacy CHARSET parameter declares the encoding of one property value
CHARSET_PARAM_BYTES = re.compile(
rb";[ \t]*CHARSET[ \t]*=[ \t]*([^;:\r\n]+)", re.IGNORECASE

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please use the possessive quantifier.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your are right, but one can detect the python version used and use possessive quantifiers when available.

Comment thread src/icalendar/tests/test_encoding.py Outdated

def test_charset_text_inside_quoted_param_value_is_not_a_directive_issue_2078():
"""UTF-8 text with a quoted parameter value mentioning CHARSET= must
still decode as UTF-8, not as the encoding named in the quotes."""

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does this come from?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear, where in the RFC is this specified? §3.1 is about gramar, not the charset.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why aren't you using threads to answer? I'm personally aren't sure how this should be implemented because I do not find suficcient information on it. When CHARSET is specified, to what part does it apply? line, properties and value(s) or just value(s)? Please give references to a specification for that. Otherwise I would reject this or at least call for other maintainers.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Point taken on threading — answering here in this thread.

Answer: CHARSET applies to the value(s) of the single property that carries it — not to the whole line, and not to other lines or properties. The primary sources are the two versit consortium specs (both dated September 18, 1996) that define the parameter — vCalendar 1.0 and vCard 2.1 — and they use identical wording in §2.1 "Encoding Characteristics" → "Character Set":

The default character set is ASCII. The default character set can be overridden for an individual property value by using the "CHARSET" property parameter. This property parameter may be used on any property. However, the use of this parameter on some properties may not make sense. Any character set registered with the Internet Assigned Numbers Authority (IANA) can be specified by this property parameter. For example, ISO 8859-8 or the Latin/Hebrew character set is specified by: DESCRIPTION;CHARSET=ISO-8859-8:...

(the vCard 2.1 spec has the same sentence with ADR;CHARSET=ISO-8859-8:...)

So the ASCII default stays in force everywhere else; each property overrides it individually, and only for its value. Both grammars also pin the rest of the line to ASCII: param = ... / "CHARSET" [ws] "=" [ws] charsetval with charsetval = <a character set string as defined in Section 7.1 of RFC 1521>, and word = <any printable 7bit us-ascii except []=:., > — property names and parameter values are 7-bit ASCII tokens, so the declared charset is only meaningful for the value after the colon.

The parameter is not part of RFC 5545 at all (§3.1.4: "There is not a property parameter to declare the charset used in a property value. The default charset for an iCalendar stream is UTF-8"), and it disappeared from vCard after 2.1 — RFC 6350 Appendix A.2 lists it among removed features ("The CONTEXT and CHARSET parameters are no more."). That is why this PR treats it purely as legacy tolerance for pre-5545 files (issue #2078), never as an iCalendar feature.

Among other parsers, the clearest reference I found is vobject, which decodes the property value with the CHARSET parameter on its QUOTED-PRINTABLE path: self.value = codecs.decode(self.value.encode("utf-8"), "quoted-printable").decode(self.params['CHARSET'][0]) — vobject/base.py L339-342. libical and ical.js do not honor CHARSET at all (both assume UTF-8 per RFC 5545), so they neither contradict nor confirm the semantics.

What this PR does: for each content line whose parameter section (before the first colon) carries an unquoted ;CHARSET=, that line is decoded with the declared charset; every other line keeps the existing UTF-8 handling. Strictly per the specs the parameter covers only the value, but the implementation decodes the whole raw content line with it — which is indistinguishable from value-only decoding for every line that conforms to the grammar above: the name and parameters are 7-bit ASCII, and the charsets legacy files actually use (ISO-8859-x, windows-125x, KOI8-R, Shift_JIS, …) are ASCII-transparent, so the header decodes identically either way. The difference only shows for non-ASCII bytes inside the parameter section, which those grammars do not allow; there the PR prefers the declared charset over emitting U+FFFD. I am happy to narrow it to decode exactly the value after the colon if you prefer strict conformance over that tolerance.

One fix that came out of re-checking this against the sources: the helper's docstring claimed valid UTF-8 input is never rerouted through the CHARSET path, but the code had no such gate — a file that had been re-encoded to UTF-8 while keeping a vestigial CHARSET=ISO-8859-1 parameter had its UTF-8 bytes decoded as latin-1 and turned into mojibake. Fixed in 6d0241e: valid UTF-8 input now parses exactly as on main, and only input that is not valid UTF-8 as a whole gets the per-line CHARSET decoding; a regression test pins the vestigial-parameter case, and the quoted-parameter test now mixes in a latin-1 line so it still exercises the scan. Locally, the parsing/encoding suites give 12 failed / 538 passed against 12 failed / 536 passed on the parent commit — the 12 are the same pre-existing timezone-data host artifacts in both runs.

@rahul05ranjan

Copy link
Copy Markdown
Author

Thanks for the review @angatha — on both points:

Possessive quantifier: happy to switch the capture group to [^;:\r\n]++, with one compatibility note: possessive quantifiers require Python 3.11+, while pyproject.toml still declares requires-python = ">=3.10" and the CI matrix includes 3.10 (where the pattern would raise re.error at import). If a 3.10 drop is planned I'll apply it right away; otherwise the greedy form is safe in this position — the class already excludes the delimiters and nothing follows the group, so there is no backtracking blow-up to guard against. Which way would you like it?

"Where does this come from?" (test_charset_text_inside_quoted_param_value...): that case is one I constructed defensively — it is not from a real-world file or from issue #2078 itself. The scenario behind the PR (legacy CHARSET= declarations in pre-RFC-5545 Outlook/Lotus exports) comes from the issue; the quoted-value test pins a boundary the CHARSET scan could easily get wrong: a CHARSET= that occurs inside a quoted parameter value (here ALTREP="cid:x;CHARSET=ISO-8859-1") is parameter data, not a directive, so the property must still decode as UTF-8. That follows from the parameter grammar in RFC 5545 §3.1, where quoted strings may contain ";" and "=". If you'd rather see it dropped or replaced with a more realistic sample, happy to adjust.

…ariants

Review feedback on PyCalendar#2080 (thank you @angatha):

- RFC 5545 §3.1.4, not the §3.1 grammar, is the section on charset: it
  states "There is not a property parameter to declare the charset used
  in a property value. The default charset for an iCalendar stream is
  UTF-8". The quoted-parameter-value test now cites what actually
  supports it: §3.1's `quoted-string = DQUOTE *QSAFE-CHAR DQUOTE`
  (QSAFE-CHAR = any character except CONTROL and DQUOTE), which lets
  ";" and "=" appear inside a quoted parameter value, and §3.1.4, which
  is why an unquoted legacy CHARSET= is only ever tolerated (issue
  PyCalendar#2078), never conforming.
- The CHARSET parameter regex now uses a possessive quantifier where
  the running Python supports it, following the existing version-guard
  pattern from the XML property (prop/xml.py): Python 3.11+ gets
  `([^;:\r\n]++)`, Python 3.10 keeps the plain greedy form (the
  character class excludes the delimiters, so both match the same
  strings). The three issue PyCalendar#2078 tests are parametrized over both
  regexes so each runs everywhere it compiles.

Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
@rahul05ranjan

Copy link
Copy Markdown
Author

Thanks @angatha — both points addressed in f8467d3.

On the RFC question: you are right, and the section on charset is §3.1.4, "Character Set": "There is not a property parameter to declare the charset used in a property value. The default charset for an iCalendar stream is UTF-8." So to be clear in return: CHARSET is not a conforming iCalendar parameter at all — this PR does not implement an RFC 5545 feature, it tolerates the vCard-style legacy marker that pre-RFC 5545 producers (old Outlook/Lotus exports, issue #2078) left behind. What §3.1's grammar does define is the other half of that test: param-value = paramtext / quoted-string, with quoted-string = DQUOTE *QSAFE-CHAR DQUOTE where QSAFE-CHAR is any character except CONTROL and DQUOTE — so ";" and "=" may occur inside a quoted parameter value, which is exactly why ALTREP="cid:x;CHARSET=ISO-8859-1" must be read as one value and not as a charset directive. The test's docstring now cites both: §3.1.4 for why an unquoted CHARSET= is only ever tolerated, and the QSAFE-CHAR grammar for why the quoted occurrence is plain data.

On the possessive quantifier: done, following the existing version-guard pattern from the XML property (src/icalendar/prop/xml.py): on Python 3.11+ the pattern uses ([^;:\r\n]++), on 3.10 (still in requires-python) it keeps the plain greedy form — the character class excludes the delimiters, so both match the same strings. The three issue #2078 tests are parametrized over both regexes so each runs everywhere it can compile; on 3.13 both variants pass, and a small differential battery (quoted values, whitespace runs, empty and malformed inputs) shows the two regexes matching byte-identically.

Full suite on this machine with the change: 21,634 passed / 584 failed / 578 xfailed, against 21,631 passed / the same 584 failed / 578 xfailed on the parent commit — the 584 are the timezone-data/dateutil host artifacts that reproduce identically without my change.

The _decode_with_charset docstring promised that valid UTF-8 input is
never rerouted through the CHARSET parameter, but the code checked no
such gate: a file that was re-encoded to UTF-8 while keeping a vestigial
legacy CHARSET=ISO-8859-1 parameter decoded its UTF-8 bytes as latin-1
and turned non-ASCII text into mojibake, where the pre-CHARSET code
parsed it correctly.

Valid UTF-8 input (a conforming RFC 5545 stream) is now decoded as
before, so a vestigial CHARSET cannot override the charset the stream
actually uses. Only input that is not valid UTF-8 as a whole gets the
per-line legacy CHARSET decoding. The quoted-parameter test gains a
latin-1 DESCRIPTION line so it still exercises the CHARSET scan under
that gate, and a new regression test pins the vestigial-parameter case.

Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
@angatha

angatha commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

@niccokunzmann, @SashankBhamidi this sounds more like a separate tool to convert old files into an RFC5545 compatible encoding. In my view, merging this is definetly unspecified behavior for RFC5545 and not some kind of backwards compatibility. What is your opinion?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agentscan:automated-account AgentScan flagged this item as coming from an automated account, and it needs review ai-suspicion This contribution is possibly created with lots of AI help without enough human understanding.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Legacy CHARSET=ISO-8859-1 property parameter is ignored: non-UTF-8 .ics files parse to U+FFFD even though the file declares its encoding

2 participants