Famedly release v1.154.0_1#271
Open
jason-famedly wants to merge 36 commits into
Open
Conversation
Based on #19708. This is on the path to porting the entire event class to Rust, as `event.content` will then return the new Rust class `JsonObject`. This PR adds a pure Rust `JsonObject` class that is a `Mapping` representing a json-style object. It uses `serde_json::Value` as its in-memory representation and `pythonize` for conversion when a field is looked up on the object. I'm not thrilled with the name, but couldn't think of a better one. This also adds `JsonObject` handling to the JSON serialisation functions we use, as well as to the `freeze(..)` function. Reviewable commit-by-commit.
…_INTERVAL` (#19772) There is no behavioral change, only a change to the tests. See element-hq/synapse#19772 (comment) for an explanation of why the tests needed changing (and diff comments). Follow-up to element-hq/synapse#19394. The test discussion originally happened in element-hq/synapse#19394 (comment) This is spawning from thinking about the problem again.
Add `event_id` so you can actually correlate everything together in the logs.
Convert `prev_state_events` to use `StrCollection` rather than requiring it to be a mutable list. None of the usages require it to be a proper list, and besides, events are immutable and therefore so should `event.prev_state_events`.
This is in prep for using the room versions more from Rust. Main changes: - Change it so each room version is defined as a delta to the last one. This is a cosmetic change that makes it easier to ensure the room version definitions are correct (as they're defined as deltas from previous versions). - Move constants to `RoomVersion` constants, like `RoomVersion::V1`, for convenience. - Change visibility of various attributes.
The reason for the change is to make it easier to support these checks when porting event class to Rust. Previously, code that needed to access `prev_state_events` had to combine a `room_version.msc4242_state_dags` boolean check with an `isinstance(event, FrozenEventVMSC4242)` cast (or `cast()`) for the type checker. Introduce `supports_msc4242_state_dag()` in a new `synapse/events/py_protocol.py` which does both in one step via `TypeIs[MSC4242Event]`, removing the need to import the concrete `FrozenEventVMSC4242` class at every call site. `MSC4242Event` is an `EventBase` subclass used purely for type narrowing — it's marked with a metaclass that rejects `isinstance()` to make accidental runtime use loud. No behavioural change: callers continue to gate on the same room version flag and access the same `prev_state_events` attribute.
…th unbounded token (#19644) Spawning from trying to find the proper way to wait for a token, see element-hq/synapse#19558 (comment) - Update `wait_for_stream_token(...)` patterns so validation/sanitization is handled upstream in usage. - Fix sync waiting for bounded token but using unbounded token to fetch data. Noticed while working on adding the new method. Part of element-hq/synapse#19647
When we port the `Event` class to Rust, the constructor will check for the existence of required fields. To support that, we tidy up the test code where we construct fake events to add all the required fields. There should be no behavioural changes. Review commit-by-commit.
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
… there are new results to sync (#19714)" (#19784) Reverts: #19714 Opens: #19783 Closes: element-hq/backend-internal#242 Related: #18880 (the performance problem that is aggravated by #19714) This reverts commit 2691d0b. --------- Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
…aim (#19508) Added details how synapse syncs the picture claim when update_profile_information setting is true. Addresses #17836 --------- Co-authored-by: Michael Hlas <3398654+mhlas7@users.noreply.github.com>
This isn't fixing any particular issue. It's just a follow-up I thought about after merging element-hq/synapse#19611 since we're now also dealing with backfill points in the nearby range ahead of the `current_depth`. And it's possible that the previous sort could bias to all nearby backfill points ahead of the `current_depth` that don't extend into the visible window of events we're paginating through.
…it_for_stream_token(...)` (#19764) In order to be able to use `wait_for_stream_token(...)`, we have to add the `quarantined_media` stream to the `StreamToken`. Even though we don't care about `/sync`'ing `quarantined_media`, this aligns with the future where all endpoints should probably use `StreamToken`, see element-hq/synapse#19647 Follow-up to element-hq/synapse#19558 and element-hq/synapse#19644
Fixes: #19689 # What This PR fixes a bug I found when I run synapse (from dockerhub) and register a `check_event_allowed` callback and my client makes use of the mentions field in messages (`cinny:latest`). The bug doesn't appear when the `check_event_allowed` callback is not loaded. After some digging I noticed that the current validation of the mentions doesn't work when an event has been frozen with `event.freeze()`. For the messages this seems to happen when a the `check_event_allowed` is registered (but not otherwise), see [where the event is frozen for check_event_allowed callback](https://github.com/element-hq/synapse/blob/b0fc0b7a612a42e6f15b87dee2a1db4c383645fb/synapse/module_api/callbacks/third_party_event_rules_callbacks.py#L289) and [where the validation function is called](https://github.com/element-hq/synapse/blob/b0fc0b7a612a42e6f15b87dee2a1db4c383645fb/synapse/handlers/message.py#L1404). To have a minimal reproduction example, the following scripts fails on `develop` but succeeds in this branch: ``` python from synapse.api.room_versions import RoomVersions from synapse.events import EventBase, make_event_from_dict from synapse.events.validator import EventValidator from tests.utils import default_config def make_message_event(content: dict) -> EventBase: return make_event_from_dict( { "room_id": "!room:test", "type": "m.room.message", "sender": "@alice:test", "content": content, "auth_events": [], "prev_events": [], "hashes": {"sha256": "aGVsbG8="}, "signatures": {}, "depth": 1, "origin_server_ts": 1000, }, room_version=RoomVersions.V9, ) event = make_message_event( { "msgtype": "m.text", "body": "@moderator:example.com hello", "m.mentions": {"user_ids": ["@moderator:jailbreak-challenge.aqtiveguard.com"]}, } ) EventValidator().validate_new(event, default_config) # Ok event.freeze() EventValidator().validate_new(event, default_config) # throws # pydantic_core._pydantic_core.ValidationError: 1 validation error for Mentions # Input should be a valid dictionary or instance of Mentions [type=model_type, input_value=immutabledict({'user_ids'...nge.aqtiveguard.com',)}), input_type=immutabledict] # For further information visit https://errors.pydantic.dev/2.12/v/model_type ``` # How I made the validation logic also validate the transformation performed by the freezing process, namely: - `immutabledict` validates as `dict`. (was already implemented for POWER_LEVELS) - `tuple` validates as array (added this to the validator in this PR). --------- Co-authored-by: Eric Eastwood <madlittlemods@gmail.com> Co-authored-by: Olivier 'reivilibre <oliverw@matrix.org>
…nc request filters out Ephemeral Data Units (EDUs). (#19787) Fixes: #19779 Fixes: element-hq/synapse#19618 See also: #19786 (which would have caught this, but currently has too many findings to enable) Fix UnboundLocalError when MSC4354 is enabled in sync and all EDUs are filtered out --------- Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
Signed-off-by: dependabot[bot] <support@github.com>
when an access token had a refresh token associated to it in the database, deleting this refresh token (for example when deleting the device using it) would cascade delete the access token, which wouldn't be returned by the sql query that was supposed to delete it on its own, and an empty array was passed to the cache invalidation function.
Accidental bump broke build for Fedora and RHEL. This reverts commit 2e9d6f7. As discussed in the [Synapse Package Maintainers](https://matrix.to/#/!rh9Uxk45AsPongyP3ypgpsCmuIufiggD6mDXFWh4_FM/$0mdulZEyJFdI6bwS8GFwYnFt-zmpyCyx2DwcA8JyuY8?via=jki.re&via=matrix.org&via=element.io) room (private)
As per the spec, a room with m.room.name value that is absent, null or empty should be treated as if there is no m.room.name event at all: https://spec.matrix.org/v1.17/client-server-api/#mroomname This fetches the full m.room.name event and checks the content.name instead of only checking the existence of the m.room.name event. This results in correctly sending heroes for those rooms. Fixes: element-hq/synapse#19447 Signed-off-by: Joe Groocock <me@frebib.net>
Signed-off-by: dependabot[bot] <support@github.com>
…(#19791) Follows: #19468 The main change is from this comment element-hq/synapse#19468 (comment) I am pretty sure it's safe and was tempted to add it to that PR, but for easier bisection and reversion in case it goes wrong, thought a separate commit would be the best. The other drive-by change is a boolean logic simplification Simplify condition (boolean equivalence) Don't fetch name state from `meta_room_state` since it's no longer used there --------- Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
These updates are handled and maintained by upstream, and to prevent having to re-delete a file each time there is a new release, just disable the system in the file so no new pull requests are opened.
Contributor
There was a problem hiding this comment.
Pull request overview
Updates this fork to Synapse 1.154.0 (Famedly release v1.154.0_1), pulling in upstream changes around Rust-backed event content handling, stream token handling, and MSC4452 URL preview capability behavior, plus accompanying test and documentation updates.
Changes:
- Port
Event.contentto a RustJsonObjectmapping and update serialization/validation paths accordingly. - Extend sync stream tokens (incl. quarantined media), and standardize “catch up to token” waiting via
Notifier.wait_for_stream_token. - Adjust URL preview endpoints/capabilities behavior (MSC4452) and update tests/docs for the new behaviors.
Reviewed changes
Copilot reviewed 90 out of 91 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_utils/event_builders.py | New helpers for building test events/PDUs with required defaults. |
| tests/test_utils/init.py | Use json_encoder for deterministic JSON in fake responses. |
| tests/test_state.py | Switch test event creation to make_test_event. |
| tests/test_notifier.py | New tests for Notifier.wait_for_stream_token. |
| tests/test_event_auth.py | Switch event construction to make_test_event. |
| tests/synapse_rust/test_json_object.py | New unit tests for Rust JsonObject mapping behavior. |
| tests/storage/test_sliding_sync_tables.py | Use make_test_event for invite/kick test events. |
| tests/storage/test_msc4242_state_dag.py | Update MSC4242 typing/narrowing to protocol helper. |
| tests/storage/test_events.py | Use make_test_pdu_event helper for federation PDUs. |
| tests/storage/databases/main/test_lock.py | Update lock timeout constants to Duration API. |
| tests/storage/databases/main/test_events_worker.py | Use make_test_event and store canonical JSON from event dict. |
| tests/state/test_v21.py | Add missing hashes field to test event dict. |
| tests/state/test_v2.py | Switch event construction to make_test_event. |
| tests/server.py | Use json_encoder when encoding request bodies. |
| tests/rest/client/test_third_party_rules.py | Add missing hashes in federation fixture. |
| tests/rest/client/test_rooms.py | Adjust typing to JsonMapping; update token fixture strings. |
| tests/rest/client/test_media.py | Add tests for preview URL behavior when previews disabled. |
| tests/rest/client/test_capabilities.py | Add MSC4452 capability tests; gate enabled case on lxml. |
| tests/rest/client/test_auth.py | Update refresh-token invalidation test to use /whoami. |
| tests/rest/client/sliding_sync/test_sliding_sync.py | Switch event construction to make_test_event. |
| tests/rest/client/sliding_sync/test_rooms_meta.py | Add regression test for “empty room name” heroes behavior. |
| tests/rest/admin/test_room.py | Update token fixture strings. |
| tests/push/test_push_rule_evaluator.py | Switch event creation to make_test_event; adjust expected flattening. |
| tests/push/test_presentable_names.py | Switch from FrozenEvent to make_test_event usage and typing. |
| tests/module_api/test_api.py | Account for Rust JsonObject by coercing event.content to dict. |
| tests/handlers/test_worker_lock.py | Strengthen worker lock tests around stale locks and retry interval. |
| tests/handlers/test_sync.py | Use make_test_pdu_event; make waiting deterministic with reactor.advance. |
| tests/handlers/test_room_summary.py | Switch event construction to make_test_event. |
| tests/handlers/test_room_policy.py | Fix policy-server signature merging tests; use Rust signatures type. |
| tests/handlers/test_room_member.py | Use make_test_pdu_event for federation membership events. |
| tests/handlers/test_federation.py | Use test event/PDU builders across federation handler tests. |
| tests/handlers/test_federation_event.py | Use make_test_event for pulled/prev/state events. |
| tests/federation/transport/test_client.py | Add required fields to parser test fixtures. |
| tests/federation/test_federation_server.py | Use test event/PDU builders; update strip-event assertions. |
| tests/federation/test_federation_out_of_band_membership.py | Use test event/PDU builders in OOB membership tests. |
| tests/federation/test_federation_client.py | Add missing depth fields to fixtures. |
| tests/federation/test_federation_base.py | Add required auth_events/prev_events/depth to fixture. |
| tests/events/test_validator.py | New test ensuring validator accepts frozen events with m.mentions. |
| tests/events/test_utils.py | Replace MockEvent with make_test_event; adjust prune expectations. |
| tests/events/test_py_protocol.py | New tests for EventProtocol type narrowing helpers. |
| tests/events/test_auto_accept_invites.py | Use make_test_pdu_event for invite PDUs. |
| tests/crypto/test_event_signing.py | Switch to make_test_event; add missing content; update expected sig/hash. |
| tests/api/test_filtering.py | Switch from MockEvent to make_test_event. |
| synapse/util/json.py | Extend JSON encoding support to JsonObject and refactor mapping handler. |
| synapse/util/frozenutils.py | Treat JsonObject as already-frozen. |
| synapse/util/events.py | Accept JsonMapping for topic extraction utility. |
| synapse/types/init.py | Add quarantined media stream key/token field and optimize token bounding. |
| synapse/synapse_rust/events.pyi | Add JsonObject typing stub. |
| synapse/streams/events.py | Include quarantined media stream in current token; add bounding diagnostics. |
| synapse/storage/databases/main/room.py | Expose quarantined media stream token/generator; remove bespoke wait helper. |
| synapse/storage/databases/main/registration.py | Reorder/clarify comments around token deletion and cache invalidation. |
| synapse/storage/databases/main/lock.py | Replace millisecond timeout constant with Duration-based values. |
| synapse/storage/databases/main/events.py | Use MSC4242 type-narrowing helper; update stored-edge typing. |
| synapse/storage/controllers/persist_events.py | Use MSC4242 type-narrowing helper for state-DAG persistence path. |
| synapse/state/init.py | Use MSC4242 type-narrowing helper for prev-state selection. |
| synapse/rest/media/preview_url_resource.py | Return 403/404 based on MSC4452 + preview disabled, instead of asserting previewer. |
| synapse/rest/media/media_repository_resource.py | Always register preview servlet; runtime handles disabled previewer. |
| synapse/rest/client/media.py | Mirror preview URL disabled behavior for client servlet. |
| synapse/rest/client/capabilities.py | Expose io.element.msc4452.preview_url capability when enabled. |
| synapse/rest/admin/media.py | Use notifier token-waiting for quarantined media pagination correctness. |
| synapse/push/bulk_push_rule_evaluator.py | Tighten typing for power-level event content mapping. |
| synapse/notifier.py | Expand docs for token catch-up semantics; remove internal bounding workaround. |
| synapse/handlers/worker_lock.py | Clarify docs around stale lock behavior. |
| synapse/handlers/sync.py | Bound future tokens earlier; fix now-token propagation for EDUs/sticky events. |
| synapse/handlers/stats.py | Update typing for event content mapping. |
| synapse/handlers/sliding_sync/init.py | Fix heroes behavior for empty room name; bound invalid future tokens. |
| synapse/handlers/room.py | Coerce JSON mappings to dicts for module callbacks; update creation content typing. |
| synapse/handlers/room_policy.py | Preserve existing signatures by adding policy signatures individually. |
| synapse/handlers/message.py | Use MSC4242 helper and StrCollection typing for prev state events. |
| synapse/handlers/federation.py | Prefer backfill points closer to current depth. |
| synapse/handlers/federation_event.py | Improve logging context/details around missing prev events. |
| synapse/handlers/admin.py | Use MSC4242 helper for prev-state handling in redactions. |
| synapse/events/validator.py | Refactor schema error conversion; add mentions schema validation. |
| synapse/events/utils.py | Ensure stripped event content is a plain dict (not JsonObject). |
| synapse/events/py_protocol.py | New type-narrowing helpers (TypeIs) for MSC4242 event attributes. |
| synapse/events/builder.py | Use StrCollection for prev_state_events. |
| synapse/events/init.py | Store content as Rust JsonObject; adjust freezing/interning/serialization. |
| synapse/event_auth.py | Use MSC4242 helper for auth rule checks. |
| synapse/config/repository.py | Force url_preview_enabled to boolean. |
| synapse/config/experimental.py | Add config flag for MSC4452 capability exposure. |
| synapse/init.py | Register canonicaljson serialization for JsonObject. |
| schema/synapse-config.schema.yaml | Bump schema $id version and extend SSO profile-sync docs. |
| rust/src/room_versions.rs | Refactor room version definitions and Python interop behavior. |
| rust/src/events/mod.rs | Register Rust JsonObject and its views as Python mappings. |
| rust/src/events/json_object.rs | New Rust implementation of immutable JSON mapping + views. |
| pyproject.toml | Bump Synapse version; relax attrs minimum version back to 19.2.0. |
| docs/usage/configuration/config_documentation.md | Update SSO profile-sync documentation (displayname + picture). |
| debian/changelog | Add Debian changelog entries for 1.154.0 and rc1. |
| CHANGES.md | Add Synapse 1.154.0/rc1 changelog section. |
| .github/dependabot.yml | Reduce dependabot PR limit to 0 for multiple ecosystems. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #271 +/- ##
=======================================
Coverage 80.41% 80.41%
=======================================
Files 501 502 +1
Lines 72403 72430 +27
Branches 10912 10917 +5
=======================================
+ Hits 58221 58248 +27
+ Misses 10906 10901 -5
- Partials 3276 3281 +5
... and 3 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
jason-famedly
force-pushed
the
famedly-release/v1.154
branch
from
e4852be to
93e5270
Compare
Member
Author
|
Sytest test failure is a known flake |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Famedly Release v1.154.0_1
Docker image tags available:
v1.154.0_1-mod028<- TIM 1.1v1.154.0_1-mod029<- TIM Pro(and TIM 1.2)Famedly additions for v1.154.0_1