fix(chat): secure and wire Open Graph link preview fetching

- Block non-public URLs (localhost, private IPs) from Open Graph fetches by
  default; add chat.open_graph_allow_private_networks for tests/dev.
- Disable redirects, cap response body size via streaming, and reject
  oversized Content-Length before buffering.
- Wire preview refresh into send_message so URLs in message content are
  fetched/cached automatically.
- Add tests for private-URL rejection, URL extraction, and the existing
  fetch/cache behavior.
- Update docs/refactoring/chat.md.
- Bump version to 0.1.115.

Signed-off-by: Eli Ma <eli@patch.sh>

Author: genedna

Cargo.lock

@@ -3,7 +3,7 @@ members = ["bin"]
 
 [package]
 name = "monoengine-core"
-version = "0.1.114"
+version = "0.1.115"
 edition = "2024"
 
 [lib]

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "monoengine"
-version = "0.1.114"
+version = "0.1.115"
 edition = "2024"
 
 # The thin composition-root binary. It depends on `monoengine-core` (which only

bin/Cargo.toml

@@ -360,6 +360,8 @@ attachment_allowed_mime_types = []
 open_graph_fetch_enabled = true
 # Network timeout in milliseconds for a single Open Graph fetch attempt.
 open_graph_fetch_timeout_ms = 5000
+# Allow fetches against localhost/private IPs. Keep false in production.
+open_graph_allow_private_networks = false
 
 [vault.audit]
 # Secret-access auditing (docs/vault.md stage H). When enabled (the default),

config/config.toml

@@ -14,14 +14,14 @@
 2. **SeaORM 实体与迁移已落地**。`attachments`、`custom_reactions`、`open_graph_links`、`channels`、`channel_memberships`、`channel_membership_updates`、`messages`、`message_notifications` 等实体/迁移存在；`reactions` 复用并扩展既有表。
 3. **HTTP Router 已接入**。`src/api/router/chat_router.rs` 已在 `src/api/api_router.rs` merge，提供 channel CRUD、message CRUD、reaction、attachment presign/confirm、read/unread 端点并带 OpenAPI 标注。
 4. **权限主干已实现并在本轮收紧**。channel list/detail/message list/send/reaction/attachment 等路径会校验 membership；2026-06-23 新增 message edit/delete 的 channel path 校验和当前 membership 校验，避免只凭 sender ownership 跨 channel path 或被移除成员继续写旧消息；同日 `chat_router` 的 message/custom-reaction 映射读取已下沉到 storage helper，减少 handler 直接 SeaORM 查询。
-5. **仍未完成**：外部数据迁移工具的真实源库导入/校验、WebSocket/Pusher 兼容网关、完整真实 HTTP 黑盒矩阵和外部通知投递。message notification 内部状态已完成首批 reply 与 `@username` mention 写入；更完整 rich-text mention 语义仍属于后续。附件 presign/confirm 已有首批 file name/type/size/object-key 校验；**2026-06-29 更新：产品级 MIME allowlist 已落地**——`[chat]` 配置新增 `attachment_allowed_mime_types`，支持精确类型（`image/png`）与子类型通配（`image/*`），并在 presign/confirm 两阶段校验；**2026-06-29 更新（二）：已上传对象存在性复核已落地**——`confirm_attachment` 端点在注册附件前调用 object storage 校验 `Attachment` 命名空间下目标对象是否存在，不存在则返回 400；**2026-06-29 更新（三）：链接预览抓取/缓存行为已落地**——`SharedChatService::fetch_or_refresh_open_graph_link` 按 URL 读取本地缓存，过期时通过 HTTP 拉取并解析 `og:title`/`og:image`/favicon，写回 `open_graph_links`，配置 `chat.open_graph_fetch_enabled`/`open_graph_fetch_timeout_ms` 支持热加载。实时事件已有 `NoopChatEvents` 默认实现和 `InMemoryChatEvents` 进程内 broadcast hub，可供测试与后续网关消费。mark unread 已按验收标准把 `last_read_at` 调到 latest message 之前（2026-06-23 补齐）。
+5. **仍未完成**：外部数据迁移工具的真实源库导入/校验、WebSocket/Pusher 兼容网关、完整真实 HTTP 黑盒矩阵和外部通知投递。message notification 内部状态已完成首批 reply 与 `@username` mention 写入；更完整 rich-text mention 语义仍属于后续。附件 presign/confirm 已有首批 file name/type/size/object-key 校验；**2026-06-29 更新：产品级 MIME allowlist 已落地**——`[chat]` 配置新增 `attachment_allowed_mime_types`，支持精确类型（`image/png`）与子类型通配（`image/*`），并在 presign/confirm 两阶段校验；**2026-06-29 更新（二）：已上传对象存在性复核已落地**——`confirm_attachment` 端点在注册附件前调用 object storage 校验 `Attachment` 命名空间下目标对象是否存在，不存在则返回 400；**2026-06-29 更新（三）：链接预览抓取/缓存行为已落地**——`send_message` HTTP 路径在消息发送后会从 content 中提取 http/https URL，调用 `SharedChatService::fetch_or_refresh_open_graph_link` 按 URL 读取本地缓存，过期时通过 HTTP 拉取并解析 `og:title`/`og:image`/favicon，写回 `open_graph_links`；配置 `chat.open_graph_fetch_enabled`/`open_graph_fetch_timeout_ms` 支持热加载。抓取默认禁用重定向、限制响应体大小，并拒绝 localhost/私有 IP（测试/隔离环境可通过 `chat.open_graph_allow_private_networks` 开启）。实时事件已有 `NoopChatEvents` 默认实现和 `InMemoryChatEvents` 进程内 broadcast hub，可供测试与后续网关消费。mark unread 已按验收标准把 `last_read_at` 调到 latest message 之前（2026-06-23 补齐）。
 
 ## 当前实现状态速览表
 
 | 能力 / 组件 | 实现状态 | 关键事实与风险 |
 |-----------|--------|-------------|
 | Chat 模块入口 | 已实现主干 | `src/chat/` 下已有 domain/engine/service；仍需继续清理文档中的旧 slice 叙述。 |
-| Shared Foundations（附件、表情） | 部分实现 | attachment/reaction/custom reaction/open graph 的实体、迁移、storage 与 service 主路径已落地；附件 presign/confirm 已补 file name/type/size/object-key 首批校验，并新增 `Config.chat.attachment_allowed_mime_types` 产品级 MIME allowlist（支持精确类型与子类型通配，空列表保持向后兼容）；custom_reactions 已补 `lower(name)` 唯一索引和应用层 lowercase；reactions 已改为 `WHERE discarded_at IS NULL` 部分唯一索引；attachment 已补 `discarded_at` 软删除（满足硬约束 #4）；**2026-06-29 更新：已上传附件对象存在性复核已落地**，`confirm_attachment` 注册前会校验 object storage 中目标对象是否存在；**2026-06-29 更新（三）：链接预览抓取/缓存行为已落地**——`SharedChatService::fetch_or_refresh_open_graph_link` 按 URL 读取本地缓存，未命中或缓存过期时通过 HTTP 拉取 HTML，解析 `og:title`/`og:image`/favicon 后写回 `open_graph_links`，支持 `chat.open_graph_fetch_enabled` 与 `chat.open_graph_fetch_timeout_ms` 配置并在热加载时生效。 |
+| Shared Foundations（附件、表情） | 部分实现 | attachment/reaction/custom reaction/open graph 的实体、迁移、storage 与 service 主路径已落地；附件 presign/confirm 已补 file name/type/size/object-key 首批校验，并新增 `Config.chat.attachment_allowed_mime_types` 产品级 MIME allowlist（支持精确类型与子类型通配，空列表保持向后兼容）；custom_reactions 已补 `lower(name)` 唯一索引和应用层 lowercase；reactions 已改为 `WHERE discarded_at IS NULL` 部分唯一索引；attachment 已补 `discarded_at` 软删除（满足硬约束 #4）；**2026-06-29 更新：已上传附件对象存在性复核已落地**，`confirm_attachment` 注册前会校验 object storage 中目标对象是否存在；**2026-06-29 更新（三）：链接预览抓取/缓存行为已落地**——`send_message` HTTP 路径会提取消息中的 http/https URL，调用 `SharedChatService::fetch_or_refresh_open_graph_link` 按 URL 读取本地缓存，过期时通过 HTTP 拉取 HTML，解析 `og:title`/`og:image`/favicon 后写回 `open_graph_links`；支持 `chat.open_graph_fetch_enabled`/`open_graph_fetch_timeout_ms` 配置并在热加载时生效；默认禁用重定向、限制响应体大小、拒绝 localhost/私有 IP，测试/隔离环境可通过 `chat.open_graph_allow_private_networks` 开启。 |
 | Channel Chat（频道、消息） | 部分实现 | channel/message/membership 实体、迁移、storage、service 已落地；create/send/edit/delete/read/unread/member service 主路径可用；reply 与 `@username` mention message notification 内部状态已写入；**2026-06-29 更新**：`@username` mention 的外部邮件投递已接入 `send_message` HTTP 路径，通过 `notification::triggers::on_chat_mention_created` 按用户偏好入队 `chat.mention.created` email job。**2026-06-29 更新（二）**：reply 的外部邮件投递也已接入，通过 `notification::triggers::on_chat_reply_created` 向被回复消息的作者入队 `chat.reply.created` email job。rich-text mention 更复杂语义（如 markdown/HTML 解析、非 ASCII handle）仍为后续。channels/channel_memberships/channel_membership_updates/messages/message_notifications 的完整索引集和 `message_notifications` 唯一约束已补齐。 |
 | HTTP API | 部分实现 | `chat_router` 已挂载，DTO/OpenAPI 标注存在；仍缺真实 HTTP 黑盒矩阵、成员管理 HTTP 端点是否暴露的产品决策，以及更完整错误码兼容性。 |
 | 实时事件 | 进程内 broadcaster 已实现 | `ChatEvents`/`NoopChatEvents` 已定义并由 service 调用；`InMemoryChatEvents` 已提供 tokio broadcast 订阅能力并覆盖 service mutation 事件。WebSocket/Pusher 兼容网关仍未实现。 |

docs/refactoring/chat.md

@@ -138,6 +138,22 @@ fn validate_chat_attachment_file_path(file_path: &str) -> Result<(), ApiError> {
     Ok(())
 }
 
+fn extract_urls(content: &str) -> Vec<String> {
+    let mut urls = Vec::new();
+    for token in content.split_whitespace() {
+        let trimmed = token.trim_matches(|c: char| {
+            matches!(
+                c,
+                '.' | ',' | ';' | ':' | '!' | '?' | ')' | ']' | '}' | '"' | '\'' | '>'
+            )
+        });
+        if trimmed.starts_with("http://") || trimmed.starts_with("https://") {
+            urls.push(trimmed.to_string());
+        }
+    }
+    urls
+}
+
 async fn map_channel_model(
     ch: crate::callisto::channel::Model,
     state: &MonoApiServiceState,
@@ -552,6 +568,26 @@ async fn send_message(
         }
     }
 
+    // Refresh Open Graph previews for any URLs in the message content. Failures
+    // are logged but do not block the send response.
+    let chat_cfg = state.storage.config().chat.clone().unwrap_or_default();
+    if chat_cfg.open_graph_fetch_enabled {
+        for url in extract_urls(&payload.content) {
+            if let Err(e) = state
+                .shared_chat_svc()
+                .fetch_or_refresh_open_graph_link(
+                    &url,
+                    true,
+                    chat_cfg.open_graph_fetch_timeout_ms,
+                    chat_cfg.open_graph_allow_private_networks,
+                )
+                .await
+            {
+                tracing::warn!(error = %e, url = %url, "failed to refresh open graph link");
+            }
+        }
+    }
+
     let mapped = map_message_model(msg, &state).await?;
     Ok(Json(CommonResult::success(Some(mapped))))
 }
@@ -1371,4 +1407,15 @@ mod tests {
         .0;
         assert!(del_ch_res.req_result);
     }
+
+    #[test]
+    fn extract_urls_finds_http_and_https_tokens() {
+        let content = "Check out https://example.com/page, and http://test.org?q=1. Also \
+            https://else.where/path.";
+        let urls = extract_urls(content);
+        assert_eq!(urls.len(), 3);
+        assert!(urls.contains(&"https://example.com/page".to_string()));
+        assert!(urls.contains(&"http://test.org?q=1".to_string()));
+        assert!(urls.contains(&"https://else.where/path".to_string()));
+    }
 }

src/api/router/chat_router.rs

@@ -1,7 +1,10 @@
-use std::{sync::LazyLock, time::Duration};
+use std::{net::IpAddr, sync::LazyLock, time::Duration};
 
+use bytes::BytesMut;
 use chrono::Utc;
+use futures::TryStreamExt;
 use regex::Regex;
+use reqwest::redirect::Policy;
 use sea_orm::{ColumnTrait, EntityTrait, QueryFilter};
 
 use crate::{
@@ -229,6 +232,7 @@ impl SharedChatService {
         url: &str,
         fetch_enabled: bool,
         timeout_ms: u64,
+        allow_private_networks: bool,
     ) -> Result<Option<open_graph_link::Model>, MegaError> {
         let cached = self
             .open_graph_storage
@@ -245,8 +249,11 @@ impl SharedChatService {
         if !fetch_enabled {
             return Ok(cached);
         }
+        if !allow_private_networks {
+            validate_preview_url(url, false)?;
+        }
 
-        let fetched = match fetch_open_graph(url, timeout_ms).await {
+        let fetched = match fetch_open_graph(url, timeout_ms, allow_private_networks).await {
             Ok(v) => v,
             Err(e) => {
                 tracing::warn!(url = %url, error = %e, "failed to fetch open graph link");
@@ -273,24 +280,27 @@ struct FetchedOpenGraph {
     favicon: Option<String>,
 }
 
-async fn fetch_open_graph(url: &str, timeout_ms: u64) -> Result<FetchedOpenGraph, MegaError> {
+async fn fetch_open_graph(
+    url: &str,
+    timeout_ms: u64,
+    allow_private_networks: bool,
+) -> Result<FetchedOpenGraph, MegaError> {
+    let parsed = validate_preview_url(url, allow_private_networks)?;
     let client = reqwest::Client::builder()
         .timeout(Duration::from_millis(timeout_ms))
+        .redirect(Policy::none())
         .user_agent("monoengine-open-graph/1.0")
         .build()
         .map_err(|e| MegaError::Other(format!("failed to build http client: {e}")))?;
 
-    let body = client
-        .get(url)
+    let response = client
+        .get(parsed)
         .send()
         .await
-        .map_err(|e| MegaError::Other(format!("open graph fetch failed: {e}")))?
-        .text()
-        .await
-        .map_err(|e| MegaError::Other(format!("open graph fetch body read failed: {e}")))?;
+        .map_err(|e| MegaError::Other(format!("open graph fetch failed: {e}")))?;
 
-    // Avoid parsing multi-megabyte pages for a handful of meta tags.
-    let body: String = body.chars().take(1_000_000).collect();
+    const MAX_BODY_BYTES: usize = 1_000_000;
+    let body = read_response_body_limited(response, MAX_BODY_BYTES).await?;
 
     let title = extract_meta_property(&body, "og:title").or_else(|| extract_title_tag(&body));
     let image = extract_meta_property(&body, "og:image");
@@ -303,6 +313,98 @@ async fn fetch_open_graph(url: &str, timeout_ms: u64) -> Result<FetchedOpenGraph
     })
 }
 
+fn validate_preview_url(
+    url: &str,
+    allow_private_networks: bool,
+) -> Result<reqwest::Url, MegaError> {
+    let parsed = reqwest::Url::parse(url)
+        .map_err(|e| MegaError::Other(format!("invalid open graph URL: {e}")))?;
+    if !matches!(parsed.scheme(), "http" | "https") {
+        return Err(MegaError::Other(
+            "only http/https URLs are allowed for open graph previews".to_string(),
+        ));
+    }
+
+    if !allow_private_networks {
+        let host = parsed
+            .host_str()
+            .ok_or_else(|| MegaError::Other("open graph URL has no host".to_string()))?;
+        if host.eq_ignore_ascii_case("localhost") || host.ends_with(".localhost") {
+            return Err(MegaError::Other(
+                "localhost URLs are not allowed for open graph previews".to_string(),
+            ));
+        }
+        if let Ok(ip) = host.parse::<IpAddr>()
+            && !is_public_ip(ip)
+        {
+            return Err(MegaError::Other(
+                "non-public IP URLs are not allowed for open graph previews".to_string(),
+            ));
+        }
+    }
+
+    Ok(parsed)
+}
+
+fn is_public_ip(ip: IpAddr) -> bool {
+    match ip {
+        IpAddr::V4(v4) => {
+            let octets = v4.octets();
+            // 10/8, 172.16/12, 192.168/16
+            if octets[0] == 10
+                || (octets[0] == 172 && (16..=31).contains(&octets[1]))
+                || (octets[0] == 192 && octets[1] == 168)
+            {
+                return false;
+            }
+            // 127/8, 169.254/16, 224/4, 0/8, 255/8, 100.64/10, 198.18/15, 192.0.2/24, ...
+            !(v4.is_loopback()
+                || v4.is_link_local()
+                || v4.is_multicast()
+                || v4.is_unspecified()
+                || v4.is_broadcast()
+                || v4.is_documentation())
+        }
+        IpAddr::V6(v6) => {
+            !(v6.is_loopback()
+                || v6.is_unspecified()
+                || v6.is_multicast()
+                || v6.is_unicast_link_local()
+                || (v6.segments()[0] & 0xfe00) == 0xfc00) // unique local (fc00::/7)
+        }
+    }
+}
+
+async fn read_response_body_limited(
+    response: reqwest::Response,
+    limit: usize,
+) -> Result<String, MegaError> {
+    if let Some(len) = response.content_length()
+        && len > limit as u64
+    {
+        return Err(MegaError::Other(
+            "open graph response body exceeds size limit".to_string(),
+        ));
+    }
+
+    let mut stream = response.bytes_stream();
+    let mut buf = BytesMut::new();
+    while let Some(chunk) = stream
+        .try_next()
+        .await
+        .map_err(|e| MegaError::Other(format!("open graph response stream error: {e}")))?
+    {
+        if buf.len() + chunk.len() > limit {
+            return Err(MegaError::Other(
+                "open graph response body exceeds size limit".to_string(),
+            ));
+        }
+        buf.extend_from_slice(&chunk);
+    }
+
+    Ok(String::from_utf8_lossy(&buf).into_owned())
+}
+
 fn extract_meta_property(html: &str, property: &str) -> Option<String> {
     for caps in META_PROPERTY_FIRST.captures_iter(html) {
         if caps[1].eq_ignore_ascii_case(property) {
@@ -460,7 +562,7 @@ mod tests {
         let shared_svc = SharedChatService::from_storage(&storage);
 
         let preview = shared_svc
-            .fetch_or_refresh_open_graph_link(&url, true, 5000)
+            .fetch_or_refresh_open_graph_link(&url, true, 5000, true)
             .await
             .expect("fetch should succeed")
             .expect("preview should be returned");
@@ -476,7 +578,7 @@ mod tests {
 
         // A second fetch should return the cached row without fetching again.
         let cached = shared_svc
-            .fetch_or_refresh_open_graph_link(&url, true, 5000)
+            .fetch_or_refresh_open_graph_link(&url, true, 5000, true)
             .await
             .expect("cached fetch should succeed")
             .expect("cached preview should exist");
@@ -494,7 +596,7 @@ mod tests {
         let shared_svc = SharedChatService::from_storage(&storage);
 
         let none = shared_svc
-            .fetch_or_refresh_open_graph_link(&url, false, 5000)
+            .fetch_or_refresh_open_graph_link(&url, false, 5000, true)
             .await
             .expect("fetch should succeed");
         assert!(none.is_none());
@@ -506,10 +608,22 @@ mod tests {
             .await
             .expect("upsert");
         let cached = shared_svc
-            .fetch_or_refresh_open_graph_link(&url, false, 5000)
+            .fetch_or_refresh_open_graph_link(&url, false, 5000, true)
             .await
             .expect("fetch should succeed")
             .expect("cached entry should be returned");
         assert_eq!(cached.title, "Cached");
     }
+
+    #[tokio::test]
+    async fn fetch_open_graph_rejects_private_urls_by_default() {
+        let temp = tempfile::tempdir().unwrap();
+        let storage = test_storage(temp.path()).await;
+        let shared_svc = SharedChatService::from_storage(&storage);
+
+        let result = shared_svc
+            .fetch_or_refresh_open_graph_link("http://127.0.0.1:8080/", true, 5000, false)
+            .await;
+        assert!(result.is_err());
+    }
 }