feat: support crawling postgrad training plans - #10
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces comprehensive support for crawling postgraduate training programs and course data from the HIT (Shenzhen) educational system. Key additions include a new crawl-postgrad CLI command, core logic for mapping and merging postgraduate courses, and auxiliary scripts for rebuilding specific major data. The review feedback identifies a potential data loss issue where existing mapping files could be overwritten rather than updated, points out a redundant variable assignment, and suggests using more concise Python idioms for dictionary sorting.
| def build_postgrad_mappings(bbhs: list[str], output_path: Path) -> dict[str, dict]: | ||
| """Fetch and persist postgrad major mappings for the given bbh values.""" | ||
| all_mappings: dict[str, dict] = {} | ||
|
|
||
| for bbh in bbhs: | ||
| logger.info(f"正在处理研究生版本: {bbh}") | ||
| raw_plans = get_postgrad_fah_list(bbh) | ||
| all_mappings[bbh] = build_postgrad_mapping(raw_plans) | ||
|
|
||
| output_path.parent.mkdir(parents=True, exist_ok=True) | ||
| with open(output_path, "w", encoding="utf-8") as f: | ||
| json.dump(all_mappings, f, ensure_ascii=False, indent=2) | ||
|
|
||
| return all_mappings |
There was a problem hiding this comment.
在 build_postgrad_mappings 函数中，代码直接初始化了一个空的 all_mappings 字典并将其写入 output_path。这会导致如果该文件已经存在其他版本（bbh）的数据，这些数据将被覆盖并丢失。考虑到该文件通常作为所有版本的汇总映射，建议在写入前先加载现有内容并进行合并更新。
| def build_postgrad_mappings(bbhs: list[str], output_path: Path) -> dict[str, dict]: | |
| """Fetch and persist postgrad major mappings for the given bbh values.""" | |
| all_mappings: dict[str, dict] = {} | |
| for bbh in bbhs: | |
| logger.info(f"正在处理研究生版本: {bbh}") | |
| raw_plans = get_postgrad_fah_list(bbh) | |
| all_mappings[bbh] = build_postgrad_mapping(raw_plans) | |
| output_path.parent.mkdir(parents=True, exist_ok=True) | |
| with open(output_path, "w", encoding="utf-8") as f: | |
| json.dump(all_mappings, f, ensure_ascii=False, indent=2) | |
| return all_mappings | |
| def build_postgrad_mappings(bbhs: list[str], output_path: Path) -> dict[str, dict]: | |
| """Fetch and persist postgrad major mappings for the given bbh values.""" | |
| all_mappings = load_postgrad_mappings(output_path) if output_path.exists() else {} | |
| for bbh in bbhs: | |
| logger.info(f"正在处理研究生版本: {bbh}") | |
| raw_plans = get_postgrad_fah_list(bbh) | |
| all_mappings[bbh] = build_postgrad_mapping(raw_plans) | |
| output_path.parent.mkdir(parents=True, exist_ok=True) | |
| with open(output_path, "w", encoding="utf-8") as f: | |
| json.dump(all_mappings, f, ensure_ascii=False, indent=2) | |
| return all_mappings |
| major_entry["major_code"] = major_code | ||
| raw_courses = _collect_courses_for_major(major_entry) |
There was a problem hiding this comment.
这一行赋值操作是多余的。major_entry 在 build_postgrad_mapping 中创建时已经包含了 major_code 字段。移除它可以使代码更简洁。
| major_entry["major_code"] = major_code | |
| raw_courses = _collect_courses_for_major(major_entry) | |
| raw_courses = _collect_courses_for_major(major_entry) |
| f.write(f'{key} = "{val}"\n') | ||
| else: | ||
| f.write(f"{key} = {val}\n") | ||
| info_data = {key: data["info"][key] for key in sorted(data["info"].keys())} |
There was a problem hiding this comment.
可以使用更简洁的 dict(sorted(data["info"].items())) 来对字典按键排序。由于项目支持 Python 3.14+，这种写法更符合现代 Python 惯用法。
| info_data = {key: data["info"][key] for key in sorted(data["info"].keys())} | |
| info_data = dict(sorted(data["info"].items())) |
|
LGTM. @Co-ding-Man Can you also update plans and mapping on https://github.com/HITSZ-OpenAuto/hoa-major-data? |
OK, but not soon. 😢 |
2 participants
📝 描述
✅ 检查清单
我已知晓并认同《参与指南》中的公约。
是否在合适的文件夹上传？
📢 通知成员
@kowyo
🤔 附加信息