changelog: 2026-08-21 - #138
Open
Yash1hi wants to merge 2 commits into
Open
Conversation
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
Contributor
🖼️ Image Usage Check ✅All 247 images in the repository are being used! No unused images found. Great job keeping the repository clean! 🎉 |
TwoTau
reviewed
Aug 24, 2026
| <Update label="August 21, 2026"> | ||
| ### Compare three records at once | ||
|
|
||
| The **Compare** page takes a third record. Pick two as before, then use the **Add record** rail to choose a third, or run a search that narrows to exactly three records and the slots fill themselves. Each card collapses to a slim rail while you read the others. The shared **Expected** column moves to the front of the row, and chat compares Traces A, B and C. At three records, human float [metrics](/features/metrics) trade the five-point scale for a 1st/2nd/3rd ranking, and binary metrics become one winner pick: Pass for the winner, Fail for the rest. |
Member
There was a problem hiding this comment.
This language sounds very AI. Here's a better version:
Suggested change
| The **Compare** page takes a third record. Pick two as before, then use the **Add record** rail to choose a third, or run a search that narrows to exactly three records and the slots fill themselves. Each card collapses to a slim rail while you read the others. The shared **Expected** column moves to the front of the row, and chat compares Traces A, B and C. At three records, human float [metrics](/features/metrics) trade the five-point scale for a 1st/2nd/3rd ranking, and binary metrics become one winner pick: Pass for the winner, Fail for the rest. | |
| The **Compare** page now allows comparing up to three records. When comparing three records, human float metrics become a 1st/2nd/3rd ranking and human binary metrics become a best-winner pick. |
The detail on how to add a third record is unnecessary here in the changelog, because it's pretty obvious from the UI. If you do want to add it, it should go in the docs for the Compare page. The detail on the Expected column moving to the front of the row is also unnecessary because you can see it from the screenshot.
Member
There was a problem hiding this comment.
Also, you should include a reason why this is helpful, not just what changed.
Why is comparing three records at a time useful to our users?
|
|
||
| ### Score per-model metrics in the preference panel | ||
|
|
||
| Human metrics that sit on one of the compared records but not the other (a reviewer's `model a preference` on one card, `model b preference` on the other) now get their own rows in the preference panel. Pass/Fail writes to that record alone. Int and float metrics show the current value with an edit button onto the scoring form. Put two of these metrics in a **metric group** and they share one row, labeled with the group's name. |
Member
There was a problem hiding this comment.
Suggested change
| Human metrics that sit on one of the compared records but not the other (a reviewer's `model a preference` on one card, `model b preference` on the other) now get their own rows in the preference panel. Pass/Fail writes to that record alone. Int and float metrics show the current value with an edit button onto the scoring form. Put two of these metrics in a **metric group** and they share one row, labeled with the group's name. | |
| We now group metrics by their metric group on the Compare page. This is useful if you have metrics like "Model A preference" and "Model B preference" that you want to see on the same row. |
|
|
||
| ### Bug Fixes and Improvements | ||
|
|
||
| - **[Compare]** Shared human float metrics get a five-point preference scale: Strong A, Slight A, Equal, Slight B, Strong B. Each pick saves a matching pair of scores to the two records. The panel's "Model preference" heading is gone. |
Member
There was a problem hiding this comment.
Suggested change
| - **[Compare]** Shared human float metrics get a five-point preference scale: Strong A, Slight A, Equal, Slight B, Strong B. Each pick saves a matching pair of scores to the two records. The panel's "Model preference" heading is gone. | |
| - **[Compare]** Shared human float metrics get a five-point preference scale: Strong A, Slight A, Equal, Slight B, Strong B. Each pick saves a matching pair of scores to the two records. |
| - **[Compare]** Shared human float metrics get a five-point preference scale: Strong A, Slight A, Equal, Slight B, Strong B. Each pick saves a matching pair of scores to the two records. The panel's "Model preference" heading is gone. | ||
| - **[Compare]** The preference panel names each side after the model that produced the record, taken from its `model:` tag, instead of by slot position. A record without a usable tag falls back to its run or record id. | ||
| - **[Compare]** Clicking a metric name in the preference panel opens its scoring guidelines, and record tags show next to the record id on the cards and on the panel's column headers. | ||
| - **[Compare]** Docx attachments render inline below a record's output on the compare cards, tracked changes and Word comments included. Filenames mentioned in the text are clickable chips, and highlight-to-annotate works over the inline document. |
Member
There was a problem hiding this comment.
grammar nit
Suggested change
| - **[Compare]** Docx attachments render inline below a record's output on the compare cards, tracked changes and Word comments included. Filenames mentioned in the text are clickable chips, and highlight-to-annotate works over the inline document. | |
| - **[Compare]** Docx attachments render inline below a record's output on the compare cards, including tracked changes and Word comments. Filenames mentioned in the text are clickable chips and highlight-to-annotate works over the inline document. |
| - **[Compare]** Clicking a metric name in the preference panel opens its scoring guidelines, and record tags show next to the record id on the cards and on the panel's column headers. | ||
| - **[Compare]** Docx attachments render inline below a record's output on the compare cards, tracked changes and Word comments included. Filenames mentioned in the text are clickable chips, and highlight-to-annotate works over the inline document. | ||
| - **[Records]** The **Add widget** menu greys out widgets already on the board. Saved or imported layouts that hold duplicates keep the first instance. | ||
| - **[API]** New endpoints for metric groups: list and create them under a project, then fetch, update, or archive one by id. |
Member
There was a problem hiding this comment.
These should be links to the API reference pages (although they don't exist yet)
Suggested change
| - **[API]** New endpoints for metric groups: list and create them under a project, then fetch, update, or archive one by id. | |
| - **[API]** New endpoints for metric groups: [list](/api-reference/list-metric-group) and [create](/...) them under a project, then [fetch](/...), [update](/...), or [archive](/...) one by id. |
| - **[Compare]** Docx attachments render inline below a record's output on the compare cards, tracked changes and Word comments included. Filenames mentioned in the text are clickable chips, and highlight-to-annotate works over the inline document. | ||
| - **[Records]** The **Add widget** menu greys out widgets already on the board. Saved or imported layouts that hold duplicates keep the first instance. | ||
| - **[API]** New endpoints for metric groups: list and create them under a project, then fetch, update, or archive one by id. | ||
| - **[Projects]** The [project](/features/projects) selector loads every project's record count in a single request rather than one request per project. |
Member
There was a problem hiding this comment.
This projects bullet point is an implementation detail. You could do:
Suggested change
| - **[Projects]** The [project](/features/projects) selector loads every project's record count in a single request rather than one request per project. | |
| - **[Projects]** The project selector loads faster. |
but IMO this is sufficiently minor that you shouldn't even include it:
Suggested change
| - **[Projects]** The [project](/features/projects) selector loads every project's record count in a single request rather than one request per project. |
Applies Vishal's review suggestions: plainer three-record compare copy with the reason it helps, simplified metric-group section, drops the removed-heading and project-selector bullets, grammar fix on the docx bullet, and links the metric-group API bullet to its api-reference pages.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated weekly changelog entry. Opened by the fallback step because the bot pushed the branch without creating a PR, so the usual summary is missing — review changelog.mdx directly.