Skip to content

Update Pulse leaderboard rows #73

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xujustinj merged 1 commit into from
Oct 16, 2026
Merged

Update Pulse leaderboard rows #73

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,16 +24,16 @@ Models and prices reflect each provider's official documentation; each system us

| Rank | Provider | Category | Overall | Short | Medium | Long | ¢ / Page |
|---:|---|---|---:|---:|---:|---:|---:|
| 1 | LlamaExtract Agentic Plus | LlamaExtract | **96.38** | <u>97.15</u> | 94.77 | <u>94.60</u> | 7.50¢ |
| 2 | Reducto Deep Extract | Specialized APIs | <u>96.08</u> | 96.84 | <u>94.80</u> | 92.85 | 6.35¢ |
| 3 | LlamaExtract Agentic | LlamaExtract | 96.03 | 96.81 | 94.21 | **95.05** | 3.12¢ |
| 4 | Pulse (Effort) | Specialized APIs | 95.91 | 96.46 | **95.01** | 93.51 | 10.50¢ |
| 1 | LlamaExtract Agentic Plus | LlamaExtract | **96.38** | <u>97.15</u> | <u>94.77</u> | <u>94.60</u> | 7.50¢ |
| 2 | Reducto Deep Extract | Specialized APIs | <u>96.08</u> | 96.84 | **94.80** | 92.85 | 6.35¢ |
| 3 | Pulse (Effort) | Specialized APIs | 96.07 | 96.83 | 94.70 | 93.24 | 10.50¢ |
| 4 | LlamaExtract Agentic | LlamaExtract | 96.03 | 96.81 | 94.21 | **95.05** | 3.12¢ |
| 5 | Claude Code (Sonnet 5.5 Evidence) | Coding Agents | 94.68 | 96.22 | 92.97 | 83.69 | 5.69¢ |
| 6 | Codex (GPT-6 Sol Evidence) | Coding Agents | 94.57 | 95.99 | 92.42 | 87.25 | 9.80¢ |
| 7 | Codex (GPT-5.6 Sol Evidence) | Coding Agents | 93.77 | 96.04 | 90.19 | 82.73 | 21.65¢ |
| 8 | LlamaExtract Cost-Effective | LlamaExtract | 93.67 | 96.07 | 90.21 | 80.48 | 1.00¢ |
| 9 | Codex (GPT-5.5) | Coding Agents | 93.57 | 95.68 | 91.15 | 78.88 | 27.83¢ |
| 10 | Codex (GPT-5.5 Evidence) | Coding Agents | 93.35 | 95.59 | 88.72 | 87.97 | 33.63¢ |
| 7 | Pulse (Non-effort) | Specialized APIs | 94.51 | 95.39 | 92.49 | 93.31 | 3.00¢ |
| 8 | Codex (GPT-5.6 Sol Evidence) | Coding Agents | 93.77 | 96.04 | 90.19 | 82.73 | 21.65¢ |
| 9 | LlamaExtract Cost-Effective | LlamaExtract | 93.67 | 96.07 | 90.21 | 80.48 | 1.00¢ |
| 10 | Codex (GPT-5.5) | Coding Agents | 93.57 | 95.68 | 91.15 | 78.88 | 27.83¢ |

Top 10 of 49 systems — full table in [leaderboard.csv](leaderboard.csv).
<!-- LEADERBOARD:END -->
Expand All @@ -47,9 +47,9 @@ Top 10 of 49 systems — full table in [leaderboard.csv](leaderboard.csv).
<tr><th align="right">Overall</th><th align="right">Short</th><th align="right">Medium</th><th align="right">Long</th><th align="right">Overall</th><th align="right">Short</th><th align="right">Medium</th><th align="right">Long</th></tr>
</thead>
<tbody>
<tr><td align="right">1</td><td>LlamaExtract Agentic Plus</td><td align="right"><strong>81.26</strong></td><td align="right"><strong>81.63</strong></td><td align="right"><u>81.18</u></td><td align="right">75.22</td><td align="right"><u>89.94</u></td><td align="right"><u>92.45</u></td><td align="right"><u>86.50</u></td><td align="right">74.71</td></tr>
<tr><td align="right">2</td><td>LlamaExtract Agentic</td><td align="right"><u>78.89</u></td><td align="right"><u>77.63</u></td><td align="right"><strong>81.86</strong></td><td align="right"><strong>80.89</strong></td><td align="right"><strong>90.64</strong></td><td align="right"><strong>92.58</strong></td><td align="right"><strong>86.68</strong></td><td align="right"><strong>87.09</strong></td></tr>
<tr><td align="right">3</td><td>Codex (GPT-6 Sol Evidence)</td><td align="right">77.11</td><td align="right">77.61</td><td align="right">75.53</td><td align="right"><u>78.31</u></td><td align="right">86.50</td><td align="right">87.20</td><td align="right">85.12</td><td align="right"><u>84.82</u></td></tr>
<tr><td align="right">1</td><td>LlamaExtract Agentic Plus</td><td align="right"><strong>81.26</strong></td><td align="right"><strong>81.63</strong></td><td align="right"><u>81.18</u></td><td align="right">75.22</td><td align="right"><u>89.94</u></td><td align="right"><u>92.45</u></td><td align="right">86.50</td><td align="right">74.71</td></tr>
<tr><td align="right">2</td><td>LlamaExtract Agentic</td><td align="right"><u>78.89</u></td><td align="right"><u>77.63</u></td><td align="right"><strong>81.86</strong></td><td align="right"><strong>80.89</strong></td><td align="right"><strong>90.64</strong></td><td align="right"><strong>92.58</strong></td><td align="right"><u>86.68</u></td><td align="right"><u>87.09</u></td></tr>
<tr><td align="right">3</td><td>Codex (GPT-6 Sol Evidence)</td><td align="right">77.11</td><td align="right">77.61</td><td align="right">75.53</td><td align="right"><u>78.31</u></td><td align="right">86.50</td><td align="right">87.20</td><td align="right">85.12</td><td align="right">84.82</td></tr>
<tr><td align="right">4</td><td>Claude Code (Opus 5.5 Evidence)</td><td align="right">71.55</td><td align="right">73.57</td><td align="right">65.54</td><td align="right">74.87</td><td align="right">76.95</td><td align="right">79.27</td><td align="right">71.36</td><td align="right">76.47</td></tr>
<tr><td align="right">5</td><td>Codex (GPT-6 Luna Evidence)</td><td align="right">65.70</td><td align="right">62.91</td><td align="right">71.11</td><td align="right">74.36</td><td align="right">83.93</td><td align="right">84.41</td><td align="right">83.47</td><td align="right">80.17</td></tr>
<tr><td align="right">6</td><td>Claude Code (Sonnet 5.5 Evidence)</td><td align="right">64.61</td><td align="right">64.59</td><td align="right">63.18</td><td align="right">71.22</td><td align="right">75.99</td><td align="right">77.93</td><td align="right">71.50</td><td align="right">74.98</td></tr>
Expand Down
4 changes: 2 additions & 2 deletions leaderboard.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
Provider,Category,Overall,Short,Medium,Long,Cost_Per_Page,Cost_Short,Cost_Medium,Cost_Long,P_Short,P_Medium,P_Long,R_Short,R_Medium,R_Long,Word_Grounding,Word_Grounding_Short,Word_Grounding_Medium,Word_Grounding_Long,Page_Grounding,Page_Grounding_Short,Page_Grounding_Medium,Page_Grounding_Long,Latency_S_Per_Doc_Short,Latency_S_Per_Doc_Medium,Latency_S_Per_Doc_Long
LlamaExtract Agentic Plus,LlamaExtract,96.38,97.15,94.77,94.60,0.0750,0.0750,0.0750,0.0750,97.17,95.31,94.35,97.18,94.29,94.86,81.26,81.63,81.18,75.22,89.94,92.45,86.50,74.71,167.8,243.9,821.6
Pulse (Effort),Specialized APIs,95.91,96.46,95.01,93.51,0.1050,0.1050,0.1050,0.1050,96.44,95.28,94.07,96.50,94.89,93.19,1.49,1.17,3.03,0.04,79.70,77.40,86.19,77.74,125.7,206.3,453.1
Pulse (Effort),Specialized APIs,96.07,96.83,94.70,93.24,0.1050,0.1050,0.1050,0.1050,96.85,94.93,93.71,96.86,94.60,92.96,14.94,12.58,20.80,15.96,83.62,81.32,87.95,90.08,118.0,249.7,507.3
Claude Code (Sonnet 5.5 Evidence),Coding Agents,94.68,96.22,92.97,83.69,0.0569,0.0735,0.0237,0.0091,96.68,94.47,83.36,95.93,91.83,84.05,64.61,64.59,63.18,71.22,75.99,77.93,71.50,74.98,82.5,87.7,187.0
Codex (GPT-6 Sol Evidence),Coding Agents,94.57,95.99,92.42,87.25,0.0980,0.1279,0.0388,0.0122,96.33,93.49,91.28,95.80,91.72,86.70,77.11,77.61,75.53,78.31,86.50,87.20,85.12,84.82,104.2,89.9,101.4
Codex (GPT-5.6 Sol Evidence),Coding Agents,93.77,96.04,90.19,82.73,0.2165,0.2813,0.0870,0.0300,96.28,91.37,86.24,95.84,89.21,82.42,54.66,50.87,64.58,69.01,83.62,85.12,80.77,78.84,95.2,106.6,197.2
Codex (GPT-5.5),Coding Agents,93.57,95.68,91.15,78.88,0.2783,0.3579,0.1214,0.0434,96.01,93.64,94.03,95.51,90.16,78.58,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,70.3,82.6,122.0
Codex (GPT-5.5 Evidence),Coding Agents,93.35,95.59,88.72,87.97,0.3363,0.4354,0.1406,0.0472,95.86,89.85,89.29,95.36,87.86,87.08,52.57,53.13,48.01,61.87,80.64,83.27,73.87,79.90,89.0,117.8,164.6
Pulse (Non-effort),Specialized APIs,92.70,93.17,92.22,89.07,0.0300,0.0300,0.0300,0.0300,93.36,92.59,90.81,93.17,92.01,87.82,0.71,0.82,0.49,0.01,73.42,73.81,69.86,84.72,87.2,140.0,314.3
Pulse (Non-effort),Specialized APIs,94.51,95.39,92.49,93.31,0.0300,0.0300,0.0300,0.0300,95.71,93.34,93.91,95.25,91.86,92.86,13.37,11.56,18.03,13.47,81.44,80.87,82.29,84.08,174.7,244.9,499.3
Codex (GPT-5.6 Terra Evidence),Coding Agents,92.26,95.60,86.69,77.36,0.0945,0.1231,0.0376,0.0121,95.99,89.59,92.33,95.38,85.20,76.71,43.68,40.48,53.32,51.17,80.52,83.51,75.18,69.32,69.4,92.0,123.3
OpenAI GPT-6 Astra,Commercial VLM,91.91,97.22,90.56,31.70,0.1109,0.1320,0.0677,0.0382,97.14,92.08,75.58,97.32,90.16,29.87,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,35.0,106.4,90.1
Codex (GPT-5.6 Luna Evidence),Coding Agents,91.16,93.50,87.43,80.05,0.0120,0.0156,0.0048,0.0016,94.06,89.54,85.29,93.25,86.66,79.28,40.03,37.18,47.66,50.89,79.81,82.85,74.07,69.50,110.4,138.1,256.1
Expand Down
Loading