Skip to content

Commit e9d8846

Browse files
mmilenkoskimmilenkoski
authored
Combine benchmark tables
1 parent dc4b6fe commit e9d8846

1 file changed

Lines changed: 117 additions & 22 deletions

File tree

‎_drafts/mlbench-vs-mlperf.md‎

Lines changed: 117 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -30,25 +30,120 @@ MLPerf restricts the set of hyperparameters that can be tuned. It also allows us
3030

3131
## Benchmark Suites
3232

33-
### MLBench
34-
35-
| Task | Benchmark | Dataset | Quality Target | Reference Implementation Model | Frameworks |
36-
|:-------: |:----------------------------: |:------------------: |:------------------: |:------------------------------: |:-------------------: |
37-
| Task 1a | Image classification | CIFAR10 (32x32) | 80% Top-1 Accuracy | ResNet-20 | PyTorch, Tensorflow |
38-
| Task 1b | Image classification | ImageNet (224x224) | TODO | TODO | TODO |
39-
| Task 3a | Language Modelling | Wikitext2 | Perplexity <= 50 | RNN-LM | PyTorch |
40-
| Task 4a | Translation (recurrent) | WMT16 EN-DE | 24.0 BLEU | GNMT | PyTorch |
41-
| Task 4b | Translation (non-recurrent) | WMT17 EN-DE | 25.0 BLEU | Transformer | PyTorch |
42-
43-
### MLPerf
44-
45-
| Benchmark | Dataset | Quality Target | Reference Implementation Model | Frameworks |
46-
|:-------------------------------: |:-----------------------: |:-----------------------------------: |:------------------------------: |:-------------------: |
47-
| Image classification | ImageNet (224x224) | 75.9% Top-1 Accuracy | Resnet-50 v1.5 | MXNet, Tensorflow |
48-
| Object detection (light weight) | COCO 2017 | 23% mAP | SSD-ResNet34 | Tensorflow, PyTorch |
49-
| Object detection (heavy weight) | COCO 2017 | 0.377 Box min AP, 0.339 Mask min AP | Mask R-CNN | Tensorflow, PyTorch |
50-
| Translation (recurrent) | WMT English-German | 24.0 BLEU | GNMT | Tensorflow, PyTorch |
51-
| Translation (non-recurrent) | WMT English-German | 25.0 BLEU | Transformer | Tensorflow, PyTorch |
52-
| Recommendation | Undergoing modification | | | |
53-
| Reinforcement learning | N/A | Pre-trained checkpoint | Mini Go | Tensorflow |
54-
33+
<table>
34+
<thead>
35+
<tr>
36+
<th rowspan="2">Benchmark</th>
37+
<th colspan="2">Dataset</th>
38+
<th colspan="2">Quality Target</th>
39+
<th colspan="2">Reference Implementation Model</th>
40+
<th colspan="2">Frameworks</th>
41+
</tr>
42+
<tr>
43+
<td>MLBench</td>
44+
<td>MLPerf</td>
45+
<td>MLBench</td>
46+
<td>MLPerf</td>
47+
<td>MLBench</td>
48+
<td>MLPerf</td>
49+
<td>MLBench</td>
50+
<td>MLPerf</td>
51+
</tr>
52+
</thead>
53+
<tbody>
54+
<tr>
55+
<td>Image classification</td>
56+
<td>CIFAR10 (32x32)</td>
57+
<td>/</td>
58+
<td>80% Top-1 Accuracy</td>
59+
<td>/</td>
60+
<td>ResNet-20</td>
61+
<td>/</td>
62+
<td>PyTorch, Tensorflow</td>
63+
<td>/</td>
64+
</tr>
65+
<tr>
66+
<td>Image classification</td>
67+
<td colspan="2">ImageNet (224x224)</td>
68+
<td>TODO</td>
69+
<td>75.9% Top-1 Accuracy</td>
70+
<td>TODO</td>
71+
<td>Resnet-50 v1.5</td>
72+
<td>TODO</td>
73+
<td>MXNet, Tensorflow</td>
74+
</tr>
75+
<tr>
76+
<td>Object detection (light weight)</td>
77+
<td>/</td>
78+
<td>COCO 2017</td>
79+
<td>/</td>
80+
<td>23% mAP</td>
81+
<td>/</td>
82+
<td>SSD-ResNet34</td>
83+
<td>/</td>
84+
<td>Tensorflow, PyTorch</td>
85+
</tr>
86+
<tr>
87+
<td>Object detection (heavy weight)</td>
88+
<td>/</td>
89+
<td>COCO 2017</td>
90+
<td>/</td>
91+
<td>0.377 Box min AP, 0.339 Mask min AP</td>
92+
<td>/</td>
93+
<td>Mask R-CNN</td>
94+
<td>/</td>
95+
<td>Tensorflow, PyTorch</td>
96+
</tr>
97+
<tr>
98+
<td>Language Modelling</td>
99+
<td>Wikitext2</td>
100+
<td>/</td>
101+
<td>Perplexity &lt;= 50</td>
102+
<td>/</td>
103+
<td>RNN-LM</td>
104+
<td>/</td>
105+
<td>PyTorch</td>
106+
<td>/</td>
107+
</tr>
108+
<tr>
109+
<td>Translation (recurrent)</td>
110+
<td>WMT16 EN-DE</td>
111+
<td>WMT English-German</td>
112+
<td colspan="2">24.0 BLEU</td>
113+
<td colspan="2">GNMT</td>
114+
<td>PyTorch</td>
115+
<td>Tensorflow, PyTorch</td>
116+
</tr>
117+
<tr>
118+
<td>Translation (non-recurrent)</td>
119+
<td>WMT17 EN-DE</td>
120+
<td>WMT English-German</td>
121+
<td colspan="2">25.0 BLEU</td>
122+
<td colspan="2">Transformer</td>
123+
<td>PyTorch</td>
124+
<td>Tensorflow, PyTorch</td>
125+
</tr>
126+
<tr>
127+
<td>Recommendation</td>
128+
<td>/</td>
129+
<td>Undergoing modification</td>
130+
<td>/</td>
131+
<td></td>
132+
<td>/</td>
133+
<td></td>
134+
<td>/</td>
135+
<td></td>
136+
</tr>
137+
<tr>
138+
<td>Reinforcement learning</td>
139+
<td>/</td>
140+
<td>N/A</td>
141+
<td>/</td>
142+
<td>Pre-trained checkpoint</td>
143+
<td>/</td>
144+
<td>Mini Go</td>
145+
<td>/</td>
146+
<td>Tensorflow</td>
147+
</tr>
148+
</tbody>
149+
</table>

0 commit comments

Comments
 (0)